A table question and answer task capability enhancement processing method of a large language model

Abstract

The application discloses a table question and answer task ability enhancement processing method of a large language model. For table data, a TABLE-UAM network including an encoding part and a decoding part is constructed; different table data sets are sequentially input into the TABLE-UAM network for two-stage training, the first stage is difference reconstruction training, and the second stage is combined with multiple large language models for training; the trained TABLE-UAM network and the large language model are spliced, and used for processing input question text and table data to output answers. The newly-built TABLE-UAM network structure can efficiently encode and feature extract table data, effectively perceive table row and column dependency features, multi-table dependency features and global relationship features; and the two-stage training method can improve the table analysis task ability, significantly enhance the cross-model migration ability, and has strong generalization and practical value.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a computer network structure and method for enhancing the ability of Large Language Models (LLMs) to perform table analysis and processing tasks. Specifically, it relates to an optimized method for building, training, and applying network models that support cross-model transfer and enhance the ability to perform table tasks. Background Technology

[0002] Tabular data is a primary form of enterprise data, widely found in scenarios such as financial transaction records, medical test results, and manufacturing quality inspection. With the widespread application of Large Language Models (LLM) in natural language processing, LLM-based tabular data analysis tasks have gradually become a research hotspot. These tasks typically take tabular data and a natural language question as input and output a natural language answer, such as tabular question answering, tabular fact verification, tabular manipulation, tabular query statistics, trend analysis, and correlation analysis. Accurate positioning, understanding, and reasoning abilities are required regarding the content of the two-dimensional table itself, the relationships between rows and columns, and the connections between the question and the tabular data.

[0003] In existing technologies, LLM mainly suffers from the following problems in table analysis and processing tasks:

[0004] Insufficient ability to process tabular data: Currently, LLM is still designed according to the input form of text sequence in terms of network structure. It only supports the input of text sequence. If it is tabular data, it is converted into a one-dimensional form of text sequence before being input into the large language model. This conversion not only increases the input length and reduces the encoding efficiency, but also loses the dependency and association information between the contents of the two-dimensional cells of the table.

[0005] The reusability and transferability of task-specific capabilities acquired after training large models are poor: Fine-tuning a domain-specific capability on an LLM typically requires significant computational resources, but this capability is tied to a specific LLM and is difficult to transfer directly to other LLM backbone models, resulting in poor cross-model reusability. When the backbone model is updated or a new LLM emerges, the entire model needs to be retrained, causing enormous waste.

[0006] Therefore, existing technologies lack a new special network structure module that can efficiently encode and understand tabular data, thereby improving the ability to perform tabular analysis and processing tasks and supporting migration and adaptation between different LLM backbone models. Summary of the Invention

[0007] The purpose of this invention is to address the problems of LLM in table analysis and processing tasks mentioned in the background art. It proposes a network component (TABLE-UAM) that can efficiently encode and process table data, and proposes a two-stage training method for this network component, which can not only improve the table analysis and processing task capabilities, but also support the transfer adaptation between different LLM backbone models.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] Step 1: For tabular data, construct a TABLE-UAM network that includes encoding and decoding parts, featuring multi-table dependencies, global association characteristics, and the ability to migrate table tasks;

[0010] The TABLE-UAM network has a unique network structure that efficiently encodes and extracts features from tabular data, effectively perceives short-term and long-term dependencies between rows and columns, multi-table dependencies, and global associations. It can adapt to different LLMs without training and transfer table tasks.

[0011] Step 2: Using different tabular datasets, input the different tabular datasets into the TABLE-UAM network in sequence for training in two stages. The first stage is to perform differential reconstruction training, and the second stage is to combine different large language models for training.

[0012] Step 3: Concatenate the trained TABLE-UAM network with the actual large language model to process the input question text and table data and output the answer.

[0013] In the TABLE-UAM network, the TABLE-UAM network module is mainly composed of an encoding part and a decoding part connected in sequence, including a text encoding network (PS1), a table encoding network (PS2), a feature extraction fusion network, and a table decoding network. The encoding part includes the text encoding network, the table encoding network, and the feature extraction fusion network, while the decoding part includes the table decoding network. The table decoding network in the decoding part is only used in the first stage of training, and only the encoding part is used in the second stage. The encoding part is used in both stages. The text encoding network and the table encoding network are set up in parallel. The question text and table data are respectively input into the text encoding network and the table encoding network. The outputs of the text encoding network and the table encoding network are respectively input into the feature extraction fusion network for processing and outputting global encoded features Fe and adapted projection features Ty. The global encoded features Fe are then input into the table decoding network to obtain the reconstructed table Fde. The reconstructed table Fde is used for the first stage of training.

[0014] The tabular data includes formats such as CSV and Excel, and includes headers and row / column content data.

[0015] The difference between unsupervised and supervised table task datasets lies in their input and output. Unsupervised table datasets contain only table data and do not require labels; the task is simply to input table data and output the same table data, representing a reconstruction learning process based on the input table data. Supervised table task datasets include tasks such as table question answering, table fact verification, table manipulation, table query statistics, and table relevance analysis. They consist of three parts: the input table, the question, and the output answer (the label). The model needs to learn how to combine the table and the question to arrive at the correct answer.

[0016] The encoding process of the TABLE-UAM network is as follows:

[0017] S1. Input the problem text Q and the table set BT into the text encoding network PS1 and the table processing network PS2 respectively to obtain the problem text encoding features QT and the table global features Fm;

[0018] The input question text Q is processed by the text encoding network PS1 to obtain the encoded text features QF, and then the text is upgraded to obtain the text upgrade features QE. Finally, the text upgrade features QE are processed by the linear dimension change network layer to obtain the question text encoded features QT.

[0019] The input set of tables BT is processed by the table processing network PS2 to obtain its own column encoding features BNC1. The column encoding features BNC1 of all tables in the set of tables BT are then merged by column to obtain a single table encoding feature Fb. The individual Fb features are then padded to the same length according to the first two dimensions to obtain their respective padded features FP(Fp1,...,Fpt). Next, the padded features FP(Fp1,...,Fpt) of all tables are concatenated to form a fused table feature Fr. Finally, a two-dimensional convolution is used to perform convolution feature extraction on the fused table feature Fr to obtain the global table feature Fm.

[0020] S2. The text encoding features QT and the global table features Fm are fused using cross-attention to obtain the fused encoding features Fa, denoted as:

[0021] Fa = Softmax( (Fm*w_q) (QT*w_k) T / dk 1 / 2 (QT*w_v)

[0022] Where dk is the Attention transformation dimension of cross-attention, w_q, w_k, and w_v are the first, second, and third weight parameters of cross-attention, respectively; T represents transpose;

[0023] S3. Use the Transformer Encoder network to process the fused coding feature Fa to obtain the global coding feature Fe. Then, use average pooling to process the global coding feature Fe to obtain the adaptation projection feature Ty. The adaptation projection feature Ty is used to be spliced ​​and adapted with the Embedding layer of the embedded coding network in the subsequent large language model LLM.

[0024] The decoding process of the TABLE-UAM network is as follows:

[0025] The global encoding feature Fe is input into the table decoding network. The table decoding network transforms the global encoding feature Fe to obtain the transformed dimension feature Fdc. Then, the transformed dimension feature Fdc is processed by the Transformer Decoder network to restore the reconstructed table Fde corresponding to the original input table.

[0026] The table encoding network PS2 internally incorporates the processing method of the text encoding network PS1. The specific processing procedure is as follows:

[0027] D1. Input a single table B, split table B into columns, and encode the column name, column position index, and cell content of each column to obtain the column name code BC1, column position code BP1, and cell content code BN respectively.

[0028] D2. Encode the cell content BN and process it through an attention mechanism network to obtain the cell transformation feature BNA1. The formula is:

[0029] BNA1=softmax( (BN * wq) * (BN * wk) T / dk 1 / 2 ) * (BN * wv),

[0030] Where BN represents the cell content code, BN is the numeric cell content code BN1 / character cell content code BN2; dk is the Attention transformation dimension of the attention mechanism network, and wq, wk, and wv are the first, second, and third weight parameters of the attention mechanism network, respectively.

[0031] D3. The column name code BC1 and the column position code BP1 are added and merged element by element to obtain the column name and position merging feature BCP1 that integrates column position index and column name information;

[0032] D4. Merge the cell transformation feature BNA1 and the column name and position merging feature BCP1 to obtain the column combination feature BNM1. Then, perform a one-dimensional convolution operation on the column combination feature BNM1 along the column dimension to obtain the final column encoding feature BNC1.

[0033] Step D1 specifically includes:

[0034] The column name encoding BC1 is specifically obtained by encoding the column name using the same method as the text encoding network PS1;

[0035] Specifically, the column position encoding BP1 is obtained by encoding the column position index in the same way as step C2B in the text encoding network PS1.

[0036] The cell content encoding BN is of two types:

[0037] If the cell content is a numeric type, then a linear network layer is used to transform the dimensions of the cell content.

[0038] If the cell content is of character type, then each character in the cell content is encoded in the same way as the text encoding network PS1, and then the encoded results of all characters are added element by element according to the encoding dimension.

[0039] The specific processing procedure of the text encoding network PS1 is as follows:

[0040] C1. Input the question text W. Each question text W is processed by word segmentation and indexing, resulting in L word vector units T1. Each element corresponds to a minimum semantic unit token.

[0041] C2. Perform word vector and position mapping and encoding processing on word vector unit T1 respectively:

[0042] After C2A and word segmentation indexing, the word vector unit T1 maps the index of each smallest semantic unit to a dense vector of fixed dimension through learnable embedding matrix parameters to obtain word vector information C1;

[0043] C2B, and at the same time, the word vector unit T1 after word segmentation indexing is encoded by position, and the index position of each smallest semantic unit is encoded to obtain position information P1;

[0044] C3. Add the word vector information C1 and the position information P1 element by element to obtain the encoded feature Ft.

[0045] Step 2 specifically involves:

[0046] 21) Using an unsupervised tabular dataset, the tabular dataset is input into the encoding and decoding parts of the TABLE-UAM network. The decoding part outputs the reconstructed tabular data, and the reconstructed tabular data is used for reconstruction difference training.

[0047] 22) The TABLE-UAM network trained by S1 is concatenated with different large language models (LLMs). Using a supervised table task dataset, the question text and table data are input into the overall network model after the encoding part of the TABLE-UAM network is connected to different large language models (LLMs). The output is the predicted answer corresponding to each large language model (LLM). The weight parameters of the large language models (LLMs) are frozen. The encoding part of the TABLE-UAM network is trained under supervision based on each predicted answer and the corresponding label answer. This results in the retrained TABLE-UAM network and each trained large language model (LLM).

[0048] The reconstruction differential training in step 21) specifically involves:

[0049] The implicit table reconstruction loss L is constructed by using the reconstructed table Fde output by the decoding part of the TABLE-UAM network and the original input table. rec , means as follows:

[0050] ;

[0051] Where α represents the table content reconstruction loss weight coefficient, β represents the table structure corresponding loss coefficient, γ represents the column position preservation coefficient, M represents the total number of rows in the input table BT, N represents the total number of columns in the input table BT, i represents the row number index of the input table BT, and j represents the column number index of the input table B; L cell (BT ij, Fde ij ) represents the cell loss function, BT ij This represents the value at the corresponding position in the i-th row and j-th column of the input table, Fde ij This represents the value at the corresponding position in the i-th row and j-th column of the reconstructed table Fde; (BT) ij ) ka This indicates the input cell category value at the i-th row and j-th column of the table. (Fde) ij ) ka This represents the cell category value at the i-th row and j-th column of the reconstructed table Fde; L j This indicates the number of characters in the j-th column name of the table, where colj represents the column name of the j-th column, and BT... （l） coljThis indicates that the j-th column in table BT is being processed by the l-th character of the column name. The superscript l indicates the l-th element to be sorted in that column. Overall, it represents the l-th character element in the j-th column of the input table BT. （l） colj This indicates the l-th element in the j-th column of the decoding table Fde; ord() represents the ASCII code value of the character; π(j) represents the actual position index of the j-th column in the reconstructed table Fde; numeric, categorical, and boolean represent numeric, categorical, and boolean types respectively; ka represents the ka-th category, and KA represents the total number of categories; II[BT ij ≠Fde ij The parentheses indicate an indicator function that returns 0 when the condition within the parentheses is true and 1 when it is false. If the original input table BT in row i and column j is the same as the reconstructed table Fde in row i and column j, the function returns 1; otherwise, it returns 0.

[0052] Then, the implicit reconstruction loss L is used in the table. rec The encoding and decoding parts of the TABLE-UAM network are optimized and trained with the goal of minimizing the impact of the network.

[0053] The supervised training in step 22) specifically includes:

[0054] The encoding parts of the TABLE-UAM network trained using reconstructed differential models are concatenated with different large language models (LLMs), using the LLMs as the backbone models. All LLMs constitute the backbone model set, and the weight parameters of all LLMs in the backbone model set are frozen. Using a supervised table task dataset, question text and table data are input into the TABLE-UAM network, and the different LLMs output predicted answers. The following table task total loss L is established between each predicted answer and the labeled answer. task Supervised training of the weight parameters of the TABLE-UAM network:

[0055] L task =δ(Σ K i=1 -((Σ T t=1 logP (t) Ali (A t |A <t , Q, BT)) / T)) / K

[0056] +2θΣ K i=1 Σ K j=i+1[1 - Sim(ALi, ALj)] / (K(K - 1))

[0057] In the formula, L task represents the total loss of the table task, that is, the difference between the text answer output ALi of the i-th LLM and the label answer A. Sim(ALi, ALj) is the text similarity function, that is, the semantic similarity between the text answer output ALi of the i-th LLM and the text answer output ALj of the j-th LLM; K represents the total number of large language models LLM / backbone models, ALi represents the complete predicted answer sequence generated by the i-th large language model LLM, P (t) ALi represents the probability distribution output by the i-th large language model LLM at time step t; A represents the set of label answers, A = [A1, A2,..., A T , T represents the sequence length of the label answer A, A T represents the T-th label answer; A<t represents all label sequences before time t, A<t = [A1, A2,..., A t-1 , Q and BT respectively represent the input question and the data of the input table; δ is the coefficient that controls the average answer task loss; θ is the weight coefficient that controls the prediction answer consistency constraint loss.

[0058] In the above losses, the first term is responsible for training the TABLE-UAM module to correctly understand the natural language question Q and the table data T and generate accurate answers A. Its direct goal is to maximize the conditional probability of the model generating the target answer sequence.

[0059] The second term forces different large language models (LLMs) to generate similar predicted answer intrinsic feature representations after receiving the output of the TABLE-UAM module, adapting to different LLM models.

[0060] The beneficial effects of the present invention are:

[0061] The newly built TABLE-UAM network structure of the present invention can efficiently encode and extract features from table data, effectively perceive the row-column dependence features, multi-table dependence features and global relationship features of the table.

[0062] The two-stage training method designed by the present invention can improve the table analysis task ability and significantly enhance the cross-model transfer ability, with strong generalization and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Att Figure 1 is a schematic topological structure diagram of the text encoding network (PS1) of the present invention;

[0064] Att Figure 2This is a schematic diagram of the table encoding network (PS2) topology of the present invention;

[0065] Appendix Figure 3 This is a schematic diagram of the TABLE-UAM network topology of the present invention;

[0066] Appendix Figure 4 This is a schematic diagram of the first-stage table reconstruction training process of the present invention;

[0067] Appendix Figure 5 This is a schematic diagram of the second-stage table task adaptation training process of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0069] This invention first constructs a table task enhancement network module (TABLE-UAM) that supports text and multiple tabular data inputs. The TABLE-UAM network includes a text encoding network, a table encoding network, and a feature extraction network. The table encoding network can encode and extract features from the input table independently, more efficiently perceiving table row and column dependencies and global two-dimensional information. The TABLE-UAM network can be adapted to different large language models (LLMs), significantly enhancing the table task capabilities of large language models (LLMs) without training.

[0070] The training of the TABLE-UAM network is divided into two stages: the first stage uses tabular data and trains the network based on the differences between the reconstructed table and the original input table to enhance the table encoding and feature extraction capabilities of the TABLE-UAM network.

[0071] The second stage uses supervised table task data for training (table tasks include table question answering, table fact verification, table manipulation, table query statistics, etc.). The tables and questions are input into the TABLE-UAM network, which is then adapted to different LLM large language models to output predicted answers. The backbone model parameters of the large language model LLM are frozen, and the TABLE-UAM parameters are updated based on the correctness of the predicted answers and labels for training, thereby enhancing the table task capabilities and the compatibility and adaptability with different LLM models.

[0072] Finally, in the application, the TABLE-UAM network and the large language model LLM are concatenated. The question text and table data are input into the TABLE-UAM network, and the answer is output from the large language model LLM.

[0073] Step 1: For table-based question-and-answer scenarios and data, construct a TABLE-UAM network that supports question text and multiple table inputs, has multi-table dependencies, global association characteristics, and the ability to transfer table tasks, including encoding and decoding parts;

[0074] like Figure 3 As shown, in the TABLE-UAM network, the TABLE-UAM network module is mainly composed of an encoding part and a decoding part connected in sequence. The encoding part includes a text encoding network, a table encoding network, and a feature extraction and fusion network, while the decoding part includes a table decoding network. The text encoding network and the table encoding network are set up in parallel. The question text and table data are input into the text encoding network and the table encoding network, respectively. The outputs of the text encoding network and the table encoding network are input into the feature extraction and fusion network, respectively, to process and output global encoded features Fe and adaptive projection features Ty. The global encoded features Fe are then input into the table decoding network to obtain the reconstructed table Fde. The reconstructed table Fde is used for the first stage of training.

[0075] During training, both the unsupervised table dataset and the supervised table task dataset contain question text data, table data, and answer data.

[0076] The overall structure of the TABLE-UAM network is shown in the appendix below. Figure 3 As shown, the encoding part includes the text encoding network (PS1), the table encoding network (PS2), and networks for feature extraction and fusion, which are introduced earlier. It supports multiple table inputs, and the decoding part includes a table decoding network.

[0077] The encoding process of the TABLE-UAM network is as follows:

[0078] S1. Input the question text Q and the table set BT into the text encoding network PS1 and the table processing network PS2 respectively to obtain the question text encoding features QT and the table global features Fm respectively;

[0079] The input question text Q is processed by the text encoding network PS1 to obtain the encoded text features QF, and then the text dimensionality is increased to obtain the text dimensionality increase features QE. Then, the text dimensionality increase features QE are processed by the linear dimension change network layer to obtain the question text encoded features QT, which are then fused with the table features.

[0080] The input is a set of tables BT, which contains at least one table. Each table in BT is processed by a table processing network PS2 to obtain its own column encoding feature BNC1. Different tables have different numbers of rows and columns. The column encoding features BNC1 of all tables in BT are then merged by column to obtain a single table encoding feature Fb. The individual Fb features are then padded to the same length according to the first two dimensions to obtain their respective padding features FP(Fp1,...,Fpt). The padding feature representations of all tables are Fp1,...,Fpt. Then, the padding features FP(Fp1,...,Fpt) of all tables are concatenated and combined by concat to form a fused table feature Fr. Finally, a two-dimensional convolution is used to perform convolutional feature extraction on the fused table feature Fr to obtain the global table feature Fm, in order to extract the dependencies between multiple tables.

[0081] S2. The text encoding features QT and the global table features Fm are fused using cross-attention to obtain the fused encoding features Fa, denoted as:

[0082] Fa = Softmax( (Fm*w_q) (QT*w_k) T / dk 1 / 2 (QT*w_v)

[0083] Where dk is the Attention transformation dimension of the cross-attention, which is 768 here, and w_q, w_k, and w_v are the first, second, and third weight parameters of the cross-attention learned during training, respectively. 1 / 2 This ensures that the Softmax function has a smoother and healthier gradient, making model training more stable and efficient; T represents transpose, which swaps the rows and columns of the matrix to align the dimensions of the matrix for effective similarity calculation.

[0084] S3. The fused coding feature Fa is processed using the Transformer Encoder network to obtain the global coding feature Fe. Then, the global coding feature Fe is processed using average pooling to obtain the adapted projection feature Ty. The adapted projection feature Ty is used to be spliced ​​and adapted with the Embedding layer of the embedded coding network in the subsequent large language model LLM.

[0085] In practice, the Transformer Encoder network is configured as follows: model dimension: 768 dimensions; number of attention heads: 4 heads; number of encoder layers: 3 layers; feedforward network dimension: 512 dimensions.

[0086] The MeanPooling operation has a dimension of 4096, which is compatible with most LLM models on the market. If the subsequent LLM Embedding layer has a dimension of 1024, only the [0:1024] of the Ty feature is used to concatenate with the LLM, and the concatenation method is selected.

[0087] The decoding process of the TABLE-UAM network is as follows:

[0088] The global encoded features Fe are input into the table decoding network. The table decoding network performs a dimensionality transformation on Fe to obtain the transformed dimensional features Fdc. Then, the transformed dimensional features Fdc are processed by the Transformer Decoder network to reconstruct the original input table Fde. Figure 3 The dashed lines indicate that Fdc and Fde are used only in the first phase of training.

[0089] The table encoding network PS2 also includes the processing method of the text encoding network PS1, as shown in the appendix. Figure 2 As shown, the table encoding network PS2 is also part of the TABLE-UAM network, and the specific processing procedure is as follows:

[0090] D1. Input a single table B. The input table B includes two-dimensional cell data with M rows and N columns and a header column name. Split table B into columns, each column including column name, column position index and column cell content. Encode the column name, column position index and cell content of each column to obtain column name code BC1, column position code BP1 and cell content code BN respectively.

[0091] In practice, column name encoding BC1 is obtained by encoding column names in the same way as in text encoding network PS1; column position encoding BP1 is obtained by encoding column position indices in the same way as in step C2B of text encoding network PS1.

[0092] Cell content encoding BN has two types:

[0093] If the cell content is a numeric type, a linear network layer is used to perform dimensional transformation on the cell content to obtain the numeric cell content encoding BN1;

[0094] If the cell content is of character type, then each character in the cell content is encoded separately using the same method as the text encoding network PS1. Then, the encoded results of all characters are added element by element according to the encoding dimension to obtain the character cell content encoding BN2.

[0095] D2. The cell content encoding BN of either numeric cells (BN1) or character cells (BN2) is processed through an attention mechanism network to obtain the cell transformation feature BNA1. The formula is:

[0096] BNA1=softmax( (BN * wq) * (BN * wk) T / dk 1 / 2 ) * (BN * wv),

[0097] Where BN represents the cell content code, BN is the numeric cell content code BN1 / character cell content code BN2; dk is the Attention transformation dimension of the attention mechanism network, which is 768 here; wq, wk, and wv are the first, second, and third weight parameters of the attention mechanism network learned during training, respectively.

[0098] D3, column name code BC1, and column position code BP1 are added and merged element by element to obtain the column name and position merging feature BCP1, which integrates column position index and column name information;

[0099] D4. Merge the cell transformation feature BNA1 and the column name and position merging feature BCP1 to obtain the column combination feature BNM1. Then, perform a one-dimensional convolution (Conv1d) operation on the column combination feature BNM1 along the column dimension. The one-dimensional convolution uses a small convolution kernel and a small movement step to extract the short-term dependencies of the column data, where kernel_size=2, stride=1, and padding=0, to obtain the final column encoding feature BNC1 with a dimension of Mx768.

[0100] The text encoding network PS1 is attached below. Figure 1 As shown, the text encoding network PS1 is part of the TABLE-UAM network, and the specific processing procedure is as follows:

[0101] C1. Input the question text W. Each question text W is processed by word segmentation and indexing and becomes a word vector unit T1 with L elements, i.e., a vector sequence T1. Word segmentation is to divide the original text into the smallest semantic unit that the model can recognize, i.e., a token. Each element corresponds to a smallest semantic unit token and a token id. The range of each token id is 0~V-1 (V=30522).

[0102] C2. Perform word vector and position mapping and encoding processing on word vector unit T1 respectively:

[0103] After C2A and word segmentation indexing, the word vector unit T1 maps the dictionary index of each smallest semantic unit to a dense vector of fixed dimensions (such as 768 dimensions) through learnable embedding matrix parameters to obtain word vector information C1;

[0104] C2B, and at the same time, the word vector unit T1 after word segmentation indexing is position-encoded, and the position information P1 is obtained by encoding the index position pos (0~L-1) of each smallest semantic unit in T1 by posencodling;

[0105] The encoding process is as follows: if the position index pos is even, then PE(pos,i) = sin(pos / (10000)). 2i / d_model)),

[0106] If pos is odd, then PE(pos,i)=cos(pos / (10000)) 2(i-1) / d_model)), where d_model is the encoding dimension (e.g., d_model is 768), and i is the index of the encoding dimension, ranging from (0 to d_model-1).

[0107] C3. Add the word vector information C1 and the position information P1 element by element to obtain the fused encoded feature Ft.

[0108] Step 2: Using different tabular datasets, input the different tabular datasets into the TABLE-UAM network in sequence for training in two stages. The first stage is to perform differential reconstruction training, and the second stage is to combine different existing large language models (LLMs) for training.

[0109] 21) For example Figure 4 As shown, an unsupervised tabular dataset is used. The tabular dataset is input into the encoding and decoding parts of the TABLE-UAM network. The table decoding network of the decoding part outputs the reconstructed table data, and the reconstructed table data is used for reconstruction difference training.

[0110] like Figure 4 As shown, the reconstruction differential training in step 21) specifically involves:

[0111] Reconstruction differential training builds an implicit table reconstruction loss L between the output and input tables of the decoding network. rec， The following details the process. The implicit table reconstruction loss L is constructed by using the reconstructed table Fde output from the decoding part of the TABLE-UAM network and the original input table. rec , means as follows:

[0112] ;

[0113] Where α represents the table content reconstruction loss weight coefficient, β represents the table structure corresponding loss coefficient, and γ represents the column position preservation coefficient, specifically α=1.0, β=0.5, and γ=0.3; M represents the total number of rows in the input table BT, N represents the total number of columns in the input table BT, i represents the row number of the input table BT, and j represents the column number of the input table B;

[0114] L cell (BT ij, Fde ij ) represents the cell loss function, BT ij This represents the value at the corresponding position in the i-th row and j-th column of the input table, Fde ij This represents the value at the i-th row and j-th column of the reconstructed table Fde; when the cell value is of categorical type, (BT) ij ) ka This indicates the input cell category value at the i-th row and j-th column of the table. (Fde) ij ) ka This represents the cell category value at the position corresponding to the i-th row and j-th column of the reconstructed table Fde;

[0115] L j This indicates the number of characters in the j-th column name of the table, where colj represents the column name of the j-th column (col is short for column, representing the j-th column name in the table), BT （l） colj This indicates that the j-th column in table BT is being processed by the l-th character of the column name. The superscript l indicates the l-th element to be sorted in that column. Overall, it represents the l-th character element in the j-th column of the input table BT. （l） colj This represents the l-th element in the j-th column of the decode table Fde;

[0116] ord() represents the ASCII code value of a character, performing a numeric operation;

[0117] The denominator 255 is used to normalize to the range [0,1] to prevent the loss function from being too large.

[0118] π(j) represents the actual position index of the j-th column in the reconstructed table Fde;

[0119] numeric, categorical, and boolean represent numeric, categorical, and boolean types, respectively. A cell is a specific value. For example, the age column might have values ​​like 25, 35, and 60, which is a numeric type. The occupation column might have values ​​like "doctor" or "teacher," which is a categorical type. The smoker status column might have values ​​like "yes" or "no," which is a boolean type.

[0120] ka represents the ka-th category, and KA represents the total number of categories; if it is a category type, such as the occupation having three types: "doctor," "teacher," and "civil servant," then KA is 3.

[0121] II[BT ij ≠Fde ij The sign indicates that II is an indicator function, also known as a characteristic function. When the condition in parentheses is true, the function returns 0; when it is false, it returns 1. If the i-th row and j-th column of the original input table BT is the same as the i-th row and j-th column of the reconstructed table Fde, it returns 1; otherwise, it returns 0.

[0122] Then, the implicit reconstruction loss L is used in the table. rec The encoding and decoding parts of the TABLE-UAM network are optimized and trained with the goal of minimizing the impact of the network.

[0123] The table reconstruction loss designed above ensures that cell content matches, table structure corresponds, and column positions remain consistent. Using the loss function above, and employing a supervised training method (SFT) with a table dataset, the weight parameters of the TABLE-UAM network are updated. The training process is shown in the appendix below. Figure 4 As shown.

[0124] 22) For example Figure 5 As shown, the TABLE-UAM network and its weight parameters trained by S1 are concatenated with different large language models (LLMs). A supervised table task dataset (obtained in advance through public channels, such as collecting and labeling via huggingface, and using three methods of public model distillation such as chatgpt or deepseek) is used. The question text and table data are respectively input into the overall network model after connecting different large language models (LLMs) to the encoding part of the TABLE-UAM network. The output is the predicted answer corresponding to each large language model (LLM). The weight parameters of the large language models (LLMs) are frozen. The encoding part of the TABLE-UAM network is trained under supervision based on each predicted answer and the corresponding pre-known label answer. This results in a retrained TABLE-UAM network and each trained large language model (LLM). The weight parameters of the large language models (LLMs) are also trained to be fixed.

[0125] like Figure 5 As shown, the supervised training in step 22) specifically involves:

[0126] After the reconstruction difference training, the encoding part in the TABLE-UAM network after the reconstruction difference training is concatenated with different large language models (LLMs) respectively. Using the LLMs as the backbone models, all the LLMs form a set of backbone models, and the original weight parameters of all the LLMs in the set of backbone models are frozen. Using the supervised table task dataset, the question text with known label answers and the table data are respectively input into the TABLE-UAM network, and each different LLM outputs a predicted answer, and the total table task loss L between each of the following predicted answers and the label answers is established task Only perform correctness supervision training on the weight parameters of the TABLE-UAM network:

[0127] L task =δ(Σ K i=1 -((Σ T t=1 logP (t) Ali (A t |A <t , Q, BT)) / T)) / K

[0128] +2θΣ K i=1 Σ K j=i+1 [(1 - Sim(ALi, ALj)) / (K(K - 1))]

[0129] In the formula, L task represents the total table task loss, that is, the difference between the predicted text answer output ALi of the i-th LLM and the label answer A. Sim(ALi, ALj) is the text similarity function, that is, the semantic similarity between the predicted text answer output ALi of the i-th LLM and the predicted text answer output ALj of the j-th LLM; K represents the total number of large language models / backbone models, ALi represents the complete predicted answer sequence generated by the i-th large language model LLM, and P (t) ALi represents the probability distribution output by the i-th large language model LLM at time step t; A represents the set of label answers, A = [A1, A2,..., A T , T represents the sequence length of the label answer A, and A T represents the T-th label answer; A<t represents all label sequences before time t, A<t = [A1, A2,..., A t-1Q and BT represent the input question and the data in the input table, respectively; δ is the coefficient that controls the average answer task loss, and a value of 0.8 is recommended; θ is the weight coefficient that controls the loss due to the consistency constraint of the predicted answer, and a value of 0.2 is recommended.

[0130] In the above scheme, each large language model LLM serves as a backbone model, forming a backbone model set. The weight parameters of the TABLE-UAM network are shared within this backbone model set and are mounted on different backbone models. Supervised training is performed using the loss function designed above. During this process, the parameters of each large language model LLM are frozen, and only the weight parameters of the TABLE-UAM network are updated.

[0131] Step 3: In the application, the trained TABLE-UAM network and the trained Large Language Model (LLM) required for the actual application are concatenated to process the input question text and table data and output the answer corresponding to the input question text under the table data.

[0132] Specific application examples and their details are as follows:

[0133] Step 1: Data Preparation

[0134] A) Collection of tabular datasets:

[0135] The types of forms include, but are not limited to: financial statements, sales records, scientific research experimental data, inventory lists, etc.

[0136] B) Construction of the table-based task question and answer dataset:

[0137] For each table, generate a related natural language question Q, either manually or automatically. Question types include:

[0138] Cell lookup (e.g., "What was the sales revenue in Q1 of 2023?")

[0139] Aggregate calculations (e.g., "What was the total sales revenue for the entire year of 2023?").

[0140] Filter by criteria (e.g., "What products have sales exceeding 1 million?").

[0141] Multi-step reasoning (e.g., "How much higher were the highest-grossing quarters in 2023 compared to the lowest-grossing quarters?")

[0142] Reference answer A was generated by manual annotation or program calculation.

[0143] Step 2: Table Restructuring Training

[0144] Supervised training (SFT) was employed using a tabular dataset, with the AdamW optimizer, gradient clipping, mixed-precision training, and updates to the TABLE-UAM network parameters.

[0145] Step 3: Parallel Adaptation Training of Multiple Models for the Table Task

[0146] A) Selection of Multi-Backbone Model

[0147] Select at least two different LLM backbone models, for example:

[0148] LLM1: GPT architecture based on Transformer

[0149] LLM2: A Transformer-based LLaMA architecture

[0150] LLM3: Qwen Architecture Based on Hybrid Experts (MoE)

[0151] B) Unified Adaptor Module (TABLE-UAM) Mounting

[0152] A TABLE-UAM module with the same parameters is mounted between the input layer and the intermediate layer of each backbone model to handle the fusion of table embedding and question embedding.

[0153] C) Parallel Inference

[0154] Inputting the same set of data and questions (BT, Q) into TABLE-UAM+ LLM1, TABLE-UAM+ LLM2, and TABLE-UAM+ LLM3 simultaneously yields:

[0155] D) Loss Calculation, Backpropagation, and Parameter Update

[0156] Freeze all LLM backbone model parameters and update only the UAM module parameters.

[0157] Using the AdamW optimizer, the learning rate is set to 1×10. -4 The weight decay coefficient is 0.01.

[0158] Gradient clipping (maximum norm 1.0) and mixed precision training (FP16) are employed to improve training stability and efficiency.

[0159] E) Verification and Early Termination

[0160] Monitor answer accuracy on the validation set, and stop training early if the metric does not improve for 5 consecutive validation periods.

[0161] Step 4: Deployment and Migration

[0162] Save the trained TABLE-UAM module parameters as a separate file.

[0163] Loading the TABLE-UAM module onto the new LLM backbone model (which was not used in training) allows for direct execution of table analysis tasks without retraining. Transfer testing demonstrates that its performance retention on the new model is no less than 90% of the original model.

[0164] Comparative Example 1:

[0165] Model Architecture: The TABLE-UAM network module proposed in this invention is not included. There is no separate table input encoding module. When processing table data, the table (along with its structural information, such as headers and row and column indices) is linearly serialized into a long text string using a fixed template (e.g., converting each row of data into a text fragment of "column name: value" and separating it with special symbols such as [ROW], [COL]). This serialized text string is concatenated with a natural language question and directly input into the Large Language Model (LLM) backbone that has not been adapted and trained in the second stage of this invention. The LLM backbone model used in this comparative example is exactly the same as the LLM backbone model (e.g., the LLaMA architecture) used in the second stage training of this invention.

[0166] Data preparation remains consistent (the steps for collecting tabular datasets and constructing tabular task question and answer datasets are consistent with this invention).

[0167] Training Process: The two-stage training method of this invention is not employed. Only the supervised table task dataset (i.e., the table-based question-and-answer data for industrial control, sensors, etc., constructed in step 1) identical to that of the second stage of this invention is used to perform end-to-end supervised fine-tuning training on the above combination of "serialized table input data construction + LLM". During training, the parameters of the LLM backbone model are not frozen and participate in updates along with the input embedding layer. The hyperparameter settings, such as optimizer, learning rate, and batch size, remain consistent with the second stage training of this invention.

[0168] Comparison results:

[0169] When evaluated on the same technical test set (such as industrial control equipment status query, sensor data aggregation and analysis, alarm log filtering, etc.), Comparison 1 shows the following disadvantages:

[0170] The accuracy of the task is significantly lower: due to the lack of separate encoding and feature extraction capabilities for tabular data, the accuracy is about 10% lower than the method of this invention in tasks that require precise location of table positions, such as cell search and conditional filtering; in tasks involving multi-step reasoning and complex aggregation calculations, the accuracy gap widens to more than 20% because the model has difficulty effectively capturing row and column dependencies from serialized text.

[0171] The model inference efficiency is low: because the table is expanded into a long sequence and input into the model one character at a time, the input processing length of the model is significantly increased, which increases the amount of computation and memory usage. The average inference time is about 20% longer than the method of this invention.

[0172] Lack of cross-model transfer capability: The entire model trained on this LLM backbone cannot be transferred and must be retrained on another LLM model with a different architecture, wasting computing power and resources. This proves that the task capabilities obtained by simply fine-tuning the LLM are deeply bound to a specific model architecture and do not possess the cross-model transfer capability of this invention.

[0173] Weak ability to understand table structure: The model is more prone to errors when dealing with problems that require understanding the two-dimensional structure of a table (such as comparing data in the same row or column), indicating that it has failed to effectively learn and utilize the inherent row and column relationship information of the table.

[0174] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

[0175] The above description is only a preferred embodiment of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of this patent application are included in the scope of this patent application.

Claims

1. A method for enhancing the capabilities of a large language model in a table-based question-answering task, characterized in that: Step 1: For tabular data, construct a TABLE-UAM network that includes encoding and decoding parts, featuring multi-table dependencies, global association characteristics, and the ability to migrate table tasks; Step 2: Using different tabular datasets, input the different tabular datasets into the TABLE-UAM network in sequence for training in two stages. The first stage is to perform differential reconstruction training, and the second stage is to combine multiple large language models for training. Step 3: Concatenate the trained TABLE-UAM network with the actual large language model to process the input question text and table data and output the answer; In the TABLE-UAM network, the TABLE-UAM network module is mainly composed of an encoding part and a decoding part connected in sequence. The encoding part includes a text encoding network, a table encoding network, and a feature extraction and fusion network, and the decoding part includes a table decoding network. The text encoding network and the table encoding network are set in parallel. The question text and table data are respectively input into the text encoding network and the table encoding network. The outputs of the text encoding network and the table encoding network are respectively input into the feature extraction and fusion network for processing and outputting global encoded features Fe and adaptive projection features Ty. The global encoded features Fe are then input into the table decoding network to obtain the reconstructed table Fde. The reconstructed table Fde is used for the first stage of training.

2. The method for enhancing the ability of a large language model for table-based question-answering tasks according to claim 1, characterized in that: The encoding process of the TABLE-UAM network is as follows: S1. Input the question text Q and the table set BT into the text encoding network PS1 and the table processing network PS2 respectively to obtain the question text encoding feature QT and the table global feature Fm. The input question text Q is processed by the text encoding network PS1 to obtain the encoded text feature QF, and then the text dimensionality increase operation is performed to obtain the text dimensionality increase feature QE. Then, the text dimensionality increase feature QE is processed by the linear dimension change network layer to obtain the question text encoding feature QT. The input table set BT, each table in the table set BT is processed by the table processing network PS2 to obtain its own column encoding feature BNC1. Then, the column encoding features BNC1 of all tables in the table set BT are merged by column to obtain the single table encoding feature Fb. Then, the Fb features are processed according to the first two dimensions and filled to the same length to obtain their respective filling features FP (Fp1,...,Fpt). Then, the filling features FP (Fp1,...,Fpt) of all tables are concatenated and combined to form the fused table feature Fr. Finally, the fused table feature Fr is convolved using two-dimensional convolution to extract the global table feature Fm. S2. Use cross-attention to fuse the problem text encoding features QT and the table global features Fm to obtain the fused encoding features Fa; S3. Input the global encoding feature Fe into the table decoding network. In the table decoding network, the global encoding feature Fe is transformed to obtain the transformed dimension feature Fdc. Then, the transformed dimension feature Fdc is processed by the Transformer Decoder network to restore the reconstructed table Fde corresponding to the original input table. Then, average pooling is used to process the global encoding feature Fe to obtain the adaptation projection feature Ty. The adaptation projection feature Ty is used to be concatenated and adapted with the Embedding layer of the embedded encoding network in the subsequent large language model LLM.

3. The method for enhancing the ability of a large language model in a table-based question-answering task according to claim 2, characterized in that: The table encoding network PS2 internally incorporates the processing method of the text encoding network PS1. The specific processing procedure is as follows: D1. Input a single table B, split table B into columns, and encode the column name, column position index, and cell content of each column to obtain the column name code BC1, column position code BP1, and cell content code BN respectively. D2. Encode the cell content BN and process it through an attention mechanism network to obtain the cell transformation feature BNA1. The formula is: BNA1=softmax( (BN * wq) * (BN * wk) T / dk 1 / 2 ) * (BN * wv), Where BN represents the cell content code, BN is the numeric cell content code BN1 / character cell content code BN2; dk is the Attention transformation dimension of the attention mechanism network, and wq, wk, and wv are the first, second, and third weight parameters of the attention mechanism network, respectively. D3. The column name code BC1 and the column position code BP1 are added and merged element by element to obtain the column name and position merging feature BCP1 that integrates column position index and column name information; D4. Merge the cell transformation feature BNA1 and the column name and position merging feature BCP1 to obtain the column combination feature BNM1. Then, perform a one-dimensional convolution operation on the column combination feature BNM1 along the column dimension to obtain the final column encoding feature BNC1.

4. The method for enhancing the ability of a large language model for table-based question-answering tasks according to claim 3, characterized in that: Step D1 specifically includes: The column name encoding BC1 is specifically obtained by encoding the column name using the same method as the text encoding network PS1; The column position encoding BP1 is specifically obtained by encoding the column position index using the same method as the text encoding network PS1. The cell content encoding BN is of two types: If the cell content is a numeric type, then a linear network layer is used to transform the dimensions of the cell content. If the cell content is of character type, then each character in the cell content is encoded in the same way as the text encoding network PS1, and then the encoded results of all characters are added element by element according to the encoding dimension.

5. The method for enhancing the ability of a large language model in a table-based question-answering task according to claim 2 or 3, characterized in that: The specific processing procedure of the text encoding network PS1 is as follows: C1. Input the question text W. Each question text W is processed by word segmentation and indexing and becomes a word vector unit T1 with L elements. Each element corresponds to a minimum semantic unit token. C2. Perform word vector and position mapping and encoding processing on word vector unit T1 respectively: After C2A and word segmentation indexing, the word vector unit T1 maps the index of each smallest semantic unit to a dense vector of fixed dimension through learnable embedding matrix parameters to obtain word vector information C1; C2B, and at the same time, the word vector unit T1 after word segmentation indexing is encoded by position, and the index position of each smallest semantic unit is encoded to obtain position information P1; C3. Add the word vector information C1 and the position information P1 element by element to obtain the encoded feature Ft.

6. The method for enhancing the ability of a large language model in a table-based question-answering task according to claim 1, characterized in that: Step 2 specifically involves: 21) Using an unsupervised tabular dataset, the tabular dataset is input into the encoding and decoding parts of the TABLE-UAM network. The decoding part outputs the reconstructed tabular data, and the reconstructed tabular data is used for reconstruction difference training. 22) The trained TABLE-UAM network is concatenated with different large language models (LLMs). Using a supervised table task dataset, the question text and table data are input into the overall network model after the encoding part of the TABLE-UAM network is connected to different large language models (LLMs). The output is the predicted answer corresponding to each large language model (LLM). The weight parameters of the large language models (LLMs) are frozen. The encoding part of the TABLE-UAM network is trained under supervision based on each predicted answer and the corresponding label answer. This results in a retrained TABLE-UAM network and each trained large language model (LLM).

7. The method for enhancing the ability of a large language model for table-based question-answering tasks according to claim 6, characterized in that: The reconstruction differential training in step 21) specifically involves: The implicit table reconstruction loss L is constructed by using the reconstructed table Fde output by the decoding part of the TABLE-UAM network and the original input table. rec , means as follows: Where α represents the table content reconstruction loss weight coefficient, β represents the table structure corresponding loss coefficient, γ represents the column position preservation coefficient, M represents the total number of rows in the input table BT, N represents the total number of columns in the input table BT, i represents the row number of the input table BT, and j represents the column number of the input table B. L cell (BT ij, Fde ij ) represents the cell loss function, BT ij This represents the value at the corresponding position in the i-th row and j-th column of the input table, Fde ij This represents the value at the corresponding position in the i-th row and j-th column of the reconstructed table Fde; (BT) ij ) ka This indicates the input cell category value at the i-th row and j-th column of the table. (Fde) ij ) ka This represents the cell category value at the position corresponding to the i-th row and j-th column of the reconstructed table Fde; L j This indicates the number of characters in the j-th column name of the table, where colj represents the column name of the j-th column, and BT... （l） colj This indicates that the j-th column in table BT is being processed by the l-th character of the column name. The superscript l indicates the l-th element to be sorted in that column. Overall, it represents the l-th character element in the j-th column of the input table BT. （l） colj This represents the l-th element in the j-th column of the decode table Fde; ord() represents the ASCII code value of a character; π(j) represents the actual position index of the j-th column in the reconstructed table Fde; numeric, categorical, and boolean represent numeric, categorical, and boolean types, respectively. ka represents the ka-th category, and KA represents the total number of categories; II[BT ij ≠Fde ij The brackets [] represent an indicator function that returns 0 when the condition within the brackets is true and 1 when it is false. If the original input table BT in row i and column j is the same as the reconstructed table Fde in row i and column j, the function returns 1; otherwise, it returns 0. Then, the implicit reconstruction loss L is used in the table. rec The encoding and decoding parts of the TABLE-UAM network are optimized and trained with the goal of minimizing the impact of the network.

8. The method for enhancing the ability of a large language model in a table-based question-answering task according to claim 6, characterized in that: The supervised training in step 22) specifically includes: The encoding parts of the TABLE-UAM network trained using reconstructed differential models are concatenated with different large language models (LLMs), using the LLMs as the backbone models. All LLMs constitute the backbone model set, and the weight parameters of all LLMs in the backbone model set are frozen. Using a supervised table task dataset, question text and table data are input into the TABLE-UAM network, and the different LLMs output predicted answers. The following table task total loss L is established between each predicted answer and the labeled answer. task Supervised training of the weight parameters of the TABLE-UAM network: L task =δ(Σ K i=1 -((S T t=1 logP (t) Ali (A t |A <t , Q, BT)) / T)) / K +2θΣ K i=1 S K j=i+1 [1-Sim(ALi,ALj)] / (K(K-1)) Where, L task represents the total loss of the table task, that is, the difference between the predicted text answer output ALi of the i-th LLM and the label answer A. Sim(ALi, ALj) is the text similarity function, that is, the semantic similarity between the predicted text answer output ALi of the i-th LLM and the predicted text answer output ALj of the j-th LLM; K represents the total number of large language models LLM / backbone models, ALi represents the complete predicted answer sequence generated by the i-th large language model LLM, P (t) ALi represents the probability distribution output by the i-th large language model LLM at time step t; A represents the set of label answers, A = [A1, A2, ..., A T , T represents the sequence length of the label answer A, A T represents the T-th label answer; A<t represents all label sequences before time t, A<t = [A1, A2, ..., A t-1 , Q and BT respectively represent the input question and the data of the input table; δ is the coefficient that controls the average answer task loss; θ is the weight coefficient that controls the consistency constraint loss of the predicted answer.

Images