Table pre-training model training method and device, equipment and medium

Abstract

The present disclosure provides a table pre-training model training method and device, equipment and medium, relates to the technical field of computers, in particular to the technical field of artificial intelligence and deep learning. The specific implementation scheme is: the sample data included in the pre-training task is processed into a sample sequence, and the sample data includes a table; the embedding layer of the table pre-training model is used to embed and encode the position information of the sample sequence and each element in the sample sequence in the table, to obtain an element vector of each element in the sample sequence; the multi-head attention layer of the table pre-training model is used to calculate the attention result of each element vector in each group according to the grouping information of each element in the sample sequence, to obtain a representation vector of each element; the output result corresponding to the pre-training task is obtained based on the representation vector of each element, and the table pre-training model is trained based on the output result. In this way, the accuracy of the table pre-training model in understanding the table can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence and deep learning. Background Technology

[0002] There are numerous tables on web pages and in various industry documents. These tables contain massive amounts of data and have diverse structures. Currently, table understanding technology can be used to extract the required information from these tables. For example, the table can be expanded and represented as natural language. This natural language can be input into a text pre-trained model. Based on the text pre-trained model, a vector representation corresponding to the table can be obtained. Then, the required information can be extracted from the table using this vector representation. Summary of the Invention

[0003] This disclosure provides a training method, apparatus, device, and medium for a table pre-trained model.

[0004] According to one aspect of this disclosure, a method for training a table pre-trained model is provided, comprising:

[0005] The sample data included in the pre-training task is processed into a sample sequence, wherein the sample data includes a table;

[0006] The embedding layer of the pre-trained table model is used to embed and encode the sample sequence and the position information of each element in the sample sequence in the table, so as to obtain the element vector of each element in the sample sequence.

[0007] The multi-head attention layer of the pre-trained model uses the grouping information of each element in the sample sequence to calculate the attention result of each element vector in each group, so as to obtain the representation vector of each element.

[0008] The output result corresponding to the pre-training task is obtained based on the representation vector of each element, and the table pre-training model is trained based on the output result.

[0009] According to a second aspect of this disclosure, a training apparatus for a table pre-trained model is provided, comprising:

[0010] The processing module is used to process the sample data included in the pre-training task into a sample sequence, wherein the sample data includes a table;

[0011] The encoding module is used to embed and encode the sample sequence and the position information of each element in the sample sequence in the table through the embedding layer of the table pre-trained model, so as to obtain the element vector of each element in the sample sequence.

[0012] The calculation module is used to calculate the attention result of each element vector in each group based on the grouping information of each element in the sample sequence through the multi-head attention layer of the pre-trained model of the table, so as to obtain the representation vector of each element.

[0013] The training module is used to obtain the output result corresponding to the pre-training task based on the representation vector of each element, and to train the table pre-training model based on the output result.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a flowchart of the training method for the table pre-trained model provided in this embodiment of the disclosure;

[0023] Figure 2 This is an exemplary schematic diagram illustrating the embedding and encoding of a sample sequence provided in an embodiment of this disclosure;

[0024] Figure 3 This is an exemplary schematic diagram of a multi-head attention layer provided in an embodiment of this disclosure;

[0025] Figure 4 This is an exemplary schematic diagram of an encoder provided in an embodiment of this disclosure;

[0026] Figure 5 This is a schematic diagram of the structure of a training device for a table pre-training model provided in an embodiment of this disclosure;

[0027] Figure 6 This is a block diagram of an electronic device used to implement the training method of the table pre-trained model according to the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] Typically, text pre-trained models are trained on semantically coherent natural sentences. Therefore, text pre-trained models can understand the entire sentence and the words in the sentence based on the coherent context in the natural sentence.

[0030] As for tabular data, since there is no coherent context in the table, it is necessary to combine the structure of the table and the content of the cells to understand it, as shown in Table 1.

[0031] Table 1

[0032] 1 A First June 1 10 2 B Second June 1 10 3 C C June 1 10 4 D Man June 2 10 5 E E June 3 10

[0033] In a table, each cell is an independent unit. Understanding the table's content requires combining the content of each cell with its specific structure. For example, based on the structure of each row in Table 1 and the text content in the cells, we can deduce that: A borrowed A on June 1st, with a borrowing fee of 10 yuan; B borrowed B on June 1st, with a borrowing fee of 10 yuan; C borrowed C on June 1st, with a borrowing fee of 10 yuan; D borrowed D on June 2nd, with a borrowing fee of 10 yuan; and E borrowed E on June 3rd, with a borrowing fee of 10 yuan.

[0034] Furthermore, the same words may have different meanings in different tables, as shown in Tables 2 and 3.

[0035] Table 2

[0036] apple 4.5 yuan / 500g banana 2.5 yuan / 500g pitaya 6 yuan / 500g Green grapes 15 yuan / 500g

[0037] Table 3

[0038] apple 11999 yuan Computer A 4999 yuan Computer B 5999 yuan C computer 4999 yuan

[0039] Taking apples as an example, in Table 2, the cells in the same column as apples include bananas, dragon fruit, and green grapes. Based on the semantic tendencies of other words in the same column, we can understand that the semantic meaning of "apple" leans towards fruit. In Table 3, the products in the same column as apples are computer A, computer B, and computer C. Based on the semantic tendencies of other words in the same column, we can understand that the semantic meaning of "apple" leans towards electronic products. It is evident that understanding the content in a table depends on the table's structure.

[0040] To better understand tables, embodiments of this disclosure provide a method for training a pre-trained table model, which can be applied to electronic devices, such as... Figure 1 As shown, the method includes:

[0041] S101. Process the sample data included in the pre-training task into sample sequences, including tables.

[0042] The embodiments of this disclosure may include a variety of pre-training tasks, each pre-training task including a sample set, and each sample data in the sample set including a table.

[0043] S102. The embedding layer of the pre-trained table model is used to embed and encode the sample sequence and the position information of each element in the sample sequence in the table, so as to obtain the element vector of each element in the sample sequence.

[0044] Electronic devices can input sample sequences and the position information of each element in the sample sequence in a table into a table pre-training model, and then the embedding layer of the table pre-training model can output the element vector of each element.

[0045] S103. Using the multi-head attention layer of the pre-trained model in the table, the attention result of each element vector in each group is calculated based on the grouping information of each element in the sample sequence, so as to obtain the representation vector of each element.

[0046] Understandably, electronic devices also need to input the grouping information of each element in the sample sequence into a table for pre-training the model. Multi-head attention can calculate the attention result of each element vector in each group, and then, for each element, perform softmax processing on the multiple attention results corresponding to the element vector to obtain the representation vector of that element.

[0047] S104. Obtain the output result corresponding to the pre-training task based on the representation vector of each element, and train the table pre-training model based on the output result.

[0048] In this embodiment of the disclosure, the pre-training task can be a supervised task or an unsupervised task. The electronic device can train the table pre-training model based on the requirements of each pre-training task until the table pre-training model converges.

[0049] By employing the above technical solution, the embedding layer of the table pre-trained model can embed and encode the sample sequence and the position information of each element in the table, thereby enabling each element's vector to incorporate its position information within the table. Subsequent multi-head attention layers calculate the attention result of each element vector within each group, resulting in a representation vector for each element that also incorporates its position information. Furthermore, since the attention result calculation only requires comparing each element vector with other element vectors within the same group, rather than calculating the attention result between a single element vector and all element vectors, memory overhead is reduced, allowing the table pre-trained model to support full processing of the corresponding sample sequence. Thus, this method enables the training of a pre-trained model capable of incorporating element position information within the table for table understanding, thereby improving the accuracy of table comprehension.

[0050] The pre-training task in this embodiment can be a pre-training task with text interaction or a pre-training task without text interaction.

[0051] The sample data for the pre-training task with text interaction includes input text and tables, where the input text can be a query and / or table name semantic information.

[0052] The sample data for the pre-training task without text interaction includes tables but does not include input text.

[0053] In some embodiments of this disclosure, the sample sequence includes, in sequence, a first element sequence, a second element sequence, and a third element sequence corresponding to the table.

[0054] The first element in the first element sequence is the starting element of the text, and the subsequent elements are each character in the query statement.

[0055] The text begins with the element [@TEXT], which is a special token used to indicate that subsequent elements are text from the query statement. For example, if the query statement is "Class 2 Homework Task", then the first element sequence is "[@TEXT]Class 2 Homework Task".

[0056] Optionally, the length of the first element sequence is a preset length. If the sum of the lengths of the starting text element and the text in the query statement is less than the preset length, a specified padding character can be added to the first element sequence to complete it to the preset length. For example, the preset length can be 32 characters.

[0057] The first element in the second element sequence is the starting element of the table name semantics, and the subsequent elements are each character included in the table name semantic information.

[0058] The table name semantic starting element is [@CAPTION]. [@CAPTION] is a special token that represents the table name and is used to indicate that the subsequent elements are table name semantic information.

[0059] The first element in the third element sequence is the starting element of the table, and the subsequent elements are each character in each cell of each row of the table, with the first character element of each cell preceded by a cell starting element.

[0060] The table starts with [@TABLE], a special token used to represent a table. The cell starts with [@CELL], a special token used to represent a cell, serving as the starting token for a cell's text range.

[0061] When a table includes merged cells, the text element corresponding to the merged cell appears once in the third element sequence. For example, if a merged cell is located in rows 1 to 3 of the first column, then that merged cell is added as the first cell in the first row and first column to the third element sequence. When converting rows 2 and 3 of the table into sequences, this cell can be ignored. This avoids repeatedly inputting the content of merged cells into the table pre-trained model, enabling the model to handle complex table structures.

[0062] As an alternative implementation, the above sample sequence can be applied to both pre-training tasks with and without text interaction.

[0063] When the sample data includes query statements and table name semantic information, the query statements can be converted into a first element sequence and the table name semantic information can be converted into a second element sequence according to the format of the first element sequence and the second element sequence described in the above embodiments.

[0064] In the absence of a query statement in the sample data, all elements in the first element sequence except the first element are specified filler characters.

[0065] In the absence of table name semantic information in the sample data, all elements in the second element sequence except the first element are specified filler characters.

[0066] In this way, various pre-training tasks can use the same format of sample sequences, thereby enabling the trained table pre-training model to simultaneously support table comprehension tasks with and without text interaction, thus broadening the applicability of the table pre-training model.

[0067] As an example, the format of the sample sequence is: [CLS][@TEXT]Query...[PAD]...[PAD][@CAPTION]Caption...[PAD]...[PAD][@TABLE][@CELL]c1[@CELL]c2......[@CELL]cN.

[0068] Where [CLS] is the delimiter. “[@TEXT]Query...[PAD]...[PAD]” is the sequence of the first element, query is the query statement, and [PAD] is the specified padding character.

[0069] "[@CAPTION]Caption...[PAD]...[PAD]" is the second element sequence, where Caption is the semantic information of the table name.

[0070] “[@TABLE][@CELL]c1[@CELL]c2......[@CELL]cN” is the third element sequence, where c1, c2, ..., cN are the contents of N cells in the table.

[0071] The following example illustrates this point, assuming that the sample data includes Table 4 below.

[0072] Table 4

[0073]

[0074] In Table 4, each cell corresponds to an index, and the indexes corresponding to each cell in Table 4 are shown in Table 5.

[0075] Table 5

[0076] 6 7 8 9 10 11 6 7 12 13 14 15 6 7 16 17 18 19 6 7 20 21 22 23 6 7 24 25 26 27 6 7 28 29 30 31

[0077] As can be seen, the indices of the two merged cells in Table 4 are 6 and 7, respectively. The text content in the two merged cells will only appear once in the sample sequence and will not be repeated.

[0078] Suppose the pre-training task is a cloze test task, and the pre-training data includes a table and a query statement. The table is Table 4. The content of some cells in Table 4 can be covered (blank). Then, the content of one of the covered cells is used as the query statement. The semantics represented by this cell is used as the semantic information of the table name. The purpose is to enable the table pre-training model to determine which covered cell the query statement should be filled into.

[0079] In Table 4, assume that cells "07", "track", "With You", and "11" are covered. "07" is used as the query statement, and "Going to Tiger Mountain Again" corresponding to cell "07" is used as the semantic information of the table name.

[0080] Using the indexes shown in Table 5, the sample data can be converted into the following sample sequence:

[0081] [CLS][@TEXT]07[PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][@CAPTION] Heading Back to Tiger Mountain [PAD][PAD][PAD][PAD][PAD][PAD][PAD][@CAPTION] D][PAD][PAD][PAD][PAD][PAD][PAD][PAD][@TABLE][@CELL] Artist[@CELL] Release Date[@CELL] Serial Number[@CELL][@BLK][@CELL] Serial Number[@CELL] Track[@CELL] Zhang San[@CELL] December 1, 1982[@CELL] 01[@CELL] "Finding Your Dream Again"[@CELL] 02[@CELL] "The Sky is Vast"[@CELL] 03[@CELL] "The Wind Rises"[@CELL] 04[@CELL] "The Passing Water"[@CELL] 05[@CELL][@BLK][@CELL] 06[@CELL] "Untitled"[@CELL][@BLK][@CELL] "Going to Tiger Mountain Again"[@CELL] 08[@CELL] "Stay"[@CELL] 09[@CELL] L] "Parting Feelings" [@CELL] 10 [@CELL] "The Story of Green Grass" [@CELL][@BLK][@CELL] "Vast Waters" [@CELL] 12 [@CELL] "Spring on the Banks of the Hailan River".

[0082] The [@BLK] option is used to cover the text content of a cell.

[0083] The subsequent pre-trained table model can calculate the representation vector of each element in the sample sequence. The representation vector of [@CELL] can represent the semantic information of the entire cell and the position information of the cell, while the representation vector of [@TABLE] can represent the semantic information of the entire table.

[0084] By employing the embodiments of this disclosure, when the sample sequence includes a first element sequence, a second element sequence, and a third element sequence, the table pre-training model can output the representation vector of each element corresponding to the query statement, the representation vector of each element corresponding to the table name semantic information, and the representation vector of each element in the table. This allows the table pre-training model to flexibly support various downstream tasks. Furthermore, since the third element sequence also includes the table start element and the cell start element, the representation vectors of the table start element and the cell start element can also be obtained. This enables the table pre-training model to support element-level (i.e., token-level), cell-level, and table-level representation vectors. Moreover, based on the element-level representation vectors, row and column-level representation vectors can also be obtained. It is evident that the table pre-training model of this disclosure supports outputting representation vectors of multiple granularities, thereby adapting to more types of table understanding tasks, having a wide range of applications, and good versatility.

[0085] In some embodiments of this disclosure, when the pre-training tasks are all pre-training tasks without text interaction, the sample sequence is the third element sequence corresponding to the table.

[0086] The first element in the third element sequence is the starting element of the table, and the subsequent elements are each character in each cell of each row of the table, with the first character element of each cell preceded by a cell starting element.

[0087] For a detailed description of the third element sequence, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0088] In scenarios where there is no need to handle table comprehension tasks involving text interaction, the sample sequence can include only the sequence of the third element corresponding to the table, thus reducing unnecessary input information and saving computation.

[0089] Based on the above embodiments, S102, which involves embedding and encoding the sample sequence and the position information of each element in the sample sequence in the table through the embedding layer of the table pre-trained model to obtain the element vector of each element in the sample sequence, can be specifically implemented as follows:

[0090] Step 1: Encode each element of the sample sequence into a word vector through an embedding layer.

[0091] Step 2: Based on the position information of each element in the sample sequence in the table, the embedding layer encodes the position code, row code, and column code of each element.

[0092] Step 3: Based on the position of each element in the sample sequence, the embedding layer encodes the segment code of each element.

[0093] The sample sequence may include multiple fields, each corresponding to a segment code. For example, if the sample sequence includes a first element sequence, a second element sequence, and a third element sequence, then the first element sequence, the second element sequence, and the third element sequence each correspond to a field. The segment codes of the elements in the first element sequence can all be 0, the segment codes of the elements in the second element sequence can all be 1, and the segment codes of the elements in the third element sequence can all be 2.

[0094] Step 4: Based on the size order of each numeric element in the same row cell in the sample sequence, the embedding layer obtains the row order code of each numeric element, and based on the size order of each numeric element in the same column cell in the sample sequence, the column order code of each numeric element is obtained; wherein, the row order code and column order code of non-numeric elements are both 0.

[0095] For example, if a row contains two numeric elements, 100 and 200, they can be sorted in descending order and encoded by column order. For example, the column order code for 200 is 0, and the column order code for 100 is 1.

[0096] Steps 1 through 4 can be executed in parallel.

[0097] Step 5: For each element, the word vector, position code, segment code, row code, column code, row order code, and column order code of the element are added together through the embedding layer to obtain the element vector of the element.

[0098] The following example illustrates this, using sample data including Table 6 and the query statement "query?".

[0099] Table 6

[0100]

[0101]

[0102] Sample sequences generated based on sample data, such as Figure 2 As shown in the first line, each character (token) corresponds to an element in a sample sequence. Figure 2 For example only. Figure 2 Some elements have been omitted.

[0103] from Figure 2 As can be seen, "[CLS]" is encoded as 0 in the dimensions of position, segment, column, row, column-rank, and row-rank.

[0104] The encoding of "Query" at the position dimension is 1, which indicates that the element ranks first in the query statement. The encodings of "Query" at the segment, column, row, col-rank and row-rank dimensions are all 0.

[0105] The encoding of "?" at the position dimension is 2, which indicates that the element ranks second in the query statement. The encodings of "?" at the segment, column, row, col-rank and row-rank dimensions are all 0.

[0106] The encodings of "[@TABLE]" at the position, segment, column, row, col-rank and row-rank dimensions are all 0.

[0107] The encodings of "[@CELL]" at the position, col-rank and row-rank dimensions are all 0, and the encodings at the segment, column and row dimensions are all 1, which indicates that the [@CELL] represents the cell at the first row and first column in the table.

[0108] The encodings of "Ban" at the position, segment, column and row dimensions are all 1, which indicates that "Ban" is the first element in the cell at the first row and first column. Since "Ban" is not a number, the encodings at the col-rank and row-rank dimensions are both 0.

[0109] The value of "Zu" at the position dimension is 2, and the encodings at the segment, column and row dimensions are all 1, which indicates that "Zu" is the second element in the cell at the first row and first column. Since "Zu" is not a number, the values at the col-rank and row-rank dimensions are both 0.

[0110] Similarly, the encodings of the subsequent "[@CELL]" at the position, col-rank and row-rank dimensions are all 0, the values at the segment and row dimensions are both 1, and the encoding at the column dimension is 2.

[0111] The encodings of "Er" at the position, segment and row dimensions are all 1, the encoding at the column dimension is 2, and the encodings at the col-rank and row-rank dimensions are both 0.

[0112] The encoding of the Chinese character "Ban" in both the position dimension and the column dimension is 2, the encoding in both the segment dimension and the row dimension is 1, and the encoding in both the col-rank dimension and the row-rank dimension is 0.

[0113] The encoding of the Chinese character "Que" in both the position dimension and the segment dimension is 1, the encoding in the row dimension is 5, the encoding in the column dimension is 3, and the encoding in both the col-rank dimension and the row-rank dimension is 0.

[0114] The encoding of the Chinese character "Ren" in the position dimension is 2, the encoding in the segment dimension is 1, the encoding in the row dimension is 5, the encoding in the column dimension is 3, and the encoding in both the col-rank dimension and the row-rank dimension is 0.

[0115] The encoding of "[SEP]" in the position, column, col-rank and row-rank dimensions is 0, and the encoding in both the segment dimension and the row dimension is 1, where [SEP] is a separator.

[0116] Further, for each element, the encodings of the element in each dimension are added separately to obtain the element vector of each element.

[0117] By adopting this method, the structural features of the table are introduced through row encoding and column encoding, and the numerical sorting features of the table are introduced through row rank encoding and column rank encoding, enabling the table pre-training model to learn the structural features and numerical sorting features of the table, so as to understand the table more deeply and obtain the representation vector of the table more accurately.

[0118] In some embodiments of the present disclosure, the grouping information in the foregoing step S103 includes row groups, column groups and cell groups; each row group includes elements corresponding to cells in a same row in a sample sequence; each column group includes elements corresponding to cells in a same column in the sample sequence; each cell group includes elements corresponding to a same cell in the sample sequence.

[0119] Wherein, each row group may specifically include numbers of elements corresponding to cells in a same row in the sample sequence. For example, if the 5th to 10th elements in the sample sequence correspond to a same row in the table, then the row group includes numbers 5 to 10.

[0120] Similarly, each column group may specifically include numbers of elements corresponding to cells in a same column in the sample sequence, and each cell group may specifically include numbers of elements corresponding to a same cell in the sample sequence.

[0121] Based on this, in S103 above, the multi-head attention layer of the pre-trained table model calculates the attention result of each element vector in each group based on the grouping information of each element in the sample sequence, including:

[0122] For each element in the same row group, the attention result between that element and each element corresponding to the row group information is calculated separately through a multi-head attention layer.

[0123] For each element in the same column group, the attention result between that element and each element corresponding to that group information is calculated separately through a multi-head attention layer.

[0124] For each element in the same cell group, the attention result between that element and each element corresponding to the group information is calculated separately using a multi-head attention layer.

[0125] This method can calculate attention results between elements that are semantically related to each other, and calculate attention results between elements that are not semantically related. This can reduce the space complexity of the multi-head attention layer and increase the table length that can be supported.

[0126] In another embodiment, the sample data also includes query statements and / or table name semantic information; correspondingly, the grouping information also includes non-table groups, which include elements in the first element sequence corresponding to the query statement and / or the second element sequence corresponding to the table name semantic information.

[0127] Based on this, in S103 above, the multi-head attention layer of the pre-trained table model calculates the attention result of each element vector in each group based on the grouping information of each element in the sample sequence. In addition to calculating the three attention results described in the above embodiments, this also includes:

[0128] For each element included in the non-table group, the attention result between that element and each element in the sample sequence is calculated through a multi-head attention layer.

[0129] In other words, for each element corresponding to the query statement, the attention result between that element and each element in the sample sequence can be calculated to obtain the semantic association between each element in the query statement and the entire sample sequence.

[0130] For each element corresponding to the semantic information of the table name, the attention result between that element and each element in the sample sequence can be calculated to obtain the semantic association between each element in the semantic information of the table name and the entire sample sequence.

[0131] This method is equivalent to introducing a sparse self-attention mechanism for row, column, cell grouping, and non-table grouping. Attention calculations for these four groupings can be performed in parallel, and there's no need to calculate attention results between elements not belonging to the same group, reducing space complexity and thus memory overhead. Experiments show that this method can increase the maximum number of supported input elements from 512 to 4096, enabling the understanding of longer tables.

[0132] As an example, taking the sample sequence corresponding to Table 6 above, the grouping of elements in this sample sequence is as follows: Figure 3 As shown, Figure 3 The left image shows an example of row grouping (also known as row heads in a multi-head row and column attention layer), and the right image shows an example of column grouping (also known as column heads in a multi-head row and column attention layer). Figure 3 Elements corresponding to the same filled circle belong to the same group. For example... Figure 3 In the left image, elements corresponding to solid circles belong to the same group, and elements corresponding to vertical lines filled with circles belong to the same group. Figure 3 The “X12” indicates that it can be processed through 12 layers of multi-head row and column attention layers.

[0133] like Figure 4 As shown, Figure 4 This is an exemplary schematic diagram of the encoder in the table pre-trained model provided in this disclosure embodiment. The encoder includes a multi-head attention layer, a residual connection and normalization network (Add&Norm), and a feedforward network. Specifically, the multi-head attention layer can be a multi-head row and column attention layer. Figure 4 An example is shown of a multi-head row and column attention layer. In actual implementation, multiple multi-head attention layers can be included. The output of one multi-head attention layer can be used as the input of the next multi-head attention layer, thereby enabling iterative attention calculation through multiple multi-head attention layers to obtain a more accurate representation vector.

[0134] As an example, the encoder may include 12 multi-head attention layers.

[0135] Residual connections are used to address the vanishing gradient problem that occurs as the number of network layers in a model increases. Normalization networks are used to ensure that the calculated features have similar scales, thereby accelerating model convergence. Feedforward networks are a type of neural network model that achieves a complex mapping from input to output through multiple composites of simple nonlinear functions.

[0136] In some embodiments of this application, the pre-training task may include element-level tasks, cell-level tasks, row-column-level tasks, and table-level tasks.

[0137] Element-level tasks are used to predict the text contained in a specified position of a specified cell in a table based on a pre-trained model.

[0138] Cell-level tasks are used to make at least one of the following predictions based on a pre-trained model: predicting cells related to a query statement, predicting text included in a specified cell, predicting cell type, predicting relationships between cells, and predicting erroneous cells included in a table.

[0139] Table-level tasks are used to predict whether a table is related to a query statement and / or predict the table type based on a pre-trained model. Thus, the table pre-trained model obtained based on the above pre-training tasks can support representation vectors of different granularities. Therefore, the table pre-trained model trained by this method is applicable to table understanding tasks at the element level, cell level, row and column level, and table level. Compared with existing technologies, which typically support only one table understanding task, the table pre-trained model trained by this method has wider applicability.

[0140] As an example, in this embodiment of the disclosure, the electronic device can construct 11 pre-training tasks based on tabular data extracted from network data. These include 6 supervised pre-training tasks and 5 self-supervised pre-training tasks.

[0141] In scenarios involving text interaction, four pre-training tasks are constructed based on tabular data extracted from network data, including:

[0142] The first type of pre-training task is the Cell Retrieval task, which is a supervised pre-training task.

[0143] The specific task is as follows: The table pre-trained model finds the answer cells in the table based on the given input text (query). The input data of the table pre-trained model for this task comes from the query and the table. The labeled cells in the table are the answer cells corresponding to the query. The output of this task is the probability that each cell in the table is an answer cell.

[0144] The purpose of choosing this task is to improve the ability of a pre-trained table model to represent cells at a granular level given a query.

[0145] The electronic device uses the probability that each cell output by the output layer after the table pre-training model is the answer cell and the labeled cells input into the table pre-training model to calculate the loss function, and then adjusts the parameters in the table pre-training model until the table pre-training model converges.

[0146] For example, taking the sample data including Table 7 as an example, the input query can be: Class 2 homework task, the corresponding labeled data is the "road leveling" cell, that is, the "road leveling" cell is the answer cell.

[0147] Table 7

[0148]

[0149] The second type of pre-training task is the Question Parsing task, which is a supervised pre-training task.

[0150] The task is as follows: The table pre-trained model finds the cells in the table that are related to the given input query. The input for this task is the source of the input data, which includes the query and the table. The labeled cells in the table are the cells that are related to the query. The output for this task is the probability that each cell is related to the query.

[0151] The purpose of choosing this task is to improve the ability of a pre-trained table model to represent cells at a granular level given a query.

[0152] Electronic devices can use the probability of each cell output by the output layer after the table pre-training model being related to the input query, as well as the actual labeled cells, to calculate the loss function, and then adjust the parameters in the table pre-training model until the table pre-training model converges.

[0153] For example, taking the sample data including Table 8 as an example, the input query is: "Second Shift Homework Task". The corresponding labeled cells are the "Second Shift" cell and the "Homework Task" cell.

[0154] Table 8

[0155]

[0156] The third type of pre-training task is the table retrieval task, which is a supervised pre-training task.

[0157] The specific task is as follows: A table pre-trained model, given a query as input, determines whether a table is related to the query. The input data for this table pre-trained model comes from the query and tables, with the labeled tables being those related to the query. The output of this task is the probability that a table is related to the query.

[0158] The purpose of choosing this task is to improve the representational ability of table pre-trained models at the table granularity. Table pre-trained models can map queries and query-related tables into the same semantic space. Thus, when applying table pre-trained models to downstream tasks involving table-text joint modeling, content transfer between tables and text can be completed accurately and quickly.

[0159] Electronic devices use the output layer of the table pre-trained model to calculate the probability of the table and query, as well as the labeled table, to calculate the loss function and adjust the parameters in the table pre-trained model until the table pre-trained model converges.

[0160] For example, taking the sample data including Table 9 as an example, the input query is: "Who is the person in charge of the second shift on January 14, 2022? What is the task?" The corresponding label is that the table is related to the query.

[0161] Table 9

[0162]

[0163] The fourth type of pre-training task is the cell-level cloze task, which is a self-supervised pre-training task.

[0164] The task is as follows: obscure N cells in a table, use the text in the N obscure cells as candidate text, use one of the candidate texts as the query statement, input the table and the query statement into the table pre-trained model in the form of a sequence, and the output of this task is the probability that the query statement should be filled in each obscure cell.

[0165] The purpose of this task is to enable the table pre-trained model to select appropriate candidate text for the occluded cell based on information from other cells in the table, and to learn the joint text distribution between cells and capture the relationships between cells.

[0166] For example, taking Table 10 as a sample data, the text content "work location" can be used as the query statement, and the "work location" cell and the flat cell "road" in Table 10 can be covered up as sample data.

[0167] Table 10

[0168]

[0169] In addition, in scenarios without text interaction, seven pre-training tasks can be constructed, including:

[0170] The first type of pre-training task is the Cell Type Classification task, which is a supervised pre-training task.

[0171] The specific task is: a table pre-trained model predicts cell types. The sample data for the table pre-trained model for this task consists of a table and the cell types of labeled cells. The output of this task is the probability that a labeled cell belongs to a certain cell type.

[0172] The purpose of choosing this task is to enable the table pre-trained model to build a mapping from cell structure features and text features to type in cells, which is helpful for downstream task transfer.

[0173] For example, taking the sample data including Table 11 as an example, Table 11 includes the following cell types: table header, table value, and useless information. Among them, the work location cell and the road leveling cell can be used as label cells. The cell type of the work location cell is table header, and the cell type of the road leveling cell is table value.

[0174] Table 11

[0175]

[0176] The second type of pre-training task is the Cell Pair Relation Classification task, which is a supervised pre-training task.

[0177] The specific task is: a table pre-trained model predicts the relationships between cells. The sample data for the table pre-trained model is the relationship type between the table and the labeled cells. The output of this task is the probability that the relationship type between the labeled cells belongs to a certain relationship type.

[0178] The purpose of choosing this task is to improve the understanding of cell relationships in the pre-trained table model at the cell-level data dimension.

[0179] For example, sample data may include Table 12, and cell relationships may include: table header and sub-header, table header and table value (which are irrelevant). The start time cell and the 2022.01.05 cell are label cells, and the label cells have a relationship type of table header and table value.

[0180] Table 12

[0181]

[0182]

[0183] The third type of pre-training task is the Corrupt Cell Detection task, which is a self-supervised pre-training task.

[0184] The task is as follows: Select a subset of cells, modify or swap the text in the selected cells, input the table as a sequence into a pre-trained table model, and the pre-trained table model predicts whether there are errors in the text content of the cells in the table. The output of this task is the probability of errors in the text content of each cell.

[0185] The purpose of this task is to improve the ability of pre-trained table models to identify partial and table cell distributions at the cell level, and to enhance the semantic understanding of table data distributions in pre-trained table models.

[0186] For example, the sample data may include Table 13, in which the text content "Work Location" in the Work Location cell is swapped with the text content "Road Leveling" in the Road Leveling cell.

[0187] Table 13

[0188]

[0189] The fourth type of pre-training task is the table type classification task, which is a supervised pre-training task.

[0190] The specific task is: to use a pre-trained table model to predict the table type. The sample data for this task includes labeled tables and the table types of the labeled tables. The output of this task is the probability that the labeled table type belongs to a certain table type.

[0191] The purpose of choosing this task is to improve the ability of pre-trained table models to represent tables at the table-level data dimension.

[0192] For example, table types include row tables, list tables, discrete tables, and stacked tables. The label table in the input sequence is shown in Table 14, where all cells are label cells, and Table 14 is a row table.

[0193] Table 14

[0194]

[0195] The fifth type of pre-training task is the Masked Language Model (MLM) task, which is a self-supervised pre-training task.

[0196] The task is as follows: a certain number of characters (tokens) are obscured, and a pre-trained table model predicts the obscured tokens using other tokens. The sample data for this task is a table with a certain number of obscured tokens, and the output of this task is the probability that the obscured token in the table is a certain token.

[0197] The purpose of choosing this task is to enable the table pre-trained model to learn the text distribution within and between cells in the table, thereby improving its ability to understand the text in the table cells.

[0198] For example, Table 15 shows a table of a certain number of occluded tokens in the input sequence, where [M] represents the occluded tokens:

[0199] Table 15

[0200]

[0201] Taking Table 15 as an example, the output corresponding to this task is that the probability of [M] in Road C of District [M] in City A being B is 90%, and the probability of it being C is 10%. Therefore, B, which has the highest probability, can be selected as the final prediction result corresponding to this task.

[0202] The sixth type of pre-training task is the Whole Cell Masking task, which is a self-supervised pre-training task.

[0203] The task is as follows: Select a certain number of cells and cover all the tokens in the selected cells. The table pre-trained model predicts the covered tokens using other tokens. The sample data for this task is a table in which all the tokens in a certain proportion of the cells are covered. The output of this task is the probability that all the tokens in each covered cell are a certain string of tokens.

[0204] The purpose of this task is to enable the pre-trained table model to predict the text content of the occluded cell using information from other cells in the table, and to learn the joint text distribution and relationships between cells.

[0205] For example, the sample data is shown in Table 16, where [M] represents the occluded token.

[0206] Table 16

[0207]

[0208] Taking Table 16 as an example, for the [M][M][M][M] to the left of the second shift, the output of the task is that the probability of [M][M][M][M] being the work shift is 70%, and the probability of being the work group is 30%. Therefore, the work shift with the highest probability can be selected as the final prediction result for the task.

[0209] The seventh type of pre-training task is the Cell Value Recovery task, which is a self-supervised pre-training task.

[0210] The specific task is as follows: A pre-trained table model predicts the Nth token in a cell based on the cell representation and position embedding. The sample data for this task includes tables, and the output is the probability that the Nth token in a given cell is a specified token.

[0211] The purpose of choosing this task is to ensure that after the table pre-trained model performs complex interactive calculations on the elements in the sample sequence, the cell representation vector can still represent the semantic information of the element, which can improve the relevance between the cell representation and the token included in the cell, and enable the table pre-trained model to distinguish between literally similar text.

[0212] It should be noted that the table pre-trained model in this embodiment is used to represent the sample sequence as a vector. During the training process, different decoding layers and output layers need to be connected after the table pre-trained model based on each pre-training task.

[0213] After training the table pre-trained model, its ability to represent tables and cells can be further verified. As an example, based on business scenarios that may occur in real-world applications, this embodiment of the disclosure can select five verification tasks to verify the table pre-trained model.

[0214] It should be noted that the table pre-training model in this embodiment is used to represent the sample sequence as a vector. During the training process, different decoding layers and output layers need to be connected after the table pre-training model based on the characteristics of each pre-training task in order to complete each pre-training task.

[0215] In this embodiment, the tables used for training the table pre-training model can be constructed based on data from publicly available web pages. For example, based on encyclopedia data and publicly available data from various industries, 1.82M relationship tables, 0.68M entity tables, 0.62M stacked entity tables, and 0.67M stacked relationship tables can be constructed. This allows for the use of structurally rich and multi-industry table data to train the table pre-training model, thus broadening its applicability.

[0216] As an example, you can choose from the following tasks: Cell Retrieval, Table Retrieval, Cell Type Classification, Cell Pair Relation Classification, and Table Type Classification. The sample data for the first four tasks comes from actual business data, while the data for the last verification task can come from academia.

[0217] The specific content of the above five verification tasks is the same as the corresponding pre-training tasks in the above embodiments. Please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0218] Five validation tasks can be used to reflect the pre-trained table model's ability to understand tables and the benefits that the pre-trained table model can bring to downstream business.

[0219] As shown in Table 17, it has been verified that in the cell type classification task, the actual business benefits of the table pre-trained model Base-v1.0 are improved by 3.3% compared with the traditional model base ERNIE-3.0.

[0220] In the cell-to-relationship classification task, based on the same dataset, the actual business benefit of the pre-trained table model Base-v1.0 is 6.8% higher than that of the traditional model base ERNIE-1.0.

[0221] In the table retrieval task, based on the same dataset, the actual business benefits of the pre-trained table model Base-v1.0 are improved by 4.6% compared to the traditional model base ERNIE-1.0.

[0222] In the table type classification task, based on the same dataset, the actual business benefit of the table pre-trained model Base-v1.0 is improved by 8.7% compared with the traditional model base ERNIE-2.0. This shows that the method provided in this disclosure can improve the pre-trained model's ability to represent tables.

[0223] Table 17

[0224]

[0225]

[0226] The collection, storage, use, processing, transmission, provision, and disclosure of sample data involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0227] It should be noted that the sample data in this embodiment comes from a publicly available dataset.

[0228] Based on the same concept, embodiments of this disclosure provide a training apparatus for a table pre-trained model, such as... Figure 5 As shown, it includes:

[0229] The processing module 501 is used to process the sample data included in the pre-training task into a sample sequence, wherein the sample data includes a table.

[0230] The encoding module 502 is used to embed and encode the sample sequence and the position information of each element in the sample sequence in the table through the embedding layer of the table pre-trained model, so as to obtain the element vector of each element in the sample sequence.

[0231] The calculation module 503 is used to calculate the attention result of each element vector in each group based on the grouping information of each element in the sample sequence through the multi-head attention layer of the pre-trained model of the table, so as to obtain the representation vector of each element.

[0232] The training module 504 is used to obtain the output result corresponding to the pre-training task based on the representation vector of each element, and to train the table pre-training model based on the output result.

[0233] Optionally, the sample data may also include query statements and / or table name semantic information; the grouping information may also include non-table groupings, which include elements in the first element sequence corresponding to the query statement and / or the second element sequence corresponding to the table name semantic information.

[0234] Calculation module 503 is also used for:

[0235] For each element included in the non-table group, the attention result between that element and each element in the sample sequence is calculated through a multi-head attention layer.

[0236] Optionally, the sample sequence is the sequence of the third element corresponding to the table.

[0237] The first element in the third element sequence is the starting element of the table, and the subsequent elements are each character in each cell of each row of the table, with the first character element of each cell preceded by a cell starting element.

[0238] Optionally, the sample sequence includes, in sequence, the first element sequence, the second element sequence, and the third element sequence corresponding to the table.

[0239] The first element in the first element sequence is the starting element of the text, and the subsequent elements are each character in the query statement.

[0240] The first element in the second element sequence is the starting element of the table name semantics, and the subsequent elements are each character included in the table name semantic information.

[0241] The first element in the third element sequence is the starting element of the table, and the subsequent elements are each character in each cell of each row of the table, with the first character element of each cell preceded by a cell starting element.

[0242] Optional, encoding module 502, specifically used for:

[0243] Each element in the sample sequence is encoded as a word vector through an embedding layer.

[0244] The embedding layer encodes each element's position, row, and column codes based on the position information of each element in the table within the sample sequence.

[0245] The segment code for each element is obtained by encoding it based on the position of each element in the sample sequence through the embedding layer.

[0246] The embedding layer obtains the row order code for each numeric element based on its size order in the same row cell of the sample sequence, and the column order code for each numeric element based on its size order in the same column cell of the sample sequence; where the row order code and column order code for non-numeric elements are both 0.

[0247] For each element, the word vector, position code, segment code, row code, column code, row order code, and column order code of that element are added together through the embedding layer to obtain the element vector of that element.

[0248] Optionally, if the sample data does not include a query statement, all elements in the first element sequence except the first element are specified filler characters.

[0249] In the absence of table name semantic information in the sample data, all elements in the second element sequence except the first element are specified filler characters.

[0250] Optionally, if the table includes merged cells, the text element corresponding to the merged cell appears once in the third element sequence.

[0251] Optionally, pre-training tasks include element-level tasks, cell-level tasks, row-column-level tasks, and table-level tasks.

[0252] Element-level tasks are used to predict the text contained in a specified position of a specified cell in a table based on a pre-trained model.

[0253] Cell-level tasks are used to make at least one of the following predictions based on a pre-trained model: predicting cells related to a query statement, predicting text included in a specified cell, predicting cell type, predicting relationships between cells, and predicting erroneous cells included in a table.

[0254] Table-level tasks are used to predict whether a table is relevant to a query statement and / or to predict the table type based on a pre-trained model.

[0255] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0256] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0257] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0258] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0259] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the training method for a table pre-trained model. For example, in some embodiments, the training method for a table pre-trained model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the training method for a table pre-trained model described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for a table pre-trained model.

[0260] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0261] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0262] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0263] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0264] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0265] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0266] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0267] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a table pre-trained model, comprising: The sample data included in the pre-training task is processed into a sample sequence, wherein the sample data includes a table; Each element of the sample sequence is encoded as a word vector through an embedding layer; The embedding layer encodes each element's position, row, and column codes based on the position information of each element in the sample sequence within the table. When the element belongs to the table, the position code describes the element's position within a cell of the table. The embedding layer encodes each element based on its position in the sample sequence to obtain a segment code. The embedding layer obtains the row order code of each numeric element based on its size order in the same row cell of the sample sequence, and obtains the column order code of each numeric element based on its size order in the same column cell of the sample sequence; wherein, the row order code and column order code of non-numeric elements are both 0. For each element, the word vector, position code, segment code, row code, column code, row order code, and column order code of the element are added together through the embedding layer to obtain the element vector of the element; the multi-head attention layer of the table pre-trained model calculates the attention result of each element vector in each group based on the grouping information of each element in the sample sequence to obtain the representation vector of each element. The output result corresponding to the pre-training task is obtained based on the representation vector of each element, and the table pre-training model is trained based on the output result.

2. The method according to claim 1, wherein, The grouping information includes row grouping, column grouping, and cell grouping; each row group includes the element corresponding to the cell in the same row in the sample sequence; each column group includes the element corresponding to the cell in the same column in the sample sequence; each cell group includes the element corresponding to the cell in the same cell in the sample sequence. The multi-head attention layer of the pre-trained model using the table calculates the attention result of each element vector in each group based on the grouping information of each element in the sample sequence, including: For each element in the same row group, the attention result between the element and each element corresponding to the row group information is calculated by the multi-head attention layer; For each element in the same column group, the attention result between the element and each element corresponding to the group information is calculated by the multi-head attention layer; For each element in the same cell group, the attention result between that element and each element corresponding to the group information is calculated through the multi-head attention layer.

3. The method according to claim 2, wherein, The sample data also includes query statements and / or table name semantic information; the grouping information also includes non-table groups, which include elements in the first element sequence corresponding to the query statement and / or the second element sequence corresponding to the table name semantic information. The multi-head attention layer of the pre-trained model using the table, calculating the attention result of each element vector in each group based on the grouping information of each element in the sample sequence, further includes: For each element included in the non-table grouping, the attention result between that element and each element in the sample sequence is calculated through the multi-head attention layer.

4. The method according to claim 2, wherein, The sample sequence is the third element sequence corresponding to the table; The first element in the third element sequence is the table start element, and the subsequent elements are each character in each cell of each row of the table, with a cell start element preceding the first character element of each cell.

5. The method according to claim 3, wherein, The sample sequence includes, in sequence, the first element sequence, the second element sequence, and the third element sequence corresponding to the table; The first element in the first element sequence is the text start element, and the subsequent elements are each character in the query statement in sequence. The first element in the second element sequence is the starting element of the table name semantics, and the subsequent elements are each character included in the table name semantics information in sequence; The first element in the third element sequence is the table start element, and the subsequent elements are each character in each cell of each row of the table, with a cell start element preceding the first character element of each cell.

6. The method according to claim 5, wherein, If the query statement is not included in the sample data, all elements in the first element sequence except the first element are specified filler characters; In the absence of the table name semantic information in the sample data, all elements in the second element sequence except the first element are specified filler characters.

7. The method according to claim 4 or 5, wherein, In the case where the table includes merged cells, the text element corresponding to the merged cell appears once in the third element sequence.

8. The method according to claim 1, wherein, The pre-training tasks include element-level tasks, cell-level tasks, row-column-level tasks, and table-level tasks. The element-level task is used to predict the text contained in a specified position of a specified cell in a table based on the pre-trained model; The cell-level task is used to make at least one of the following predictions based on the pre-trained model: predicting cells related to the query statement, predicting the text included in a specified cell, predicting the cell type, predicting the relationship between cells, and predicting erroneous cells included in the table. The table-level task is used to predict whether a table is related to a query statement and / or to predict the table type based on the pre-trained model.

9. A training apparatus for a table-based pre-trained model, comprising: The processing module is used to process the sample data included in the pre-training task into a sample sequence, wherein the sample data includes a table; The encoding module is used to encode each element of the sample sequence into a word vector through an embedding layer; to encode each element into a position code, row code, and column code based on the position information of each element in the sample sequence in the table through the embedding layer; and to encode each element into a segment code based on the position of each element in the sample sequence through the embedding layer. The embedding layer obtains the row order code for each numeric element based on its size order within the same row of cells in the sample sequence, and the column order code for each numeric element based on its size order within the same column of cells in the sample sequence; wherein, the row order code and column order code for non-numeric elements are both 0; for each element, the embedding layer adds the word vector, position code, segment code, row code, column code, row order code, and column order code of the element to obtain the element vector; when the element belongs to a table, the position code is used to describe the position of the element in the cell of the table; The calculation module is used to calculate the attention result of each element vector in each group based on the grouping information of each element in the sample sequence through the multi-head attention layer of the pre-trained model of the table, so as to obtain the representation vector of each element. The training module is used to obtain the output result corresponding to the pre-training task based on the representation vector of each element, and to train the table pre-training model based on the output result.

10. The apparatus according to claim 9, wherein, The grouping information includes row grouping, column grouping, and cell grouping; each row group includes the element corresponding to the cell in the same row in the sample sequence; each column group includes the element corresponding to the cell in the same column in the sample sequence; each cell group includes the element corresponding to the cell in the same cell in the sample sequence. The computing module is specifically used for: For each element in the same row group, the attention result between the element and each element corresponding to the row group information is calculated by the multi-head attention layer; For each element in the same column group, the attention result between the element and each element corresponding to the group information is calculated by the multi-head attention layer; For each element in the same cell group, the attention result between that element and each element corresponding to the group information is calculated through the multi-head attention layer.

11. The apparatus according to claim 10, wherein, The sample data also includes query statements and / or table name semantic information; the grouping information also includes non-table groups, which include elements in the first element sequence corresponding to the query statement and / or the second element sequence corresponding to the table name semantic information. The computing module is also used for: For each element included in the non-table grouping, the attention result between that element and each element in the sample sequence is calculated through the multi-head attention layer.

12. The apparatus according to claim 10, wherein, The sample sequence is the third element sequence corresponding to the table; The first element in the third element sequence is the table start element, and the subsequent elements are each character in each cell of each row of the table, with a cell start element preceding the first character element of each cell.

13. The apparatus according to claim 11, wherein, The sample sequence includes, in sequence, the first element sequence, the second element sequence, and the third element sequence corresponding to the table; The first element in the first element sequence is the text start element, and the subsequent elements are each character in the query statement in sequence. The first element in the second element sequence is the starting element of the table name semantics, and the subsequent elements are each character included in the table name semantics information in sequence; The first element in the third element sequence is the table start element, and the subsequent elements are each character in each cell of each row of the table, with a cell start element preceding the first character element of each cell.

14. The apparatus according to claim 13, wherein, If the query statement is not included in the sample data, all elements in the first element sequence except the first element are specified filler characters; In the absence of the table name semantic information in the sample data, all elements in the second element sequence except the first element are specified filler characters.

15. The apparatus according to claim 12 or 13, wherein, In the case where the table includes merged cells, the text element corresponding to the merged cell appears once in the third element sequence.

16. The apparatus according to claim 9, wherein, The pre-training tasks include element-level tasks, cell-level tasks, row-column-level tasks, and table-level tasks. The element-level task is used to predict the text contained in a specified position of a specified cell in a table based on the pre-trained model; The cell-level task is used to make at least one of the following predictions based on the pre-trained model: predicting cells related to the query statement, predicting the text included in a specified cell, predicting the cell type, predicting the relationship between cells, and predicting erroneous cells included in the table. The table-level task is used to predict whether a table is related to a query statement and / or to predict the table type based on the pre-trained model.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images