Joint text enhancement based table entity and type annotation method using graph convolutional network

Abstract

This invention discloses a joint text enhancement method for table entity and type annotation based on graph convolutional networks. The steps of this invention are: 1. Adaptive extraction and preprocessing of entity-related definitional text data; 2. Modeling the transformation of table data sets, annotation data sets, and text data sets into graph structure data; 3. Feature vectorization representation: using BERT to perform feature vectorization representation based on the text information of graph vertices, transforming it into the feature matrix required by the graph convolutional network model, and establishing the adjacency matrix required by the model based on the edge set; 4. Parallel multi-task learning and result prediction. This invention includes text extraction and preprocessing, table-to-graph structure modeling, and multi-task learning, which can improve the model's ability to parse the structure and semantics of tables, enhance robustness, and improve the model's predictive ability. Simultaneously, it can introduce entity-related text data easily obtained from knowledge bases without relying on meta-information, and its performance is significantly better than existing models.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and deep learning network models, specifically to a method for joint text enhancement of table entities and type annotations based on graph convolutional networks. Background Technology

[0002] Entity annotation and type annotation of tabular data are two subtasks of the task of matching tabular data to knowledge graphs. This task involves assigning semantic labels from a knowledge graph (such as Wikidata or DBpedia) to tabular elements. Specifically, entity annotation adds entity annotations to cells within a table that relate to a specific entity, while type annotation adds type annotations to columns within a table that relate to a specific type. In recent years, with the rapid development of the internet and the arrival of the big data era, tabular datasets have become increasingly prevalent online. Extracting valuable information from these complex and diverse datasets has become a research hotspot in data mining, data analysis, machine learning, and knowledge discovery. The sheer size of these datasets also renders traditional probabilistic inference models that rely on manually constructed features inapplicable, posing a significant challenge to this task.

[0003] With the popularity of deep learning-based neural network models in the field of natural language processing, research on the application of deep neural network models to table datasets has become a hot topic. Currently, the mainstream annotation methods for table datasets using neural network models can be divided into two types based on whether they adopt the pre-training-fine-tuning paradigm (pre-training on a large corpus first, and then achieving the best accuracy with only a small number of training sessions on the target training set). The representative methods are TURL and TCN, respectively. Both methods use BERT to vectorize the information within the table. The difference is that TURL inputs the table meta-information (table name, topic, etc.) together with the information within the table into the Transformer. It establishes a visibility matrix between different types of information so that the attention mechanism can calculate the attention vectors of different types of information separately. It also proposes a pre-training method for masked entity recovery based on the masked language model. (2) TCN designed a table convolutional network model and proposed an attention mechanism that can aggregate relevant information within and between tables. Both methods have achieved better results on the table type annotation task. However, TURL encodes table content column-wise, ignoring inter-row relationships, and therefore cannot fully capture the table's structural information. Furthermore, both methods rely on table metadata; due to inconsistent network data quality and transmission errors caused by network fluctuations, table data obtained from the network often suffers from missing metadata, significantly impacting model performance. Additionally, table information is mostly presented as text strings, and existing methods fail to recognize the role of joint text in enhancing the model's understanding of tables. Moreover, both methods only consider type annotations for table columns, failing to address entity annotation of table cells, thus severing the connection between the two tasks.

[0004] Therefore, this invention aims to address the problems of existing deep learning models in table annotation tasks, such as over-reliance on meta-information, inability to fully acquire structural information, and failure to consider the relationship between tables and text. It introduces entity-related text from a knowledge base that is easily accessible to help the model better understand the semantic and structural information of tables. The invention proposes a table entity and type annotation method that can overcome the problem of missing meta-information, enhance the model's table understanding ability through joint text, capture complete table structural information based on graph convolutional networks, and adopt parallel multi-task learning. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a joint text enhancement method for table entity and type annotation based on graph convolutional networks. This multi-task learning method for joint text enhancement based on graph convolutional networks improves the robustness and predictive ability of the model through parallel multi-task learning.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] Step 1: Adaptive extraction and preprocessing of entity-related definitional text data;

[0008] This invention designs an adaptive method for extracting entity-related definitional text from a knowledge base and preprocessing it. The text defines the corresponding entity from the perspective of its type, thus improving the model's understanding of the table dataset. This invention uses an adaptive method to extract text data related to all entities in each table from the knowledge base. Open information operations and part-of-speech tagging are then performed to filter out invalid text containing only pronouns or stop words as subjects and objects. The extraction results (text data after filtering out invalid text) are stored in the order of the original table content. Adaptability solves the text extraction problem for tables without entity annotations, and for tables with entity annotations, the method of determining whether to directly retrieve all annotated entity names based on the number of entity annotations speeds up text extraction, improving the efficiency of the method while ensuring a certain amount of meaningful text.

[0009] Input: A set of table data T = {τ} 1 , τ 2 ,,...,,τ n Each table contains multiple columns. Each column contains information from several cells. Entity annotation set corresponding to each table (Entity comments for some tables may be empty sets) ).

[0010] 1-1. Adaptive Candidate Set Selection Method; Adaptive means for cases without entity annotation information. or number of entity annotations The number of non-empty cells in the primary key (PK) column of the i-th table does not exceed the number of cells in the primary key column. Half of the table, A candidate set is constructed by semantically filtering all cell information. Conversely, all annotated entity names in that table will be used as the candidate set.

[0011] 1-2. Text extraction from the knowledge base: Each element in the two candidate sets is used as a keyword. Text related to the elements in the candidate sets is retrieved from the knowledge base, and the results are stored in the candidate text set S. c ={s1, ...}.

[0012] 1-3. Open Information Extraction: Extracting open information from the candidate text set S cAll text in R was extracted using Stanford OpenIE, an open information extraction tool, to extract relational triples. s = {(subject, relation, object), ...}, where each relation triple consists of a subject, a relation term, and an object, thus obtaining the relation set R of a given text. s .

[0013] 1-4. Part-of-Speech Tagging: Spacy natural language processing tool is used to tag the triple relations with part-of-speech tags to obtain all triple relations and the set of parts of speech for each relation.

[0014] 1-5. Filtering of candidate texts: After steps 1-3 and 1-4, the triple relations and their parts of speech for each text in each candidate set are obtained. Texts that cannot generate triple representations and texts in which the subject and object of all triple relations are stop words and pronouns are filtered out, resulting in a cleaned text set.

[0015] Step 2: Modeling the joint graph structure of the table data set, annotation data, and text data. This invention designs a graph structure modeling rule based on the table data set T, annotation data set, and text data set. Construct the corresponding graph structure vertices to distinguish different elements, and construct the edge set to illustrate the relationships between vertices. (Based on the input instance) Figure 4 Examples of constructed graph structures Figure 2 As shown.

[0016] Input: Table data set T = {τ} 1 , τ 2 ,...,τ n}, each table τ i =(R i C i X i It contains several lines. List and cells (The cell collection can be retrieved by traversing in row or column order), entity annotation collection. (in Type annotation collection (in ), text collection This indicates that for each table τ i Each has a corresponding element item S iHowever, it can be an empty element.

[0017] 2-1. Construction of the vertex set of the graph: Given all table data sets, annotation data sets, and related text data sets, this invention constructs different types of vertex sets to distinguish various types of elements, specifically including table vertices n τ n, row vertex r Column vertex n c Cell vertex n x Entity vertex n e Type vertex n t and text vertex n s .

[0018] 2-2. Construction of Graph Edge Sets: This invention constructs different types of edge sets to model the structural and semantic relationships between different vertices, specifically including table structure edges E representing table structure relationships. t Knowledge edge E representing the relationship between entities and types k Annotation edges E representing information about specific entities or types a Auxiliary edge E representing the relationship between text and corresponding entity columns. u Similar to the word edge E l .

[0019] Step 3: Feature vectorization representation based on graph structure data. BERT is used to vectorize the text features of graph vertices, transforming them into the feature matrix required by the graph convolutional network model. An adjacency matrix is ​​then built based on the edge set.

[0020] Input: The set of vertices of the graph N = {n τ n r n c n x n e n t n s The set of edges of the graph is E = {E} t E k E a E u E l}

[0021] 3-1. Feature Matrix Construction: For the four types of vertices with original text information—cells, entities, types, and text—this invention uses BERT to vectorize their features. For the three types of vertices representing table structure relationships—rows, tables, and columns—we use the mean of the input feature vectors of all cells related to that vertex (subordination, such as a row vertex including all cell vertices under that row) to construct the feature matrix V for model input.

[0022] 3-2. Adjacency Matrix Construction: In this invention, all edges are treated as undirected edges. Considering that vertices have autocorrelation, the adjacency matrix is ​​first initialized to an identity matrix I with the same number of vertices. The adjacency matrix A of the graph is obtained by traversing all edge sets and assigning edge weights according to position indices.

[0023] Step 4: Parallel Multi-Task Learning and Result Prediction. This invention designs a multi-task learning training process, adding a type predictor for entity-related text to the entity predictor and type predictor. By training the text predictor, the model focuses its attention on words related to entities and types in the text. All three predictors output prediction results simultaneously. Then, the loss function is calculated based on the true labels, resulting in a weighted sum. Backpropagation is then performed to continuously fine-tune and optimize the model parameters. This process is iterated until the loss function converges, at which point training stops, yielding the final model parameters. The feature matrix V and adjacency matrix A are input into the trained model. A two-layer Graph Convolutional Network (GCN) is used to obtain the embedding representation h of the graph vertex set. The embedding features of all cells and column vertices to be predicted are input into the entity predictor and type predictor, respectively, to obtain the prediction results.

[0024] Input: Feature matrix V and adjacency matrix A, and the set of true labels for the training samples of the three prediction tasks.

[0025] 4-1. Embedding Representation Based on Graph Convolutional Neural Network (GCN): This invention uses two layers of GCN to obtain latent semantic feature information in graph structure data and outputs the embedding vector representation h of all vertices.

[0026] 4-2. Based on parallel multi-task model training, this invention designs a training module that jointly performs three prediction tasks: column-type, cell-entity, and text-type, to improve model performance and enhance model robustness. By simultaneously learning these three tasks for joint training, our model parameters are optimized.

[0027] 4-3. Result Prediction, Entity Annotation, and Type Annotation: The features of the table data set of the entities or categories to be predicted are input into the trained model. After computation, the entity predictor p... x With category predictor p c The result is the final model prediction result, and we accept the prediction with the highest probability exceeding the threshold. Category and If there is no probability threshold If the prediction fails, the prediction is rejected (and the prediction result is left blank). We match the predicted entity (or type) with the corresponding cell (or column), which is the process of annotating the entity and type.

[0028] The beneficial effects of this invention are as follows:

[0029] To address the issue that existing research often directly employs pre-trained and fine-tuned deep learning language models without considering the relationship between tabular data sets and textual data, this invention proposes a method that combines entity-related textual features with entity-specific annotations for tabular data sets. This invention includes text extraction and preprocessing methods, a table-to-graph structure modeling method, and a multi-task learning method, which enhances the model's ability to parse the structure and semantics of tables, improves robustness, and increases the model's predictive ability.

[0030] Secondly, compared to existing models that rely on the existence of table metadata, this invention can introduce entity-related text data that is easily obtained from the knowledge base without relying on metadata, resulting in significantly better performance than existing models. Furthermore, since it does not require extensive pre-training, this invention also offers some improvement in time efficiency.

[0031] This invention can significantly improve upon existing state-of-the-art methods on small and medium-sized datasets. Most current methods require a large amount of training data, which means that a lot of manual annotation is needed. This invention can effectively save the cost of manual annotation. Attached Figure Description

[0032] Figure 1 This is a flowchart of the text extraction and preprocessing process of this invention;

[0033] Figure 2 This is an example of how the present invention models table data, annotation data, and text data as graph structure data;

[0034] Figure 3 This is a diagram illustrating the training and prediction process of the model in this invention;

[0035] Figure 4 This is an example of an implementation of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples.

[0037] like Figure 4 As shown, the data input for this invention is a table (left sub-figure), through... Figure 1 The text extraction and preprocessing workflow yielded the entity-related text instances on the right (right subgraph). The overall training and prediction process of the model is as follows: Figure 3 As shown, the input passes through Figure 2 After the modeling process is converted into graph-structured data, the input model simultaneously outputs prediction results for entities and categories. The specific steps are as follows:

[0038] Step 1: Adaptive method for extracting and preprocessing entity-related definitional text data;

[0039] like Figure 1 As shown, this invention designs an adaptive method for extracting entity-related definitional text from a knowledge base and preprocessing it. The text defines the corresponding entity from the perspective of the entity's type, thus improving the model's ability to understand the table data set.

[0040] Input: A set of table data T = {τ} 1 , τ 2 ,,...,,τ n Each table contains several cell information. Entity annotation set corresponding to each table (Entity comments for some tables may be empty sets) ).

[0041] 1-1. Adaptive Candidate Set Selection Method; Adaptive means for cases without entity annotation information. or number of entity annotations The number of non-empty cells in the primary key (PK) column of the i-th table does not exceed the number of cells in the primary key column. Half of the table, A candidate set is constructed by semantically filtering all cell information. Conversely, all annotated entity names in that table will be used as the candidate set.

[0042] 1-2. Text extraction from the knowledge base: Each element in the two candidate sets is used as a keyword to search for relevant text in the DBpedia knowledge base, and the results are stored in the candidate text set S. c ={s1, ..., s z}

[0043] 1-3. Open Information Extraction: Extracting open information from the candidate text set S c All text in the dataset was extracted using Stanford OpenIE, an open information extraction tool, to represent relation triples. Each relation triple consists of a subject, a relation term, and an object, resulting in a relation set R for a given text. s ={(subject, relation, object),...}.

[0044] 1-4. Part-of-Speech Tagging: Part-of-speech tagging was performed on all triple relations using the Spacy natural language processing tool to obtain a set of all triple relations and the parts of speech belonging to different relations.

[0045] 1-5. Filtering of Candidate Text: After steps 1-3 and 1-4, the triple relations and their parts of speech for each text in each candidate set are obtained. Texts that cannot generate triple representations and invalid texts where the subject and object of all triple relations are stop words and pronouns are filtered out, resulting in the cleaned text set corresponding to each table.

[0046] Step 2: A modeling method for transforming the table data set, annotation data set, and text data set into graph structure data;

[0047] Based on the table data set T and the entity annotation set Type annotation collection and the extracted text set Establish relevant rules to construct the corresponding graph vertex set and edge set, based on the input instance. Figure 4 Examples of constructed graph structures Figure 2 As shown.

[0048] Input: Table data set T = {τ} 1 , τ 2 ,...,τ n}, each table τ i =(R i C i X i It contains several lines. List and cells (The cell collection can be retrieved by traversing in row or column order), entity annotation collection. (in Type annotation collection (in Text collection (This indicates that for each table τ) i Each has a corresponding element item S i However, it can be an empty element. ).

[0049] 2-1. Graph Vertex Set Construction: To preserve structural information and capture key features, this invention constructs various types of vertices for each table. This is based on a given set of annotations for all entities. and type annotation collection Integrate into a collection of entities contained in all tables. and type collection To fully represent the content and structure of the tabular data, we construct corresponding table element vertices for each table, including table vertex n. τ n, row vertex r Column vertex n c and cell vertex n x Each element corresponds one-to-one with the elements τ, r, c, and x in the table. Then, to capture the semantic features of entities and types, we construct entity vertices n for all entities and types. e and type vertex n t Finally, to combine the additional textual information, we construct text vertices n for each text. s Thus, we obtain the set of all vertices N = {n} of the graph structure modeled from table data sets, entity and type information, and text data. τ n r n c n x n e n t n s}

[0050] 2-2. Graph Edge Set Construction: After constructing a series of graph vertices, we construct different types of edges to represent the structural and semantic relationships between different vertices, in order to better capture the deep semantic and structural information hidden in the table data set. Specifically, this is divided into table structure edges E representing the table's structural relationships. t Knowledge edge E representing the relationship between entities and types k Annotation edges E representing information about specific entities or types a Auxiliary edge E representing the relationship between text and corresponding entity columns. u Similar to the word edge E l .

[0051] Table structure edge E representing the table structure relationship t Capture the basic shallow structure information of table data, specifically divided into cell vertices n x and corresponding row vertex n r The edges between them, the vertex of the cell n x and corresponding column vertex n c The edges between them, the table vertices n τ and all rows of the table with vertex n r The edges between and the table vertex n τ and all columns of the table, vertex n c There are four types of edges between them.

[0052] The knowledge edge E represents the relationship between an entity and its type. k Used to express entity vertex n e and corresponding type vertex n tThe relationship between entities is such that if a cell in a column with type annotations is annotated by an entity, then that entity belongs to the corresponding type; that is, there is a knowledge edge E between the entity vertex and the type vertex. k However, due to data loss caused by poor network quality, not all entities and types may have knowledge edges.

[0053] Annotation edge E represents specific entity or type information of a cell or type. a There are n vertices in the cell x and corresponding entity vertex n e Comments between, column vertices n c and corresponding type vertex n t There are two types of annotation edges, which are used to model the existing annotation information in the training set into edge features that the model can understand and obtain.

[0054] Auxiliary edge E representing text type information u Used to connect text vertices n that serve to interpret entities. s and the corresponding column vertex n c The text information we obtain from the knowledge base is a triplet relation structure, which explains and describes the type of the corresponding entity. Therefore, adding text features and establishing auxiliary edges in the model can help the model better complete the cell-entity annotation task and the column-type annotation task. If at text vertex n... s and cell vertex n x Establish auxiliary edge E between u This increases the overhead of the model in capturing latent features, and makes the model more susceptible to noise and weakly correlated vertices. Therefore, we [details about text vertices n]. s and the corresponding column vertex n c Constructing auxiliary edges E u .

[0055] Lexical similarity edge E l Vertices n of cells that exhibit lexical similarity across all table data sets. x This enhances the model's ability to capture relationships and information between cell vertices. If the word-based Jaccard similarity between two cell vertices is higher than a set threshold (experimentally verified to be optimal at 0.8), then the two cell vertices are considered to have a word-similar edge E. l Jaccard similarity based on words is a text... With text The formula for dividing the number of words in the intersection by the number of words in the union is:

[0056]

[0057] Thus, we obtain the graph edge set E = {E tE k E a E u E l}

[0058] Step 3: Feature vectorization representation: Based on the text information of the graph vertices, use BERT to perform feature vectorization representation, transform it into the feature matrix required by the graph convolutional network model, and build the adjacency matrix required by the model based on the edge set.

[0059] Input: The set of vertices of the graph N = {n τ n r n c n x n e n t n s The set of edges of the graph is E = {E} t E k E a E u E l}

[0060] 3-1. Feature Matrix Construction: For cell vertices, entity and type vertices, and text vertices containing original text information, we use the BERT word vector representation method to transform the original text information into the input features of the corresponding vertices. That is, the input feature vectors of these four types of vertices are the average of the feature vectors of all words contained in the original text after BERT transformation.

[0061]

[0062] For the three types of vertices—row, table, and column—we represent them using the mean of the input feature vectors of all cell vertices related to that vertex (dependence, such as a row vertex including all cell vertices in that row). Specifically, a row vertex is represented by the mean of the feature vectors of all cell vertices in that row:

[0063] v r =mean(∑ x∈r v x (3)

[0064] Similarly, column vertices are represented by the vector mean of all the vertices in that column:

[0065] v c =mean(∑ x∈c v x (4)

[0066] The characteristic of a table vertex is the average of the characteristic vectors of all row vertices (or column vertices) in that table:

[0067] v τ=mean(∑ r / c∈τ v r / C (5)

[0068] Where x∈r represents all cells belonging to a certain row, x∈c represents all cells belonging to a certain column, and r / c∈τ represents all rows or columns belonging to a certain table (either row or column can be selected, one of the two).

[0069] In this way, we have constructed the feature matrix for all vertices of the model input. (The superscript T in the moment reduction operation represents the transpose operation), N is the total number of vertices, and the feature dimension of all vertex vectors is 768.

[0070] 3-2. Adjacency Matrix Construction: Obtain the binary symmetric adjacency matrix of the graph based on the edge set E of the constructed graph. The specific process is as follows: Considering the autocorrelation of vertices, matrix A is first initialized to an identity matrix A = I, which is the same as the number of vertices. Then, the edge set E (treating edges as undirected and setting all weights to 1) is traversed to assign values ​​to the adjacency matrix A. Thus, the adjacency matrix A of the graph is a symmetric matrix, and its values ​​are only 0 and 1. For example, if there exists an edge... Then let the nth adjacency matrix a row n b Column and nth b row n a All columns are 1: This yields the adjacency matrix A of the graph.

[0071] Step 4: Parallel multi-task learning and outcome prediction;

[0072] The overall process is as follows Figure 3 As shown, this invention designs a multi-task learning training method. Based on the entity predictor and type predictor, a type predictor for entity-related text is added. By training the text predictor, the model focuses its attention on words in the text related to entities and types. All three predictors output prediction results simultaneously. Then, a loss function is calculated based on the true labels, resulting in a weighted sum. Backpropagation is then performed to continuously fine-tune and optimize the model parameters. This process is iterated until the loss function converges, at which point training stops, yielding the final model parameters. The feature matrix V and adjacency matrix A are input into the trained model. A two-layer graph convolutional network is used to obtain the embedding representation h of the graph vertex set. The embedding features of all cells and column vertices to be predicted are input into the entity predictor and type predictor, respectively, to obtain the prediction results.

[0073] Input: Feature matrix V and adjacency matrix A, and the set of true labels for the training samples of the three prediction tasks.

[0074] 4-1. Embedding Representation Based on Graph Convolutional Neural Network (GCN): The model first inputs the transformed graph structure data set feature matrix V and adjacency matrix A into the graph convolutional neural network. Two layers of the graph convolutional network (the number of layers refers to the farthest distance that vertex features can be transmitted; for a single-layer GCN network, each vertex can only obtain information from its neighbors; the receptive field of the network increases with the number of GCN layers, so higher-order neighborhood information can be merged by stacking multiple GCN layers. However, if the network is too deep, each vertex will be affected by irrelevant vertices, and the effect will decrease. Extensive experiments have shown that setting the number of GCN layers to two yields the best results) acquire the latent semantic feature information in the graph structure data and output the embedding vector representation h of all vertices. The dimensions of the two graph convolutional layers are 512 and 256, respectively. The calculation formula for each graph convolutional network layer is:

[0075]

[0076] Where j represents the GCN layer number, and the input of the zeroth layer GCN is the feature matrix itself L. (0) =V, and the output of each subsequent layer serves as the input of the next layer. The output of the second layer is the embedding representation h=L. (2) W j This is the weight matrix for this layer, with the corresponding specific dimensions being... It is a Laplace regularization matrix, where D is the degree matrix, and the elements on its diagonal are the sum of the in-degree and out-degree of the corresponding vertices (i.e., D). ii =∑ j A ij +∑ k A ki =2×∑ j A ij ), or the Laplace regular matrix This is the result of performing a symmetric normalization operation on the adjacency matrix A. ρ is the activation function; here we use the ReLU activation function, i.e., ρ(x) = max(0, x). Thus, we obtain the embedding representation of all vertices through the graph convolutional layer.

[0077] 4-2. Model training method based on parallel multi-task joint learning: In order to improve the performance of the model on two annotation tasks, after obtaining the embedding representation of all vertices, this invention designs a multi-task learning training module, which consists of three prediction modules: column-type, cell-entity, and text-type.

[0078] 4-2-1. Column-Type Prediction Task: To predict the type t of an unannotated column c. c We will use the embedding vector h of the column to be predictedc First, through the projection matrix P t The vectors are projected onto a type space and then fed into a type predictor. Specifically, we use a fully connected layer to change the vector dimensions and then use a softmax function to output the predicted value for each class, as shown in the formula:

[0079] p c =softmax(W t (P t h c )+b t (7)

[0080] in, These are the weight matrix and bias term of the fully connected layer, respectively, where q represents the total number of classes to be predicted, and the softmax calculation formula is... U represents the total set of categories.

[0081] 4-2-2. Cell-Entity Prediction Task: Similar to the column-type predictor, we will predict the category e of the entity to be predicted. x cell embedding vector h x First, through the projection matrix P e The image is projected onto the entity space and then fed into the entity predictor (fully connected layer + softmax function) to output the predicted value for each entity category. The formula is expressed as:

[0082] p x =softmax(W e (P e h x )+b e (8)

[0083] in, These are the weight matrix and bias term of the fully connected layer, respectively, and g represents the total number of entities to be predicted.

[0084] 4-2-3. Text-Type Prediction Task: To focus the model's attention on the entity and type information of the text, we constructed a text type predictor. The type of the text comes from the type corresponding to the entity associated with the text, so the set of text types is a subset of the set of column types. Similarly, we will embed the text vector h of the type to be predicted. s First, through the projection matrix P s Projecting onto the text type space, a text type predictor (fully connected layer + softmax function) is then constructed to output predicted values ​​for each category. The formula is expressed as:

[0085] p s =softmax(W s (P s hs )+b s (9)

[0086] in, These are the weight matrix and bias term of the fully connected layer, respectively, and z represents the total number of text types to be predicted.

[0087] 4-2-4. Joint Learning: Our model parameters are optimized through joint training by simultaneously learning three tasks, using Adam as the optimizer. (The text then abruptly shifts to unrelated topics: cell's true entity label.) and column real type tags The text's true type labels are derived from the column type labels, derived from the dataset annotation file. We use the cross-entropy loss function as the loss function for the three prediction tasks. The final calculated loss is the weighted sum of the losses from the three tasks, with weight ratios set to δ1 = 1, δ2 = 2, and δ3 = 0.5 (experimentally adjusted to the optimal values). The specific formula is as follows:

[0088]

[0089] Where, N C N x N s These represent the number of samples in the training column, cell, and text, respectively; q, g, and z represent the number of corresponding classes, respectively. The sign function (0 or 1) represents the true class of the three tasks respectively. Specifically, it takes 1 if the true class of sample i is j, and 0 otherwise. This represents the predicted probability that sample i belongs to category j in the three tasks.

[0090] By using backpropagation of the loss function, the model parameters are continuously fine-tuned and optimized through iterative processes. Training stops when the loss function converges, resulting in the final model parameters.

[0091] 4-3. Result Prediction, Entity Annotation, and Type Annotation: From step 4-2, we obtained the trained model parameters. We then input the features of the table data set of the entities or categories to be predicted into the model. After computation, the entity predictor p... x With category predictor p c The result is the final model prediction result, and we accept the prediction with the highest probability exceeding the threshold. Category and If there is no probability threshold Then reject the prediction (set the prediction result to null), the specific formula is:

[0092]

[0093]

[0094] Here, A? B:C is a ternary operator, meaning that if A is true, return B; otherwise, return C. The argmax function returns the index corresponding to the maximum value. g and q correspond to the total number of entities and types. This represents an array of probabilities generated by the entity predictor that a specific cell could belong to each entity. This represents an array of probabilities generated by the type predictor for a specific column, indicating that it might belong to each type. We then match the predicted entities (or types) with the corresponding cells (or columns), which is the process of annotating entities and types.

[0095] Example: We conducted experiments on five datasets: Wiki M, Web M, Limaye, T2Dv2, and Wikipedia. During the experiments, we replicated four mainstream models: ColNet, TaBERT, TURL, and Doduo, to compare the results with our method. The statistics for the five datasets are shown in Table 1. Note that T2Dv2 and Wikipedia lack entity annotations, therefore the cell-to-entity prediction task cannot be completed on these two datasets.

[0096] Table 1 shows the statistics for the datasets (- indicates that there is no corresponding content for this type of dataset).

[0097]

[0098] The experimental results are shown in Table 2, and the micro-average F1 score was used as the evaluation metric. The results show that, except for the column-type prediction task in Wiki M, where it doesn't perform as well as the TURL model (because TURL's pre-trained corpus contains all table data from Wiki M, leading to overfitting on this dataset), our model achieves significant improvements in accuracy compared to other models on all datasets (with a maximum improvement of 30% on the column-type prediction task in T2Dv2). ColNet, TaBERT, TURL, and Doduo can only perform column type prediction, while our model (Ours), employing a multi-task prediction module, can simultaneously perform cell-entity prediction and column-type prediction tasks. This demonstrates the superior performance of our proposed method.

[0099] Table 2 Experimental Results (- indicates that this type of model cannot complete the specific task)

[0100]

Claims

1. A method for joint text enhancement of table entities and type annotations based on graph convolutional networks, characterized in that... Includes the following steps: Step 1: Adaptive extraction and preprocessing of entity-related definitional text data; Input: A collection of table data Each table contains multiple columns. Each column contains information from several cells. The entity annotation set corresponding to each table ; 1-1. Adaptive candidate set selection method; Adaptive refers to adapting to situations where there is no entity annotation information or the number of entity annotations is limited. Not exceeding the number Compare the key columns of the table to the number of non-empty cells. Half of the table, A candidate set is constructed by semantically filtering all cell information. Conversely, all annotated entity names in that table will be used as the candidate set. ; 1-2. Text extraction from the knowledge base: Each element in the two candidate sets is used as a keyword. Text related to the elements in the candidate sets is retrieved from the knowledge base, and the results are stored in the candidate text set. ; 1-3. Open Information Extraction: Extracting from Candidate Text Sets All text was extracted using Stanford OpenIE, a tool for extracting relational triples. Each relation triple consists of a subject, a relational noun, and an object, thus obtaining the relation set of a given text. ; 1-4. Part-of-speech tagging: Spacy natural language processing tool is used to perform part-of-speech tagging on triple relations to obtain a set of all triple relations and the parts of speech of different relations. ; 1-5. Filtering of candidate texts: Texts that cannot generate triples and texts in which all triple relations consist of stop words and pronouns as both subject and object are removed, resulting in a cleaned text set. ; Step 2: A modeling method for transforming the table data set, annotation data set, and text data set into graph structure data; Step 3: Feature vectorization: Based on the text information of the graph vertices, use BERT to perform feature vectorization, transform it into the feature matrix required by the graph convolutional network model, and build the adjacency matrix required by the model based on the edge set; Step 4: Parallel multi-task learning and outcome prediction.

2. The method for joint text enhancement of table entities and type annotation based on graph convolutional networks according to claim 1, characterized in that... Step 2 is implemented as follows: Input: Table data set Each table It contains several lines ,List and cells Entity annotation set Type annotation collection Text collection This means that for each table Each has a corresponding element item And can be an empty element. ; 2-1. Graph vertex set construction: Given all table data sets, annotation data sets, and related text data sets, construct different types of vertex sets to distinguish various types of elements, specifically including table vertices. , row vertex Column vertices Cell vertex Entity vertices Type Vertex and text vertices ; 2-2. Construction of edge sets in a graph: Constructing different types of edge sets to model the structural and semantic relationships between different vertices, specifically table-structured edges representing table-like relationships. Knowledge edges representing the relationship between entities and types Annotation edges representing information about specific entities or types Auxiliary edges representing the correspondence between text and entity columns. Similar to words .

3. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 2, characterized in that... Step 2-2 is implemented as follows: Table structure edges representing table structure relationships Capture basic shallow structural information of table data, specifically divided into cell vertices. and corresponding row vertex Edges and cell vertices between and corresponding column vertices Edges and table vertices between and all rows of the table vertices The edges between and the table vertices and all column vertices of the table There are four types of edges between them; Knowledge edges representing the relationship between entities and types Used to represent entity vertices and corresponding type vertices The relationship between entities is such that if a cell in a column with type annotations is annotated by an entity, then that entity belongs to the corresponding type; that is, there is a knowledge edge between the entity vertex and the type vertex. However, due to data loss caused by poor network quality, not all entities and types may have knowledge edges; Annotation edges that represent specific entity or type information for a cell or type There are cell vertices and corresponding entity vertices Comments and column vertices and corresponding type vertices There are two types of annotation edges, which are used to model the existing annotation information in the training set into edge features that the model can understand and obtain. Auxiliary edges representing text type information Used to connect text vertices that serve to interpret entities. and corresponding column vertices The text information obtained from the knowledge base is a triple relation structure. Therefore, adding text features and establishing auxiliary edges in the model can help the model better complete the cell-entity annotation task and column-type annotation task. Lexical similarity edges Cell vertices used to connect all table data sets where there is lexical similarity. This enhances the model's ability to capture relationships and information between cell vertices; if the word-based Jaccard similarity between two cell vertices exceeds a set threshold, then it is considered that there is a word-similar edge between these two cell vertices. Jaccard similarity based on words is a text-based similarity method. With text The formula for dividing the number of words in the intersection by the number of words in the union is: This yields the set of edges of the graph. .

4. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 2 or 3, characterized in that... The input to the feature vectorization representation described in step 3 is the set of vertices of the graph. The set of edges of a graph .

5. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 4, characterized in that... The specific implementation of step 3 includes: 3-1. Feature Matrix Construction: For cell vertices, entity and type vertices, and text vertices containing original text information, the BERT word vector representation method is used to transform the original text information into the input features of the corresponding vertices. That is, the input feature vectors of these four types of vertices are the average of the feature vectors of all words contained in the original text after BERT transformation. For the three types of vertices—row, table, and column—the mean of the input feature vectors of all cell vertices related to that vertex is used. Specifically, a row vertex is represented by the mean of the feature vectors of all cell vertices in that row. The vertex of a column is represented by the vector mean of all the vertices of that column: The characteristic of a table vertex is the average of the characteristic vectors of all row vertices or column vertices in that table. in, This represents all cells belonging to a given row. This represents all cells belonging to a certain column. This indicates all rows or columns belonging to a certain table; Therefore, the feature matrix of all vertices used as input to the model is constructed. , The total number of vertices is 768, and the feature dimension of all vertex vectors is 768.

6. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 5, characterized in that... The specific implementation of step 3 also includes: 3-2. Adjacency Matrix Construction: Treat all edges as undirected edges. Considering the autocorrelation of vertices, first initialize the adjacency matrix to an identity matrix with the same number of vertices. The adjacency matrix of the graph was obtained by traversing all edge sets and assigning edge weights according to their position indices. .

7. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 6, characterized in that... Step 4 designs a multi-task learning training process. Based on the entity predictor and type predictor, a type predictor for entity-related text is added. By training the text predictor, the model focuses its attention on words in the text related to entities and types. All three predictors output prediction results simultaneously. Then, a loss function is calculated based on the true labels, resulting in a weighted sum. Backpropagation is then performed to continuously fine-tune and optimize the model parameters. This process is iterated until the loss function converges, at which point training stops, yielding the final model parameters. The feature matrix is ​​then... and adjacency matrix Input the trained model and use a two-layer graph convolutional network to obtain the embedding representation of the graph vertex set. The embedding features of all cells and column vertices to be predicted are input into the entity predictor and type predictor respectively to obtain the prediction results.

8. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 6 or 7, characterized in that... Step 4 is implemented as follows: Input: Feature matrix and adjacency matrix The set of true labels for the training samples of the three prediction tasks ; 4-1. Embedding Representation Based on Graph Convolutional Neural Network (GCN): A two-layer GCN is used to extract latent semantic feature information from graph structure data and output the embedding vector representation of all vertices. ; 4-2. Based on parallel multi-task model training, a training module for jointly predicting three tasks—column-type, cell-entity, and text-type—was designed to improve model performance and enhance model robustness. By simultaneously learning the three tasks for joint training, our model parameters were optimized. 4-3. Result Prediction, Entity Annotation, and Type Annotation: The features of the table data set of the entities or categories to be predicted are input into the trained model. After processing, the entity predictor... With category predictors The result is the final model prediction result, which is the one with the highest probability of acceptance and exceeds the threshold. Category and If there is no probability threshold If the prediction fails, the predicted entity or type is matched with the corresponding cell or column, which is the process of annotating the entity and type.

9. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 8, characterized in that... Step 4-2 is implemented as follows: 4-2-1. Column-Type Prediction Task: To predict unannotated columns Type The embedding vector of the column to be predicted First through the projection matrix The vector is projected onto the type space and then fed into the type predictor; specifically, a fully connected layer is used to change the vector dimensions, and then a softmax function is used to output the predicted value for each class, as shown in the formula: in, These are the weight matrix and bias term of the fully connected layer, respectively. This represents the total number of categories to be predicted. The softmax calculation formula is: , For the total category set; 4-2-2. Cell-Entity Prediction Task: Predict the category of the entity to be predicted. Cell embedding vector First through the projection matrix Projected onto the entity space, and then input into the entity predictor, the predicted values ​​for each entity category are output; the formula is expressed as: in, These are the weight matrix and bias term of the fully connected layer, respectively. This represents the total number of entities to be predicted; 4-2-3. Text-Type Prediction Task: To focus the model's attention on the entity and type information of the text, a text type predictor was constructed; the type of the text comes from the type corresponding to the entity associated with the text, so the set of text types is a subset of the set of column types. Similarly, embed the text of the type to be predicted into a vector. First through the projection matrix Project the data onto the text type space, then construct a text type predictor to output predicted values ​​for each category; the formula is expressed as: in, These are the weight matrix and bias term of the fully connected layer, respectively, and z represents the total number of text types to be predicted; 4-2-4. Joint Learning: The model parameters are optimized by simultaneously learning three tasks; the optimizer used is Adam. (The cell contains the actual entity labels.) and column real type tags The text's true type labels are derived from the column type labels, derived from the dataset annotation file. The cross-entropy loss function is used as the loss function for the three prediction tasks, and the final calculated loss is the weighted sum of the losses from the three tasks, with the weight ratio set to... The specific formula is as follows: in, These represent the number of samples in the training column, cell, and text, respectively. Each represents the number of the corresponding category; These are the sign functions representing the true categories of the three tasks. Specifically, they are 1 if the true category of sample i is j, and 0 otherwise. This represents the predicted probability that sample i belongs to category j in the three tasks; By using backpropagation of the loss function, the parameters of the model are continuously fine-tuned and optimized, and the process is repeated iteratively until the loss function converges, at which point training stops, and the final model parameters are obtained.

10. The method for joint text enhancement of table entities and type annotations based on graph convolutional networks according to claim 9, characterized in that... The specific formula for step 4-3 is as follows: in, This is a ternary operator, meaning that if A is true, return B; otherwise, return C. The argmax function returns the index of the maximum value. The total number of corresponding entities and types, This represents an array of probabilities generated by the entity predictor that a specific cell could belong to each entity. This represents an array of probabilities that a specific column might belong to each type, generated by the type predictor.

Images