A method and device for OCR table semantic recognition based on graph neural network

Abstract

The present invention discloses a graph neural network-based OCR table semantic recognition method and device. The method inputs a PNG table image into a trained GKVR recognition model. The model accurately determines whether the table node attribute is a key or value by analyzing the sentence vector features, node image features, and position features of the text nodes. Furthermore, by setting a partitioning rule tree to achieve matching between keys and values, the method improves the ability to identify the relationship between keys and values in a table. The present invention combines deep learning network structures such as graph neural networks and gated recurrent units to propose a GKVR network model for table key-value recognition, enabling one-click recognition and meeting practical industrial needs such as automated table review.

Description

Technical Field

[0001] The present invention relates to the field of OCR recognition technology, and in particular to an OCR table semantic recognition method and device based on graph neural network. Background Art

[0002] With the rapid development of computing power and parallel computing technology in the current computer field, the number of trainable parameters for deep learning is increasing, which also makes deep learning's ability to learn complex data gradually enhanced, and eventually it can be applied in various fields.

[0003] In traditional OCR (optical character recognition), methods such as template matching and structural analysis are used for text recognition. Based on OCR text recognition, researchers have further proposed the problem of how to identify table structures and table text information, and have added steps such as table detection, table structure decomposition, and table structure recognition to the workflow of text recognition. For example, in the prior art, Marcin Namysl et al. proposed combining semantic information with a rule-based algorithm to extract tables in the article "Flexible Table Recognition and Semantic Interpretation System". Yiren Li et al. proposed a GFTE model in "GFTE: Graph-based Financial Table Extraction" that combines image features, position features, and text features to improve the ability to extract tabular unstructured data files.

[0004] However, in the field of intelligent manufacturing, users' needs are not only satisfied with the recognition of table structures, but also increasingly attach importance to the recognition of table key values and their corresponding relationships. In addition, there are various forms in real life. For example, in the quality inspection report review workflow, when the user submits their quality inspection report (usually provided as a PDF file), the reviewer needs to verify whether the corresponding inspection indicators meet industry standards and whether information such as product names meet the review requirements. The recognition of key values in the table and their corresponding relationships can affect recognition efficiency and accuracy. The current practical application of OCR technology in table recognition is difficult to meet the needs of automation and industrialization, and lacks efficient and accurate solutions. Summary of the Invention

[0005] The present invention provides an OCR table semantic recognition method and device based on graph neural network, which can effectively identify the key values in the table and discover the corresponding relationships between the key values, thereby meeting the actual industrial needs such as automated table review.

[0006] To solve the above technical problems, an embodiment of the present invention provides an OCR table semantic recognition method based on a graph neural network, comprising:

[0007] Obtain a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table;

[0008] Input the first PNG table image into the trained GKVR recognition model, so that the GKVR recognition model performs OCR recognition on the first PNG table image, obtains first text information, table frame information and position information of each text node, and generates sentence vector features corresponding to the first text information through the GRU network based on the first text information and a preset vocabulary, and then converts the table frame information into node image features through a convolutional neural network and a grid_simple algorithm, and then normalizes the position information of each text node to obtain position features, and finally inputs the sentence vector features and the position features corresponding to the first text information into a graph attention network respectively, and after splicing with the node image features, passes through a multi-layer perceptron MLP, and outputs a key-value information set corresponding to the first PNG table image; wherein the key-value information set includes: a key information set and a value information set;

[0009] According to the preset partitioning rule tree, the key-value information set is traversed and matched, and each key-value pair in the key-value information set is output.

[0010] As a preferred solution, the trained GKVR recognition model includes: a sentence vector feature extraction module;

[0011] The training process of the sentence vector feature extraction module is specifically as follows:

[0012] According to the preset vocabulary, the text content of each text node in the training sample is recognized to generate a string. After one-hot encoding, each string is applied to a single-layer feedforward network for word embedding to obtain the word sequence corresponding to each text node.

[0013] The semantics of each word sequence is learned through the GRU network to generate sentence vector features for each text node.

[0014] As a preferred solution, the trained GKVR recognition model includes: a node image feature extraction module;

[0015] The training process of the node image feature extraction module is specifically as follows:

[0016] Acquire multiple table frame information in the training sample, and extract image structure information from each table frame information through a convolutional neural network to obtain multiple first feature maps;

[0017] The multiple first feature maps are scaled to a grid using a bilinear interpolation method through the grid_simple algorithm, and the grid features of the coordinates corresponding to each text node are used as the node image features of the text node.

[0018] As a preferred solution, the trained GKVR recognition model includes: a position feature extraction module;

[0019] The training process of the position feature extraction module is specifically as follows:

[0020] Obtain the position information of each text node in the training sample;

[0021] The coordinates of each position information are converted, and the coordinate system is normalized to the interval [-1, 1], and the position features corresponding to each text node are output.

[0022] As a preferred solution, the training process of the trained recognition model is specifically as follows:

[0023] The sentence vector features, node image features, and position features corresponding to each text node in the training sample are used as the input of the GKVR recognition model, and the key information and value information corresponding to each text node are used as the output of the GKVR recognition model;

[0024] For each text node, the sentence vector feature and position feature are respectively input into the graph attention network, and after being spliced with the node image feature, the node feature of each text node is formed. Combined with the output of the GKVR recognition model, the graph attention network and multi-layer perceptron MLP are trained.

[0025] As a preferred solution, the first PNG table image is obtained by pre-processing the PDF table, specifically:

[0026] The PDF document to be processed is obtained, and the table portion is intercepted from the PDF document using the KVLabel tool to generate the first PNG table image.

[0027] As a preferred solution, the KVLabel tool is also used to pre-process the training samples of the GKVR recognition model, specifically:

[0028] The KVLabel tool is used to select the table frame of the PDF document in the initial sample, and each text node in the table frame is labeled with key values and key-value pairs to generate a PNG table image corresponding to each initial sample, and all PNG table images, key-value labels and key-value pair labels are used as the training samples.

[0029] As a preferred solution, the key-value information set is traversed and matched according to a preset partitioning rule tree, and each key-value pair in the key-value information set is output, specifically:

[0030] The key information set is gradually divided by traversing the partition rule tree in a breadth-first manner, and when a leaf node is reached, a value in the value information set is selected to generate several key-value pairs.

[0031] As a preferred solution, the partitioning rule tree is set in the GKVR recognition model.

[0032] Another embodiment of the present invention provides an OCR table semantic recognition device based on a graph neural network, comprising: an acquisition unit, a recognition unit, and an output unit;

[0033] The acquisition unit is used to acquire a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table;

[0034] The recognition unit is used to input the first PNG table image into the trained GKVR recognition model, so that the GKVR recognition model performs OCR recognition on the first PNG table image, obtains first text information, table frame information and position information of each text node, and generates a sentence vector feature corresponding to the first text information through a GRU network based on the first text information and a preset vocabulary, and then converts the table frame information into a node image feature through a convolutional neural network and a grid_simple algorithm, and then normalizes the position information of each text node to obtain a position feature, and finally inputs the sentence vector feature and the position feature corresponding to the first text information into a graph attention network respectively, and after splicing with the node image feature, passes through a multi-layer perceptron MLP, and outputs a key-value information set corresponding to the first PNG table image; wherein the key-value information set includes: a key information set and a value information set;

[0035] The output unit is used to traverse and match the key-value information set according to a preset partitioning rule tree, and output each key-value pair in the key-value information set.

[0036] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0037] The present invention provides a graph neural network-based OCR table semantic recognition method and device. By inputting a PNG table image into a trained GKVR recognition model, the model accurately determines whether the attributes of table nodes are keys or values by analyzing the sentence vector features, node image features, and position features of text nodes. Furthermore, by setting a partitioning rule tree to achieve matching between keys and values, the ability to identify the relationship between table keys and values is improved. Compared to existing technologies that are difficult to directly extract from portable document formats and images, the present invention combines deep learning network structures such as graph neural networks and gated recurrent units to propose a GKVR network model for table key-value recognition, enabling one-click recognition and meeting practical industrial needs such as automated table review. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 : A flowchart of an embodiment of an OCR table semantic recognition method based on a graph neural network provided by an embodiment of the present invention;

[0039] Figure 2 : This is a schematic diagram of the effect of drawing the original chunk rectangle annotation provided by the existing technology on the original PNG image;

[0040] Figure 3 : A schematic diagram of the effect of creating a PNG image by extracting a table portion from an original PDF file, provided by an embodiment of the present invention;

[0041] Figure 4 : A schematic diagram of chunk data provided by an embodiment of the present invention;

[0042] Figure 5 : A schematic diagram of a GKVR recognition model provided by an embodiment of the present invention;

[0043] Figure 6 : A schematic diagram showing the loss values during the training process of the GCN-based and GAT-based GKVR models provided in an embodiment of the present invention;

[0044] Figure 7 : A schematic diagram showing the accuracy of the GCN-based and GAT-based GKVR model training processes provided by an embodiment of the present invention;

[0045] Figure 8 : A schematic diagram of a partitioning rule tree used in key-value matching of the SciTSR-Key-Value dataset provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] Example 1

[0048] Please refer to Figure 1 , which is a flow chart of an embodiment of an OCR table semantic recognition method based on a graph neural network provided by an embodiment of the present invention, the method includes steps 101 to 103, each of which is specifically as follows:

[0049] Step 101: Obtain a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table.

[0050] In this embodiment, the first PNG table image is obtained by preprocessing the PDF table, specifically: obtaining the PDF document to be processed, and intercepting the table part from the PDF document through the KVLabel tool to generate the first PNG table image.

[0051] Specifically, the KVLabel tool is a tool for annotating PDF documents. It can implement functions such as region information annotation, node attribute annotation (such as whether a node is a key or a value), and node key-value pair annotation. For the PDF document to be identified, the KVLabel tool is first used to intercept its table portion and then convert it into a first PNG table image.

[0052] Step 102: Input the first PNG table image into the trained GKVR recognition model so that the GKVR recognition model performs OCR recognition on the first PNG table image to obtain first text information, table frame information and position information of each text node, and generate sentence vector features corresponding to the first text information through the GRU network based on the first text information and the preset vocabulary, and then convert the table frame information into node image features through the convolutional neural network and the grid_simple algorithm, and then normalize the position information of each text node to obtain position features, and finally input the sentence vector features and the position features corresponding to the first text information into the graph attention network respectively, and after splicing with the node image features, pass through the multi-layer perceptron MLP to output the key-value information set corresponding to the first PNG table image; wherein, the key-value information set includes: a key information set and a value information set.

[0053] In this embodiment, before executing step 102, the GKVR recognition model needs to be trained using training samples. The trained GKVR recognition model is used to recognize the first PNG table image and output a corresponding key-value information set. The key-value information set includes: a key information set, a value information set, and an other information set. The other information set is a set of content other than keys and values, such as a table header.

[0054] In this embodiment, it is possible but not limited to extracting a number of tabular data from the SciTSR dataset, and then preprocessing the tabular data using the KVLabel tool to obtain the training sample dataset SciTSR-Key-Value. The training samples of the GKVR recognition model are preprocessed, specifically by selecting a table frame of the PDF document in the initial sample using the KVLabel tool, and annotating each text node in the table frame with key values and key-value pairs, generating a PNG table image corresponding to each initial sample, and using all PNG table images, key-value annotations, and key-value pair annotations as the training samples.

[0055] In this embodiment, compared to the prior art of directly converting the PDF document in the SciTSR dataset into a PNG file and then framing the table based on the rectangular annotation of the image, this embodiment first intercepts the table frame corresponding to the table from the PDF document, then converts the intercepted table frame into a PNG table image, and finally performs key-value annotation and key-value pair annotation. This ensures that the coordinate information of the table frame is consistent with the image, avoiding the problem of misalignment or mismatch caused by converting the file format from PDF to PNG. Figure 2 and Figure 3 As shown, Figure 2 Misalignment or inconsistency problems caused by existing technologies, Figure 3 This is a PNG table image obtained by adopting the technical means of this embodiment.

[0056] As an example of this embodiment, the key value information and the coordinate information of the rectangular frame in the picture marked with the rectangular frame are stored in the chunk data. The specific schematic diagram of the chunk data can be but is not limited to see Figure 4 .like Figure 4 As shown, chunk data can also be used to save the position of text nodes, and the index of the text node in the list is its node number.

[0057] As an example of this embodiment, when labeling key-values and key-value pairs, the type attribute of the text node can be used to distinguish whether the text node is a key, a value, or other information. The type attribute of the text node is saved using info data, and the key of the dictionary is the node number and the value is the attribute. In addition, the labeling of key-value pairs is saved using pair data to record the key-value pair relationship of the text node, where the meaning of the elements is [Key node number, Value node number].

[0058] It can be seen that the KVLabel tool developed through the embodiment of the present invention can realize functions such as importing data sets to be labeled, selecting data to be labeled, performing rectangular selection and labeling, setting the properties of the selected rectangular nodes, and setting key-value relationships between nodes.

[0059] In this embodiment, the table key value recognition process uses a highly structured table as input data. After OCR recognition, the positional information and text information of each text area in the table are obtained, which can be considered as graph data. Furthermore, since OCR technology is relatively mature and can easily recognize text, this embodiment of the present invention focuses on identifying the key value categories of text nodes in a table.

[0060] Before training each module, several perturbations can be added. The default perturbations include: color space conversion (cvtColor), blur, jitter, noise (Gasuss noise), random crop, perspective, color inversion, etc. The data required for training is shown in the following table:

[0061]

[0062] In this embodiment, the trained GKVR recognition model includes: a sentence vector feature extraction module, a node image feature extraction module, and a position feature extraction module.

[0063] The training process of the sentence vector feature extraction module is as follows: according to the preset vocabulary, the text content of each text node in the training sample is recognized to generate a string, and each string is one-hot encoded and then a single-layer feedforward network is applied for word embedding to obtain the word sequence corresponding to each text node; the semantics of each word sequence is learned through the GRU network to generate the sentence vector features of each text node.

[0064] In order to enable the model to obtain the semantic information of the tabular text, this article uses the common processing method of text information in the field of natural language processing. First, a vocabulary table vocab is established. Here, vocab is 26 letters and various symbols. The data type is: string. The first part is relatively ordered ("0123456789abcdefghijklmnopqrstuvwxyz") and is the result of a simple traversal of numbers and lowercase letters. The second part is relatively disordered. In addition to being composed of a variety of symbols and uppercase letters, some Roman numerals are also added, and it is still in the form of a string. The structural feature is a simple horizontal "xxxx" with no special arrangement. Secondly, the characters that do not exist in the vocab are converted into words whose vocab represents unknown symbols.

[0065] Each character is then one-hot encoded and a single-layer feedforward network (existing techniques are not described here) is applied to create a word embedding representation. One-hot encoding uses an N-bit state register to encode N states. Each state is represented by its own independent register bit, and only one bit is active at any given time. One-hot encoding first requires mapping the categorical value to a whole number. Each integer value is represented as a binary vector, with all values zero except for the integer index, which is labeled as 1.

[0066] Finally, GRU is used to learn the semantic information in the word sequence and finally obtain the sentence vector features used to represent the text information of the graph node. The process is as follows:

[0067] word_vector i [j]=embedding(one_hot(test i [j]))

[0068] sentence_feature i [j] = GRU(word_vector i [j])

[0069] Among them, word_vector i [j] represents the character embedding set of the j-th node text in the i-th graph. i [j] is the sentence vector feature. The network parameters for text sentence vector feature representation are shown in the following table:

[0070] parameter value Vocabulary size 105 Word vector size after embedding 64 Sentence vector size 64

[0071] The training process of the node image feature extraction module is specifically as follows: obtaining multiple table frame information in the training sample, and extracting the image structure information of each table frame information through a convolutional neural network to obtain multiple first feature maps; scaling the multiple first feature maps to a grid using a bilinear interpolation method through the grid_simple algorithm, and using the grid features of the coordinates corresponding to each text node as the node image features of the text node.

[0072] For tables, in addition to the position and text information of each text node, the table frame information itself is also of certain value for key-value recognition. The difference in the table frame structure between the key node and the value node indicates that it has certain reference value. In order to better complete the work of key-value recognition, the embodiment of the present invention uses a convolutional neural network (CNN) to extract the image structure information of the table, and then uses the grid_simple algorithm to scale the feature map obtained by the convolutional network to a grid using a bilinear interpolation method and obtain the grid features of the corresponding coordinates of each text node as the image features of the text node. The detailed process is as follows:

[0073] img_feature_map i =CNNs(img i )

[0074] img_feature_box i [j]=grid_simple(img_feature_map i ,pos i [j])

[0075] Among them, img_feature_map i is the image feature map of the i-th table, img_feature_box i [j] is the node image feature of the jth text node in the i-th table. The specific CNNs network parameters are shown in the following table:

[0076] Layer In channel Out channel Kernel size Stride Padding Activation Function 0 1 64 3x3 1 1 Relu 1 64 64 3x3 1 1 Relu 2 64 64 3x3 1 1 Relu

[0077] In this embodiment, the training process of the position feature extraction module is specifically as follows: obtaining the position information of each text node in the training sample; performing coordinate conversion on each position information, and normalizing the coordinate system to the interval [-1,1], and outputting the position feature corresponding to each text node.

[0078] In tabular data, due to the varying sizes of tables, the absolute positions of table nodes with similar structures can vary significantly. Therefore, using absolute positioning directly as network input can result in low learning efficiency. To avoid these issues and enable the network to better learn the table structure, this paper converts the absolute position information of the nodes into relative position information and normalizes the coordinate system to the range of -1 to 1. The process is as follows:

[0079] min_x i =min(X i )

[0080] min_y i =min(Y i )

[0081]

[0082] Among them, X i , Y i They represent the x-coordinate value set and y-coordinate set of the i-th table respectively, and set the absolute position of a node j to {(x1,y1),(x2,y2)}. tabel_width represents the width of the table, and tabel_height represents the height of the table.

[0083] In this embodiment, the sentence vector features, node image features and position features corresponding to each text node in the training sample are used as the input of the GKVR recognition model, and the key information and value information corresponding to each text node are used as the output of the GKVR recognition model; for each text node, the sentence vector features and position features are respectively input into the graph attention network, and after splicing with the node image features, they constitute the node features of each text node. Combined with the output of the GKVR recognition model, the graph attention network and multi-layer perceptron MLP are trained.

[0084] The Graph Attention Network (GAT) uses the self-attention mechanism to determine the weight of neighbor node features during aggregation, thereby achieving weight adaptation for different neighbor nodes and avoiding the influence of the number of neighbor nodes on the output features. Since the dataset does not provide edge information between nodes, if the full connection is used to construct the edge set, the complexity will reach O(|N| 2 ), in order to reduce the complexity and take into account the characteristics of the neighbor nodes of the table nodes having close locations, this paper adopts the nearest neighbor algorithm (K Nearest Neighbor, KNN) to generate the edge set of the table graph, so that the complexity can be reduced to O(K*|N|).

[0085] In the overall step, the nearest neighbor algorithm is used to reduce the complexity, such as Figure 5The "Position of Node" section in the previous section plays a role in the transition to grid simple. Each node on the graph has its own relative position attribute. Using nearest neighbor, we select the k nodes closest to a node, and then set an edge between that node and the other k nodes. This is because graph convolution is only possible with both edge and node sets.

[0086] The calculation process is as follows:

[0087] edges i [j]=KNN(pos i [j])

[0088] pos_h_feature i [j] = GAT θ1 (normlized_pos i [j],edges i [j])

[0089] text_h_feature i [j] = GAT θ2 (sentence_feature i [j],edges i [j])

[0090] h_f i [j] = concat(pos_h_feature i [j],text_h_feature i [j],img_feature_box i [j])

[0091] prediction i [j]=Softmax(MLP((h_f i [j])))

[0092] Among them, edges i Represents the edges of the i-th table obtained by the KNN algorithm. i [j] represents the edge of node j in the i-th table obtained by the KNN algorithm, pos i [j] represents the absolute position of node j, normlized_pos i [j] represents the relative position of node j, sentence_feature i [j] represents the sentence vector of the text information of node j, pos_h_feature i [j] represents the position feature of the jth text node in the i-th table, text_h_featurei [j] represents the text feature of the jth text node in the i-th table, img_feature_box i [j] represents the image feature of the j-th text node, h_f i [j] represents the feature information of node j, prediction i [j] represents the prediction result of node j category; GAT θ1 and GAT θ2 It is used to distinguish GAT's processing of position features and sentence features.

[0093] In order to better illustrate the beneficial effects of this embodiment, it can be verified through comparative experiments based on GCN and GAT. In the model design, the graph neural network is used to enable the nodes on the graph to combine the information of nearby nodes, so as to better infer the type of the node. In GFTE, a key-value row-column relationship derivation model based on graph neural network, GCN is used as the underlying network for its node information aggregation and performs well in its work. However, since the fusion of neighbor nodes by GCN itself is affected by the degree of its neighbor nodes and it is impossible to generate corresponding weights according to the different eigenvalues of different nodes, in the table key derivation task, this article believes that the influence factor of the neighbor node on the central node contains the eigenvalue of the neighbor node, so this article uses GAT as the underlying network for node aggregation and the accuracy and convergence stability of the model are greatly improved when completing the key recognition work.

[0094] like Figure 6 As shown in the figure, the GCN-based GKVR model's loss convergence trend on the training set is basically the same as that of the GAT-based GKVR model, but the former has a larger minimum convergence value. On the test set, the GCN-based GKVR model exhibits strong loss jitter, which shows that using GAT as the underlying network for node information aggregation can improve the convergence stability of the GKVR model.

[0095] like Figure 7 As shown in the figure, the GAT-based GKVR model significantly outperforms the GCN-based GKVR model in terms of recognition accuracy, with its highest accuracy on the training set exceeding the latter by 6 percentage points, and on the test set exceeding the latter by 7 percentage points. Therefore, for table node key value recognition, replacing GCN with GAT is a reasonable solution.

[0096] In this embodiment, after the GKVR recognition model is trained, the first PNG table image will be input and the corresponding first text information, table frame information and position information of each text node will be extracted. Among them, the first text information mainly includes the text content of the table, and the sentence vector feature extraction module generates the sentence vector feature corresponding to the first text information. The node image feature extraction module converts the table frame information into a node image feature, and then the position feature extraction module normalizes the position information of each text node to obtain the position feature. Finally, the sentence vector feature and the position feature corresponding to the first text information are respectively input into the graph attention network, spliced with the node image feature, and then passed through the multi-layer perceptron MLP to output the key-value information set corresponding to the first PNG table image.

[0097] Step 103: According to a preset partitioning rule tree, the key-value information set is traversed and matched, and each key-value pair in the key-value information set is output.

[0098] In this embodiment, step 103 specifically includes: gradually dividing the key information set by traversing the partition rule tree in a breadth-first manner, and selecting a value in the value information set when reaching a leaf node, to generate a plurality of key-value pairs.

[0099] After identifying the key attributes of the nodes in the table, the table node set can be divided into Key={k1, k2, ..., k n}, Value = {v1, v2, ..., v m}, Other={o1,o2,…,o k}Three collections, which mainly discuss how to obtain the corresponding relationship between the elements in the Key collection and the Value collection in the table.

[0100] The existence of key-value pairs between nodes is considered as two categories, and the graph neural network is used to extract features of the nodes on the graph, and finally the problem is converted into a binary classification problem to predict whether there is a key-value pair relationship between nodes. In order to be able to discover all key-value pairs between keys, a reasonable design is to construct the Key set and the Value set into a complete bipartite graph, so as to predict each edge on the bipartite graph.<Node1,Node2> The above solution is a prior art. Designing a binary classification neural network based on this approach will result in an extremely unbalanced distribution of sample labels. Experiments have shown that this imbalance causes the model to classify all edges between nodes as non-key-value pairs, achieving high accuracy. However, the confusion matrix shows that the model is unable to identify key-value relationships in the graph.

[0101] In this embodiment, there is obvious prior knowledge about table key-value matching, such as the Key node and the Value node being in the same row or column; the distance or Euclidean distance between the Value node and its corresponding Key in a certain coordinate system being the minimum, etc. To introduce this prior knowledge into the key-value matching problem, this paper defines the partitioning rule tree PT as follows:

[0102] 1.PT is not empty.

[0103] 2. If a node i in PT is not a leaf node, it contains the partition rule p i .

[0104] If a node i in PT is not a leaf node, then the number of its child nodes is equal to p i The number of set categories to partition the set into.

[0105] 3. If a node i in PT is a leaf node, it contains the selection rule s i .

[0106] The above is to identify key values, and the rule tree algorithm is used to explore the relationship between key and value pairs. i It is a partitioning rule set on the rule tree, which divides the key set into multiple subsets. i Represents the selection rules defined by the rule tree, matching key-value pairs that comply with the unified rules.

[0107] By gradually partitioning the key set through a breadth-first traversal of the rule tree (PT), keys are selected and key-value pairs are generated when reaching leaf nodes. This allows key-value matching for keys that conform to the same rule tree. The matching rule can be expressed as: breadth-first traversal, selecting a subset of the key set according to the rules in the rule tree nodes, and ultimately finding the key corresponding to a specific value. For example, the key set is first partitioned, then matched against the value set in the rule tree, ultimately finding a value that conforms to the same rule tree and generating a key-value pair.

[0108] As an example of this embodiment, the partition-based key-value matching algorithm can be, but is not limited to, as shown in the following table.

[0109]

[0110] In order to better illustrate the application of the partition-based key-value matching algorithm of this embodiment, the following example is used for illustration. Figure 8 ,, define its corresponding partitioning rule tree PT as Figure 8As shown, the root node is the partitioning rule, containing horizontal and vertical sets and a specified range within the scope. The left child node is an element of the horizontal set, and the right child node is an element of the vertical set. Both nodes adhere to the principle of closest proximity for their key-value pairs. Ultimately, key-value pair matching using this rule tree can effectively identify key-value pairs in the SciTSR-Key-Value dataset. D(x, y) is the angle between the edge connecting the x and y nodes and the x-axis.

[0111] As an example of this embodiment, the partitioning rule tree is set in the GKVR recognition model. In this example, the partitioning rule tree is also integrated into the GKVR recognition model to simplify the operation and improve the efficiency.

[0112] On the other hand, an embodiment of the present invention provides an OCR table semantic recognition device based on a graph neural network, comprising: an acquisition unit, a recognition unit, and an output unit;

[0113] The acquisition unit is used to acquire a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table;

[0114] The recognition unit is used to input the first PNG table image into the trained GKVR recognition model, so that the GKVR recognition model performs OCR recognition on the first PNG table image, obtains first text information, table frame information and position information of each text node, and generates a sentence vector feature corresponding to the first text information through a GRU network based on the first text information and a preset vocabulary, and then converts the table frame information into a node image feature through a convolutional neural network and a grid_simple algorithm, and then normalizes the position information of each text node to obtain a position feature, and finally inputs the sentence vector feature and the position feature corresponding to the first text information into a graph attention network respectively, and after splicing with the node image feature, passes through a multi-layer perceptron MLP, and outputs a key-value information set corresponding to the first PNG table image; wherein the key-value information set includes: a key information set and a value information set;

[0115] The output unit is used to traverse and match the key-value information set according to a preset partitioning rule tree, and output each key-value pair in the key-value information set.

[0116] The detailed working principle and process of this device can be found in the relevant records above, but are not limited to

[0117] As can be seen from the above, the embodiment of the present invention provides an OCR table semantic recognition method and device based on graph neural network, which inputs a PNG table image into a trained GKVR recognition model. Through the sentence vector features, node image features and position features of the text nodes in the model, it can accurately determine whether the attribute of the table node is a key or a value; and by setting a partitioning rule tree to achieve matching between keys and values, it can improve the ability to recognize the relationship between the key and value of the table. Compared with the convenient document formats and images that are difficult to directly extract in the prior art, the present invention combines deep learning network structures such as graph neural networks and gated recurrent units, and proposes a GKVR network model for table key value recognition, which can achieve one-click recognition and is an important supplement to the existing, traditional and widely used table recognition methods, meeting the actual needs of industry such as automated table review.

[0118] Furthermore, existing techniques for fusing neighbor nodes using graph convolutional neural networks are influenced by the degree of their neighbors and fail to generate corresponding weights based on the different eigenvalues of different nodes. In the table key value derivation task, because the influence factor of neighbor nodes on the central node should include the eigenvalues of the neighbor nodes, the present invention uses a graph attention network as the underlying network for node aggregation, significantly improving the accuracy and convergence stability of the model when completing key value recognition.

[0119] Furthermore, although some methods have explored identifying key values in tables, tables often contain portable document formats and images that are difficult to directly extract. This paper combines deep learning network structures such as graph neural networks and gated recurrent units to propose a network model for table key value recognition, GKVR (Graph-based Key and Value Recognition). This model can use the text information and location information of the text in the table and the image information of the table image to classify the key value of a node in the table, improving the ability to recognize the relationship between the key and value of the table.

[0120] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for OCR table semantic recognition based on graph neural network, characterized by: include: Obtain a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table; Input the first PNG table image into a trained GKVR recognition model so that the GKVR recognition model performs OCR recognition on the first PNG table image to obtain first text information, table frame information, and position information of each text node, and generate sentence vector features corresponding to the first text information through a GRU network based on the first text information and a preset vocabulary, and then convert the table frame information into node image features through a convolutional neural network and a grid_simple algorithm, and then normalize the position information of each text node to obtain position features, and finally input the sentence vector features and the position features corresponding to the first text information into a graph attention network respectively, and after splicing with the node image features, pass through a multi-layer perceptron MLP, and output a key-value information set corresponding to the first PNG table image; wherein the key-value information set includes: a key information set and a value information set; According to a preset partitioning rule tree, the key-value information set is traversed and matched, and each key-value pair in the key-value information set is output; The training process of the trained GKVR recognition model is as follows: The sentence vector features, node image features, and position features corresponding to each text node in the training sample are used as the input of the GKVR recognition model, and the key information and value information corresponding to each text node are used as the output of the GKVR recognition model; For each text node, the sentence vector feature and position feature are respectively input into the graph attention network, and after being spliced with the node image feature, the node feature of each text node is formed. Combined with the output of the GKVR recognition model, the graph attention network and multi-layer perceptron MLP are trained.

2. The OCR table semantic recognition method based on graph neural network according to claim 1 is characterized in that: The trained GKVR recognition model includes: a sentence vector feature extraction module; The training process of the sentence vector feature extraction module is specifically as follows: According to the preset vocabulary, the text content of each text node in the training sample is recognized to generate a string. After one-hot encoding, each string is applied to a single-layer feedforward network for word embedding to obtain the word sequence corresponding to each text node. The semantics of each word sequence is learned through the GRU network to generate sentence vector features for each text node.

3. The OCR table semantic recognition method based on graph neural network according to claim 2 is characterized in that: The trained GKVR recognition model includes: a node image feature extraction module; The training process of the node image feature extraction module is specifically as follows: Acquire multiple table frame information in the training sample, and extract image structure information from each table frame information through a convolutional neural network to obtain multiple first feature maps; The multiple first feature maps are scaled to a grid using a bilinear interpolation method through the grid_simple algorithm, and the grid features of the coordinates corresponding to each text node are used as the node image features of the text node.

4. The OCR table semantic recognition method based on graph neural network according to claim 3 is characterized in that: The trained GKVR recognition model includes: a position feature extraction module; The training process of the position feature extraction module is specifically as follows: Obtain the location information of each text node in the training sample; The coordinates of each position information are converted, and the coordinate system is normalized to the interval [-1,1], and the position features corresponding to each text node are output.

5. The OCR table semantic recognition method based on graph neural network according to claim 1 is characterized in that: The first PNG table image is obtained by pre-processing the PDF table, specifically: The PDF document to be processed is obtained, and the table portion is intercepted from the PDF document using the KVLabel tool to generate the first PNG table image.

6. The OCR table semantic recognition method based on graph neural network according to claim 5 is characterized in that: The KVLabel tool is also used to pre-process the training samples of the GKVR recognition model, specifically: The KVLabel tool is used to select the table frame of the PDF document in the initial sample, and each text node in the table frame is labeled with key values and key-value pairs to generate a PNG table image corresponding to each initial sample, and all PNG table images, key-value labels and key-value pair labels are used as the training samples.

7. The OCR table semantic recognition method based on graph neural network according to claim 1 is characterized in that: The key-value information set is traversed and matched according to the preset partitioning rule tree, and each key-value pair in the key-value information set is output, specifically: The key information set is gradually divided by traversing the partition rule tree in a breadth-first manner, and when a leaf node is reached, a value in the value information set is selected to generate several key-value pairs.

8. The OCR table semantic recognition method based on graph neural network according to claim 7 is characterized in that: The partitioning rule tree is set in the GKVR recognition model.

9. An OCR table semantic recognition device based on graph neural network, characterized in that: include: Acquisition unit, recognition unit and output unit; The acquisition unit is used to acquire a first PNG table image to be identified; wherein the first PNG table image is obtained by pre-processing a PDF table; The recognition unit is used to input the first PNG table image into a trained GKVR recognition model, so that the GKVR recognition model performs OCR recognition on the first PNG table image to obtain first text information, table frame information and position information of each text node, and generate a sentence vector feature corresponding to the first text information through a GRU network based on the first text information and a preset vocabulary, and then convert the table frame information into a node image feature through a convolutional neural network and a grid_simple algorithm, and then normalize the position information of each text node to obtain a position feature, and finally input the sentence vector feature and the position feature corresponding to the first text information into a graph attention network respectively, and after splicing with the node image feature, pass through a multi-layer perceptron MLP to output a key-value information set corresponding to the first PNG table image; wherein the key-value information set includes: a key information set and a value information set; The output unit is used to traverse and match the key-value information set according to a preset partitioning rule tree, and output each key-value pair in the key-value information set; The training process of the trained GKVR recognition model is as follows: The sentence vector features, node image features, and position features corresponding to each text node in the training sample are used as the input of the GKVR recognition model, and the key information and value information corresponding to each text node are used as the output of the GKVR recognition model; For each text node, the sentence vector feature and position feature are respectively input into the graph attention network, and after being spliced with the node image feature, the node feature of each text node is formed. Combined with the output of the GKVR recognition model, the graph attention network and multi-layer perceptron MLP are trained.

Images