Table recognition model training method and device, table recognition method and device, electronic equipment and computer program product

Abstract

The invention discloses a table recognition model training method and device, a table recognition method and device, electronic equipment and a computer program product, and the training method comprises the steps: obtaining a table image sample, and coding the table image sample into a table structure tree; encoding the table image sample by using an encoder in the table recognition model to obtain table encoding features; decoding the table coding features by using a tree decoder in the table recognition model to obtain a table decoding result; and calculating a loss value of the table recognition model according to the table decoding result and the corresponding table structure tree, and optimizing parameters of the table recognition model by using the loss value of the table recognition model to obtain a trained table recognition model. According to the method, the table is coded by using the tree structure, and then the table structure is decoded by using the tree decoder, so that the table structure information can be fully utilized, and the robustness and the recognition precision of the model are greatly improved in the scenes such as inclination.

Description

Technical Field

[0001] The present application relates to the field of table recognition technology, and in particular to a method and device for training a table recognition model, a table recognition method and device, an electronic device, and a computer program product. Background Art

[0002] In the financial industry, form recognition has a wide range of applications and is of great significance in improving the information level of the financial industry, enhancing data processing capabilities, and strengthening risk control. For example, when processing insurance claims, it is necessary to identify the claim application form for subsequent review and processing; when performing tasks such as customer information entry and contract management, it is necessary to extract information from the form for information management and data analysis.

[0003] In the form recognition of the financial industry, it is mainly divided into two categories according to the different collection methods:

[0004] 1) Scanned forms: This type of form refers to a form image obtained by scanning or printing data and information in a paper or electronic form, which is not prone to tilt.

[0005] 2) Photographed forms: This type of form is usually obtained by photographing a paper document. Due to the photographing angle or the tilt of the paper document itself, this type of form is more seriously tilted.

[0006] In actual applications, due to various reasons, the above two forms may be deformed or interfered, such as broken table lines, overlapping text, etc., which puts higher requirements on table recognition. In addition, since the table data in the financial industry often involves commercial secrets and other issues, there are also extremely high requirements for the accuracy, security and efficiency of table recognition.

[0007] Current form recognition solutions are mainly based on image processing and deep learning algorithms.

[0008] 1) Table recognition based on image processing: This method usually includes preprocessing, edge detection, row and column recognition, cell recognition, and content recognition. This is a traditional method. For layout analysis and table structure extraction, the image processing method relies on the selection of various thresholds and parameters, which is difficult to meet the complex and diverse table scenarios in real life, and the robustness is relatively poor.

[0009] 2) Table recognition based on deep learning: With the maturity of artificial intelligence applications, deep learning algorithms are increasingly used in table recognition. Most current solutions first detect the text lines in the table, extract the visual, semantic and positional features of the text lines, and then use various methods to perform feature fusion and feature enhancement. Finally, they classify the two lines to analyze the three categories of whether they are in the same row, column and unit. This type of method often works well in scanning tables, but the effect will be greatly reduced when taking photos of tilted tables. Summary of the invention

[0010] Embodiments of the present application provide a method and device for training a table recognition model, a table recognition method and device, an electronic device, and a computer program product to improve the accuracy and robustness of table recognition.

[0011] The present application embodiment adopts the following technical solutions:

[0012] In a first aspect, an embodiment of the present application provides a method for training a table recognition model, the method for training a table recognition model comprising:

[0013] Acquire a table image sample, and encode the table image sample into a table structure tree;

[0014] Using an encoder in a table recognition model to encode the table image sample to obtain table encoding features;

[0015] Decoding the table encoding features using a tree decoder in a table recognition model to obtain a table decoding result;

[0016] The loss value of the table recognition model is calculated according to the table decoding result and the corresponding table structure tree, and the loss value of the table recognition model is used to optimize the parameters of the table recognition model to obtain a trained table recognition model.

[0017] Optionally, encoding the table image sample into a table structure tree comprises:

[0018] Define each text line in each cell of the table image sample as a child node, and encode each child node to obtain multiple encoded child nodes;

[0019] According to the predefined parent node encoding rule, determine the parent node corresponding to each encoded child node;

[0020] Determine the node relationship between each encoded child node and the corresponding parent node according to the predefined node relationship;

[0021] Constructing a spatial relationship library according to the encoded multiple child nodes, the parent node corresponding to each encoded child node, and the node relationship between each encoded child node and the corresponding parent node;

[0022] The spatial relationship library is used to encode a table structure tree.

[0023] Optionally, the step of defining each text line in each cell of the table image sample as a child node and encoding each child node to obtain the encoded multiple child nodes includes:

[0024] If a cell in the table image sample contains multiple text lines, encoding each text line according to the actual reading order;

[0025] If there are cross-row cells or cross-column cells in the table image sample, the first text line in the cross-row cells is encoded multiple times according to the number of cross-rows, or the first text line in the cross-column cells is encoded multiple times according to the number of cross-columns.

[0026] Optionally, the encoding process of the table image sample using an encoder in the table recognition model to obtain the table encoding feature includes:

[0027] Processing the table image sample using an optical character recognition model to obtain a first text line feature;

[0028] Processing the table image sample using the encoder to obtain encoding features of the table image sample;

[0029] Using the first text line feature to crop the encoding feature of the table image sample to obtain a second text line feature;

[0030] A context feature of the text line is generated according to the first text line feature and the second text line feature.

[0031] Optionally, the tree decoder includes a parent node decoder, a child node decoder and a node relationship decoder, and the tree decoder in the table recognition model is used to decode the table encoding feature to obtain a table decoding result including:

[0032] According to the encoding features of the table, using the parent node decoder to perform parent node decoding processing to obtain a parent node decoding result;

[0033] According to the parent node decoding result and the table encoding feature, using the child node decoder to perform child node decoding processing to obtain a child node decoding result;

[0034] The node relationship decoding process is performed using the node relationship decoder according to the parent node decoding result and the child node decoding result to obtain a node relationship decoding result.

[0035] Optionally, the calculating the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree comprises:

[0036] Calculate the loss value of the parent node decoder according to the parent node decoding result and the parent node true value in the corresponding table structure tree;

[0037] Calculating the loss value of the sub-node decoder according to the sub-node decoding result and the sub-node true value in the corresponding table structure tree;

[0038] The loss value of the node relationship decoder is calculated according to the node relationship decoding result and the node relationship true value in the corresponding table structure tree.

[0039] Optionally, the optimizing the parameters of the table recognition model by using the loss value of the table recognition model to obtain a trained table recognition model includes:

[0040] Performing weighted fusion on the loss value of the parent node decoder, the loss value of the child node decoder, and the loss value of the node relationship decoder to obtain a fused loss value;

[0041] The fused loss value is used to optimize the parameters of the table recognition model to obtain a trained table recognition model.

[0042] In a second aspect, an embodiment of the present application further provides a table recognition method, the table recognition method comprising:

[0043] Obtaining a table image to be recognized;

[0044] Inputting the table image to be recognized into a table recognition model to obtain a table recognition result;

[0045] The table recognition model is trained based on any of the training methods for the table recognition model described above.

[0046] In a third aspect, an embodiment of the present application further provides a training device for a table recognition model, the training device for the table recognition model comprising:

[0047] A construction unit, used for acquiring a table image sample and encoding the table image sample into a table structure tree;

[0048] An encoding unit, used for encoding the table image sample using an encoder in a table recognition model to obtain a table encoding feature;

[0049] A decoding unit, used for decoding the table encoding feature using a tree decoder in a table recognition model to obtain a table decoding result;

[0050] The optimization unit is used to calculate the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree, and optimize the parameters of the table recognition model using the loss value of the table recognition model to obtain a trained table recognition model.

[0051] In a fourth aspect, an embodiment of the present application further provides a table recognition device, the table recognition device comprising:

[0052] An acquisition unit, used for acquiring a table image to be recognized;

[0053] A recognition unit, used for inputting the table image to be recognized into a table recognition model to obtain a table recognition result;

[0054] Wherein, the table recognition model is trained based on the training device of the aforementioned table recognition model.

[0055] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0056] Processor; and

[0057] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform any of the aforementioned table recognition model training methods, or to perform the aforementioned table recognition method.

[0058] In a sixth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements any of the aforementioned training methods for the table recognition model, or implements the aforementioned table recognition method.

[0059] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the training method of the table recognition model in the embodiments of the present application first obtains a table image sample, and encodes the table image sample into a table structure tree; then uses the encoder in the table recognition model to encode the table image sample to obtain the table encoding feature; then uses the tree decoder in the table recognition model to decode the table encoding feature to obtain the table decoding result; finally, the loss value of the table recognition model is calculated according to the table decoding result and the corresponding table structure tree, and the loss value of the table recognition model is used to optimize the parameters of the table recognition model to obtain a trained table recognition model. The training method of the table recognition model in the embodiments of the present application first encodes the table using a tree structure, and then uses a tree decoder to decode the structural information of the table, which can make full use of the structural information of the table, greatly improves the robustness of the model in scenes such as tilt, and also improves the recognition accuracy in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 A schematic diagram of a training process of a table recognition model in an embodiment of the present application;

[0062] Figure 2 This is a schematic diagram of a table in an embodiment of the present application;

[0063] Figure 3 This is a schematic diagram of the structure of an encoder in an embodiment of the present application;

[0064] Figure 4 This is a schematic diagram of the structure of a tree decoder in an embodiment of the present application;

[0065] Figure 5 A flowchart of a table recognition method in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of a training device for a table recognition model in an embodiment of the present application;

[0067] Figure 7 This is a structural diagram of a table recognition device in an embodiment of the present application;

[0068] Figure 8 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0070] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0071] In order to facilitate the understanding of the embodiments of the present application, the main technical terms involved in the present application are first introduced:

[0072] 1) Table recognition: Table recognition is generally divided into two parts: table structure recognition and table text recognition. Table structure recognition analyzes the table area, extracts the data and structural information in the table, and obtains the row and column distribution and logical structure (same row, same column, same cell information). Table text recognition is to extract the text content in the table cell frame. Generally speaking, table recognition refers specifically to table structure recognition.

[0073] 2) Tree decoder: Tree decoder is a concept in deep learning, which is mainly used to process sequence data. In the sequence-to-sequence (Seq2Seq) model, the tree decoder can provide a more structured output. In the traditional sequence-to-sequence model, a recurrent neural network (RNN) or a long short-term memory network (LSTM) is usually used to process the input sequence and generate the output sequence. However, when the output sequence has a specific structure (such as a syntax tree or a decision tree), the use of a tree decoder can better represent this structure. Tree decoders usually use a top-down approach for decoding. First, the decoder predicts the root node of the output sequence, and then decodes it step by step until the leaf node is predicted. At each step, the decoder selects the most likely node as the next output based on the currently generated subtree and context information.

[0074] 3) Deep learning: This application involves the field of deep learning, which refers to a set of algorithms that use various machine learning algorithms on multi-layer neural networks to solve various problems such as images and texts. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through a layered network, thereby solving important problems that previously required artificially designed features.

[0075] 4) OCR: Optical Character Recognition (OCR) technology. This application relates to OCR technology, which refers to the processing and analysis of images to identify text information in images. The OCR training process includes two models, the text detection model and the text recognition model. In the inference stage, these two models are combined to build a complete image and text recognition system.

[0076] The present application embodiment provides a method for training a table recognition model, such as Figure 1 As shown, a schematic diagram of a training process of a table recognition model in an embodiment of the present application is provided, and the training method of the table recognition model at least includes the following steps S110 to S140:

[0077] Step S110, obtaining a table image sample, and encoding the table image sample into a table structure tree.

[0078] When training a table recognition model, it is necessary to first obtain sample data for training the model, namely table image samples, and then construct spatial structural relationships based on the annotation information in the table image samples to encode the table image samples into the form of a table structure tree, so as to facilitate the subsequent extraction of the table's structural information. The encoding process here is equivalent to converting the original annotated table image samples into the form of a table structure tree, which is used as a supervisory signal for subsequent model training to train the model's ability to extract table structures.

[0079] Step S120: Encode the table image sample using an encoder in the table recognition model to obtain table encoding features.

[0080] The table recognition model trained in the embodiment of the present application adopts the encoder-decoder paradigm to extract the structural information of the table. Therefore, after obtaining the table image sample, it is necessary to use the encoder to encode the table image sample to obtain the table encoding features.

[0081] Step S130: using the tree decoder in the table recognition model to decode the table encoding feature to obtain a table decoding result.

[0082] After the encoding process is completed, the tree decoder in the table recognition model is used to decode the table encoding features. The tree decoder decodes the structural features of the table according to the tree structure, and pays more attention to the structural information of the table.

[0083] Step S140, calculating the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree, and optimizing the parameters of the table recognition model using the loss value of the table recognition model to obtain a trained table recognition model.

[0084] The above table decoding result is the prediction result of the table recognition model in the training phase. Combined with the true value information of the table structure tree constructed based on the spatial relationship library, the loss value of the model recognition result can be calculated. The loss value is used to iteratively optimize the parameters of the model so that the model recognition result can be closer and closer to the true value. When the application requirements are met, the iteration is stopped and the trained table recognition model is output.

[0085] The training method of the table recognition model in the embodiment of the present application first uses a tree structure to encode the table, and then uses a tree decoder to decode the structural information of the table. This can make full use of the structural information of the table, greatly improve the robustness of the model in scenarios such as tilt, and also improve the recognition accuracy in practical applications.

[0086] In some embodiments of the present application, encoding the table image sample into a table structure tree includes: defining each text row in each cell of the table image sample as a child node, and encoding each child node to obtain multiple encoded child nodes; determining the parent node corresponding to each encoded child node according to a predefined parent node encoding rule; determining the node relationship between each encoded child node and the corresponding parent node according to a predefined node relationship; constructing a spatial relationship library according to the encoded multiple child nodes, the parent node corresponding to each encoded child node, and the node relationship between each encoded child node and the corresponding parent node; and encoding the table structure tree using the spatial relationship library.

[0087] Based on the characteristics of the table structure, when encoding the table structure tree, the [child node id, parent node id, child node attribute, node spatial relationship] four-tuple is first defined, that is, all structural information in a table can be represented by such a four-tuple, each four-tuple information can construct a subtree, and the list of all subtrees together constitutes a complete table structure tree.

[0088] Based on the above-defined quaternary group, the encoding process of each part of the information in the quaternary group is further described in detail:

[0089] (1) Child nodes

[0090] The embodiment of the present application uses text lines as the basic unit frame to encode the table structure relationship. Assuming that there are multiple text lines in the same cell, the first text line is called text line F, and the other text lines are called text lines N. If there is only one text in the cell, it is text line F. In addition, if there is no text line in the cell, blank text lines also need to be detected. During training, the annotator needs to mark the reading order between text lines as the text line ID, which represents the order in which it is read, and is also the decoding order of the model.

[0091] Assuming that a table has M rows and N columns, the reading order defined in the embodiment of the present application is to start from the 1st row and the 1st column, first read in sequence to the 1st row and the Nth column, then jump to the 2nd row and the 1st column, then read in sequence to the 2nd row and the Nth column, and so on, all the way to the Mth row and the Nth column.

[0092] In order to facilitate the understanding of the embodiments of the present application, Figure 2 As shown, a table schematic diagram is provided in an embodiment of the present application. Figure 2 The part marked by the rectangular box is the basic unit box, and its regular sorting order (from top to bottom, from left to right) is text1 to text6, but its reading order is: text1, text2, text3, text4, text6, text5, text4 (the text line ID is to omit the preceding text).

[0093] In addition to encoding the subnode ID, it is also necessary to encode the attributes of the subnode. The subnode attributes defined in the embodiment of the present application are divided into two categories: empty text and non-empty text.

[0094] (2) Parent node

[0095] The parent node can be divided into the following four cases according to the situation of the child node:

[0096] 1) The first text line in the first row and first column, its parent node is the root node;

[0097] 2) If there are multiple text lines in a cell, except for the first text line, the parent nodes of the other text lines are the previous text line in the reading order;

[0098] 3) If it is the first text line in the first column but not the first row, its parent node is the first text line in the previous column;

[0099] 4) For text lines in other cases, their parent node is the previous text line F in the reading order.

[0100] (3) Relationship

[0101] The embodiment of the present application defines the following relationships between nodes: same row (sr), same column (sl), same unit (su), start (start), end (end). The relationship of the first decoded text line is start, and the relationship of the last decoded text line box is end. The encoding end will add an additional all-zero feature F0. During testing, when the End relationship is decoded, the model stops decoding and outputs a structure tree. Other text line relationships are divided into the following three cases:

[0102] 1) If there are multiple text lines in the cell, except for the first text line, the relationship between the other text lines and the parent node is su;

[0103] 2) If it is the first text line in the first column but not the first row, its relationship with the parent node is sl;

[0104] 3) For other text lines, the relationship between them and the parent node is sr.

[0105] Based on the above information, a table structure tree can be constructed for any table. Figure 2 Taking the table as an example, the table structure tree obtained after encoding is as follows:

[0106] [1,root,non-empty text,start]

[0107] [2,1,non-empty text,sr]

[0108] [3,1,non-empty text,sl]

[0109] [4,3,non-empty text,sr]

[0110] [6,4, non-empty text, su]

[0111] [5,3,non-empty text,sl]

[0112] [4,5,non-empty text,sr]

[0113] [F0,4, non-empty text, end]

[0114] Based on the above information, a spatial relationship library is constructed, from which all structural information in the table can be obtained, including the same row, column, unit information, etc., and then the structure of the table can be restored through post-processing rules.

[0115] In some embodiments of the present application, each text row in each cell of the table image sample is defined as a child node, and each child node is encoded to obtain multiple encoded child nodes, including: if a cell in the table image sample contains multiple text rows, each text row is encoded according to the actual reading order; if there are cross-row cells or cross-column cells in the table image sample, the first text row in the cross-row cell is encoded multiple times according to the number of cross-rows, or the first text row in the cross-column cell is encoded multiple times according to the number of cross-columns.

[0116] In the process of encoding child nodes, the following two situations may occur:

[0117] 1) There are multiple lines of text in a cell: In this case, the text can be encoded in the actual reading order. For example Figure 2In the example, text4 and text6 belong to different text lines in a cell, so text4 is encoded first and then text6 according to the reading order.

[0118] 2) There are cells that span rows or columns in the table: In this case, the first text line in the cells that span rows or columns needs to be encoded multiple times. The same applies to the number of rows and columns. Figure 2 In the example, text4 and text6 belong to two spanning text lines in the same cell and span two lines. Then, the first text line in the cell, i.e., text4, is encoded twice. Referring to the reading order given in the above embodiment: text1, text2, text3, text4 ,text6,text5, text4 .

[0119] In some embodiments of the present application, the encoding process of the table image sample using the encoder in the table recognition model to obtain the table encoding feature includes: processing the table image sample using an optical character recognition model to obtain a first text row feature; processing the table image sample using the encoder to obtain the encoding feature of the table image sample; cropping the encoding feature of the table image sample using the first text row feature to obtain a second text row feature; and generating a context feature of the text row based on the first text row feature and the second text row feature.

[0120] like Figure 3 As shown, a schematic diagram of the structure of an encoder in an embodiment of the present application is provided. When the encoder is used to encode the table image sample, the table image sample can be first input into the encoders of the OCR model and the table recognition model respectively, and the first text line features of the table are extracted by the OCR model, including text line position features and content features, etc., and the encoding features of the table image sample are extracted by the encoder. The encoder here can be implemented based on a network such as ResNet50, for example.

[0121] Since the OCR model extracts features of the text line dimension and the encoder extracts the visual feature map of the entire table image, the features of the corresponding positions on the feature map are cropped using the text line position coordinates extracted by the OCR model to obtain the visual features of the text line. The text lines can be sorted in order from top to bottom and from left to right.

[0122] Then, the visual features of the above text line are spliced ​​with the position features and semantic features extracted by the OCR model to obtain the fused text line features. When splicing features, the position features can be the features after the upper left corner and lower right corner coordinates are projected, and the semantic features can be the features obtained by inputting the transcription results of the OCR model into Bert.

[0123] Finally, the fused text line features are input into a bidirectional GRU (Gated Recurrent Unit) network, which is a special recurrent neural network (RNN) architecture that can simultaneously consider forward and backward context information when processing sequence data. The embodiment of the present application uses a bidirectional GRU network to further extract the context information of the text line, obtain the context features of the text line, and thus enhance the robustness of the features.

[0124] In some embodiments of the present application, the tree decoder includes a parent node decoder, a child node decoder and a node relationship decoder, and the tree decoder in the table recognition model is used to decode the table encoding features to obtain the table decoding result, including: according to the table encoding features, using the parent node decoder to perform parent node decoding processing to obtain the parent node decoding result; according to the parent node decoding result and the table encoding features, using the child node decoder to perform child node decoding processing to obtain the child node decoding result; according to the parent node decoding result and the child node decoding result, using the node relationship decoder to perform node relationship decoding processing to obtain the node relationship decoding result.

[0125] like Figure 4 As shown, a schematic diagram of the structure of a tree decoder in an embodiment of the present application is provided. The tree decoder in the embodiment of the present application is mainly composed of a parent node decoder, a child node decoder and a node relationship decoder.

[0126] (1) Parent node decoder

[0127] The hidden state vector of the subnode text line at the previous moment is The embedding of its category is Input both into the pre Parent GRU module to obtain the intermediate hidden state vector of the parent node

[0128]

[0129] Then The context feature map A of the text line is input into the attention module of the parent node to obtain the attention probability of the parent node According to the attention probability of the parent node And the context feature map A of the text line obtains the context feature expression vector of the current parent node

[0130]

[0131]

[0132] in, is the attention probability of the parent node, is the context feature expression vector of the parent node, a i is the i-th value of feature map A. The specific implementation is as follows:

[0133]

[0134] in, These are the parameters that the parent node decoder needs to learn. is the attention probability of the parent node, and the i-th value on the feature map corresponds to the i-th text row in the table, so It itself corresponds to the parent node ID.

[0135] Finally, the intermediate hidden state vector of the parent node is and context feature expression vector Send them to the ParentGRU module together to obtain the final parent node hidden layer state vector

[0136]

[0137] (2) Subnode decoder

[0138] In the parent node decoder, the parent node ID and hidden state vector are solved. After that, since the parent node is the child node solved by history, the feature representation of its attributes can be obtained synchronously (The true value groundtruth is used during training, and the attribute decoded by the parent node is used during testing). Both are sent to the Child GRU module to solve the intermediate hidden state vector of the child node.

[0139]

[0140] Then The context feature map A of the text line is input into the attention module of the child node to obtain the context feature expression vector of the child node at the current moment

[0141]

[0142]

[0143] in, is the attention probability of the child node, is the context feature expression vector of the child node, a i is the i-th value of feature map A. The specific implementation is as follows:

[0144]

[0145] in, are the parameters that the child node decoder needs to learn. is the attention probability of the child node, and the i-th value on the feature map corresponds to the i-th text row in the table, so It itself corresponds to the current child node ID.

[0146] Next, the intermediate hidden state vector of the child node and context feature expression vector Input the ChildGRU module together to obtain the final child node hidden layer state vector

[0147]

[0148] Finally, the text line attribute of the child node at the current moment Its hidden state vector and context feature expression vector Solve together:

[0149]

[0150] in, are the parameters to be learned.

[0151] (3) Node Relation Decoder

[0152] Since the context feature expression vector contains the semantic information and spatial position information of the current node, after solving the context feature expression vectors of the parent node and the child node at the current moment, the context feature expression vectors and the hidden layer state vector of the parent node and the child node can be directly sent to the classification layer to predict and solve the structural relationship between the parent node and the child node, which is specifically expressed as follows:

[0153]

[0154] in, Parameters to be learned for the node relationship decoder.

[0155] In some embodiments of the present application, calculating the loss value of the table recognition model based on the table decoding result and the corresponding table structure tree includes: calculating the loss value of the parent node decoder based on the parent node decoding result and the parent node truth value in the corresponding table structure tree; calculating the loss value of the child node decoder based on the child node decoding result and the child node truth value in the corresponding table structure tree; calculating the loss value of the node relationship decoder based on the node relationship decoding result and the node relationship truth value in the corresponding table structure tree.

[0156] Based on the above embodiments, the decoding results of the parent node, the decoding results of the child node, and the decoding results of the node relationship can be obtained respectively. Combined with the true value of the table structure information in the table structure tree, the loss value of the parent node decoder, the loss value of the child node decoder, and the loss value of the node relationship decoder can be calculated respectively. Specifically, the calculation can be performed in the following manner:

[0157] (1) The classification loss function of the parent node is:

[0158]

[0159] in, Represents the true value of the parent node at the tth moment. The loss value of the parent node decoder can be calculated by the above formula.

[0160] (2) The classification loss function of the sub-node and the classification loss function of the sub-node attribute are:

[0161]

[0162]

[0163] in, Represents the true value of the child node attribute at the tth moment. The loss value of the child node decoder can be calculated by the above formula.

[0164] (3) The classification loss function of the spatial structure relationship is:

[0165] ξ re =-∑ t log p re (v t ), (18)

[0166] Among them, v t It represents the true value of the spatial structure relationship at the tth moment. The loss value of the node relationship decoder can be calculated by the above formula.

[0167] In some embodiments of the present application, the use of the loss value of the table recognition model to optimize the parameters of the table recognition model to obtain a trained table recognition model includes: weighted fusion of the loss value of the parent node decoder, the loss value of the child node decoder, and the loss value of the node relationship decoder to obtain a fused loss value; and using the fused loss value to optimize the parameters of the table recognition model to obtain a trained table recognition model.

[0168] The loss value of the parent node decoder, the loss value of the child node decoder and the loss value of the node relationship decoder calculated in the above embodiment are further weighted and fused to obtain a total loss value, which can be specifically expressed as follows:

[0169] Loss=λ1ξ p +λ2ξ c +λ3ξ c_ly +λ4ξ re , (19)

[0170] Among them, λ1, λ2, λ3, and λ4 are configurable fusion weights.

[0171] The above total loss value is used to iteratively optimize the parameters of the model so that the finally trained model can fully recognize structural information such as parent nodes, child nodes, and the relationship between nodes in the table.

[0172] The present application also provides a table recognition method, such as Figure 5 As shown, a flow chart of a table recognition method in an embodiment of the present application is provided, and the table recognition method includes the following steps S510 to S520:

[0173] Step S510, obtaining a table image to be recognized;

[0174] Step S520, inputting the table image to be recognized into a table recognition model to obtain a table recognition result;

[0175] The table recognition model is trained based on any of the training methods for the table recognition model described above.

[0176] In the application stage of the table recognition model, the table image to be recognized can be directly obtained and input into the trained table recognition model to obtain the final table recognition result.

[0177] The present application embodiment also provides a training device 600 for a table recognition model, such as Figure 6As shown, a schematic diagram of the structure of a training device for a table recognition model in an embodiment of the present application is provided, wherein the training device 600 for the table recognition model includes: a construction unit 610, an encoding unit 620, a decoding unit 630, and an optimization unit 640, wherein:

[0178] A construction unit 610 is used to obtain a table image sample and encode the table image sample into a table structure tree;

[0179] The encoding unit 620 is used to encode the table image sample using an encoder in the table recognition model to obtain a table encoding feature;

[0180] A decoding unit 630, configured to decode the table encoding feature using a tree decoder in a table recognition model to obtain a table decoding result;

[0181] The optimization unit 640 is used to calculate the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree, and optimize the parameters of the table recognition model using the loss value of the table recognition model to obtain a trained table recognition model.

[0182] In some embodiments of the present application, the construction unit 610 is specifically used to: define each text line in each cell of the table image sample as a child node, and encode each child node to obtain multiple encoded child nodes; determine the parent node corresponding to each encoded child node according to a predefined parent node encoding rule; determine the node relationship between each encoded child node and the corresponding parent node according to a predefined node relationship; construct a spatial relationship library according to the encoded multiple child nodes, the parent node corresponding to each encoded child node, and the node relationship between each encoded child node and the corresponding parent node; and use the spatial relationship library to encode the table structure tree.

[0183] In some embodiments of the present application, the encoding unit 620 is specifically used to: if a cell in the table image sample contains multiple text lines, encode each text line according to the actual reading order; if there are cross-row cells or cross-column cells in the table image sample, encode the first text line in the cross-row cell multiple times according to the number of cross-rows, or encode the first text line in the cross-column cell multiple times according to the number of cross-columns.

[0184] In some embodiments of the present application, the encoding unit 620 is specifically used to: process the table image sample using an optical character recognition model to obtain a first text line feature; process the table image sample using the encoder to obtain an encoding feature of the table image sample; crop the encoding feature of the table image sample using the first text line feature to obtain a second text line feature; and generate a context feature of the text line based on the first text line feature and the second text line feature.

[0185] In some embodiments of the present application, the tree decoder includes a parent node decoder, a child node decoder and a node relationship decoder, and the decoding unit 630 is specifically used to: perform parent node decoding processing using the parent node decoder according to the table encoding characteristics to obtain a parent node decoding result; perform child node decoding processing using the child node decoder according to the parent node decoding result and the table encoding characteristics to obtain a child node decoding result; perform node relationship decoding processing using the node relationship decoder according to the parent node decoding result and the child node decoding result to obtain a node relationship decoding result.

[0186] In some embodiments of the present application, the optimization unit 640 is specifically used to: calculate the loss value of the parent node decoder based on the parent node decoding result and the corresponding parent node truth value in the table structure tree; calculate the loss value of the child node decoder based on the child node decoding result and the corresponding child node truth value in the table structure tree; calculate the loss value of the node relationship decoder based on the node relationship decoding result and the corresponding node relationship truth value in the table structure tree.

[0187] In some embodiments of the present application, the optimization unit 640 is specifically used to: perform weighted fusion on the loss value of the parent node decoder, the loss value of the child node decoder, and the loss value of the node relationship decoder to obtain a fused loss value; and use the fused loss value to optimize the parameters of the table recognition model to obtain a trained table recognition model.

[0188] It can be understood that the above-mentioned table recognition model training device can implement the various steps of the table recognition model training method provided in the aforementioned embodiment. The relevant explanations about the table recognition model training method are applicable to the table recognition model training device and will not be repeated here.

[0189] The present application embodiment also provides a table recognition device 700, such as Figure 7 As shown, a schematic diagram of the structure of a table recognition device in an embodiment of the present application is provided, wherein the table recognition device 700 includes: an acquisition unit 710 and a recognition unit 720, wherein:

[0190] An acquisition unit 710 is used to acquire a table image to be recognized;

[0191] The recognition unit 720 is used to input the table image to be recognized into a table recognition model to obtain a table recognition result;

[0192] Wherein, the table recognition model is trained based on the training device of the aforementioned table recognition model.

[0193] It can be understood that the above-mentioned table recognition device can implement each step of the table recognition method provided in the above-mentioned embodiment, and the relevant explanations about the table recognition method are applicable to the table recognition device and will not be repeated here.

[0194] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 8 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0195] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0196] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0197] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a training device or a table recognition device for the table recognition model at the logical level. The processor executes the program stored in the memory.

[0198] The above application Figure 1 The method performed by the training device of the table recognition model disclosed in the illustrated embodiment and Figure 4 The method performed by the table recognition device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0199] An embodiment of the present application further proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements any of the aforementioned table recognition model training methods or the aforementioned table recognition methods.

[0200] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0205] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0206] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0207] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0208] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0209] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for training a table recognition model, characterized in that: The training method of the table recognition model includes: Acquire a table image sample, and encode the table image sample into a table structure tree; Using an encoder in a table recognition model to encode the table image sample to obtain table encoding features; Decoding the table encoding features using a tree decoder in a table recognition model to obtain a table decoding result; The loss value of the table recognition model is calculated according to the table decoding result and the corresponding table structure tree, and the loss value of the table recognition model is used to optimize the parameters of the table recognition model to obtain a trained table recognition model.

2. The method for training a table recognition model according to claim 1, characterized in that: The encoding of the table image sample into a table structure tree comprises: Define each text line in each cell of the table image sample as a child node, and encode each child node to obtain multiple encoded child nodes; According to the predefined parent node encoding rule, determine the parent node corresponding to each encoded child node; Determine the node relationship between each encoded child node and the corresponding parent node according to the predefined node relationship; Constructing a spatial relationship library according to the encoded multiple child nodes, the parent node corresponding to each encoded child node, and the node relationship between each encoded child node and the corresponding parent node; The spatial relationship library is used to encode a table structure tree.

3. The method for training a table recognition model according to claim 2, characterized in that: The method defines each text line in each cell of the table image sample as a child node, and encodes each child node to obtain a plurality of encoded child nodes, including: If a cell in the table image sample contains multiple text lines, encoding each text line according to the actual reading order; If there are cross-row cells or cross-column cells in the table image sample, the first text line in the cross-row cells is encoded multiple times according to the number of cross-rows, or the first text line in the cross-column cells is encoded multiple times according to the number of cross-columns.

4. The method for training a table recognition model according to claim 1, characterized in that: The encoding process of the table image sample by using the encoder in the table recognition model to obtain the table encoding features includes: Processing the table image sample using an optical character recognition model to obtain a first text line feature; Processing the table image sample using the encoder to obtain encoding features of the table image sample; Using the first text line feature to crop the encoding feature of the table image sample to obtain a second text line feature; A context feature of the text line is generated according to the first text line feature and the second text line feature.

5. The method for training a table recognition model according to claim 1, characterized in that: The tree decoder includes a parent node decoder, a child node decoder and a node relationship decoder. The tree decoder in the table recognition model is used to decode the table encoding feature to obtain a table decoding result including: According to the encoding features of the table, using the parent node decoder to perform parent node decoding processing to obtain a parent node decoding result; According to the parent node decoding result and the table encoding feature, using the child node decoder to perform child node decoding processing to obtain a child node decoding result; The node relationship decoding process is performed using the node relationship decoder according to the parent node decoding result and the child node decoding result to obtain a node relationship decoding result.

6. The method for training a table recognition model according to claim 5, characterized in that: The calculating the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree comprises: Calculate the loss value of the parent node decoder according to the parent node decoding result and the parent node true value in the corresponding table structure tree; Calculating the loss value of the sub-node decoder according to the sub-node decoding result and the sub-node true value in the corresponding table structure tree; The loss value of the node relationship decoder is calculated according to the node relationship decoding result and the node relationship true value in the corresponding table structure tree.

7. The method for training a table recognition model according to claim 6, characterized in that: The step of optimizing the parameters of the table recognition model by using the loss value of the table recognition model to obtain a trained table recognition model comprises: Performing weighted fusion on the loss value of the parent node decoder, the loss value of the child node decoder, and the loss value of the node relationship decoder to obtain a fused loss value; The fused loss value is used to optimize the parameters of the table recognition model to obtain a trained table recognition model.

8. A table recognition method, characterized in that: The table recognition method comprises: Obtaining a table image to be recognized; Inputting the table image to be recognized into a table recognition model to obtain a table recognition result; Wherein, the table recognition model is trained based on the training method of the table recognition model described in any one of claims 1 to 7.

9. A training device for a table recognition model, characterized in that: The training device of the table recognition model comprises: A construction unit, used for acquiring a table image sample and encoding the table image sample into a table structure tree; An encoding unit, used for encoding the table image sample using an encoder in a table recognition model to obtain a table encoding feature; A decoding unit, used for decoding the table encoding feature using a tree decoder in a table recognition model to obtain a table decoding result; The optimization unit is used to calculate the loss value of the table recognition model according to the table decoding result and the corresponding table structure tree, and optimize the parameters of the table recognition model using the loss value of the table recognition model to obtain a trained table recognition model.

10. A table recognition device, characterized in that: The table recognition device comprises: An acquisition unit, used for acquiring a table image to be recognized; A recognition unit, used for inputting the table image to be recognized into a table recognition model to obtain a table recognition result; Wherein, the table recognition model is trained based on the table recognition model training device according to claim 9.

11. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to execute the method for training a table recognition model according to any one of claims 1 to 7, or to execute the table recognition method according to claim 8.

12. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the training method of the table recognition model described in any one of claims 1 to 7 is implemented, or the table recognition method described in claim 8 is implemented.

Images