A method and device for table parsing of heterogeneous document images

Abstract

This invention discloses a method and device for table parsing of heterogeneous document images, relating to the field of computer vision. The method includes: extracting the coordinates of the four vertices of a table based on a table detection algorithm determined by a PP-LCNet network, a CornerNet-10 region proposal generation module, and a Fast R-CNN module; extracting the coordinates of the four points of text boxes within the table using a text detection algorithm based on the coordinates of the four vertices; extracting the content of the text boxes using a text recognition algorithm based on the coordinates of the four points; obtaining the coordinates of the four points of cells and table structure information using a table structure prediction model based on the coordinates of the four vertices; aggregating cell coordinates; aggregating cells and text; and combining the aggregated text and cells with the table structure information to obtain a complete table structure. This invention can improve the efficiency and accuracy of information entry and reduce overhead.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for table parsing of heterogeneous document images. Background Technology

[0002] In a narrow sense, a table generally refers to a page with a closed rectangular frame structure composed of horizontal and vertical lines, accompanied by simple text instructions to guide the entry of information. In a broader sense, any page that guides people to fill in a certain type of information is called a table. Tables are characterized by highly structured data and the expression of information according to a certain logic. Table parsing typically refers to the process of identifying and extracting tables from documents, images, or other data sources. In daily life, people generally first use table processing tools to display the table data, and then perform information extraction operations. This manual method of table recognition brings some problems:

[0003] (1) Because there are a large number of tables and they are of various and complex types, during the process of manually updating the data, the same content may be written into different tables, or the content in some cells may not be updated. Once an error occurs, the time available for manual error finding is very limited. Therefore, manual processing of table data often leads to problems such as table content errors and inconsistencies. Such errors and inconsistencies may damage the company's reputation and even cause economic losses.

[0004] (2) Manually extracting information from tables is a tedious and time-consuming process. The company needs to invest a lot of personnel to do this work at the same time, which ultimately results in additional expenses.

[0005] (3) In some special industries such as finance and manufacturing, tables are often published as unstructured digital files such as portable document format (PDF), which are difficult to extract and process manually.

[0006] Based on the above problems, in order to improve the efficiency and accuracy of information entry and reduce costs, there is an urgent need to provide a new table parsing method. Summary of the Invention

[0007] The purpose of this invention is to provide a table parsing method and device for heterogeneous document images, which can improve the efficiency and accuracy of information entry and reduce overhead.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] A table parsing method for heterogeneous document images, the method comprising:

[0010] Convert the document to be processed into a heterogeneous document image;

[0011] Based on heterogeneous document images, the coordinates of the four vertices of the table are extracted using a table detection algorithm determined by the PP-LCNet network, the CornerNet-lo region proposal generation module, and the FastR-CNN module.

[0012] Based on the coordinates of the four vertices of the table, a text detection algorithm is used to extract the coordinates of the four points of the text boxes in the table;

[0013] Based on the four coordinates of the text box, a text recognition algorithm is used to extract the content of the text box;

[0014] Based on the coordinates of the four vertices of the table, a table structure prediction model is used to obtain the coordinates of the four points of the cell and the table structure information; the table structure prediction model includes an encoder module and a decoder module;

[0015] The coordinates of the four points of the text box and the four points of the cell are compared using the Interchange of Union (IOU) method to aggregate the coordinates of the cells. The IOU is the result of dividing the overlapping part of the two regions by the sum of the two regions.

[0016] Aggregate cells and text based on the content of the text box and the coordinates of the aggregated cells;

[0017] The aggregated text and cells are combined with the table structure information to obtain the complete table structure.

[0018] Optionally, the table detection algorithm extracts the coordinates of the four vertices of the table as follows:

[0019] The PP-LCNet network is used as the backbone network to compute shared convolutional feature maps of heterogeneous document images;

[0020] Using CornerNet-lo's region proposal generation module, the top-left and bottom-right corners of the table in the shared convolutional feature map are detected to generate table proposals, and all potential table proposals are counted.

[0021] The FastR-CNN model is used to prune non-table proposals and refine the bounding boxes of retained table proposals.

[0022] Optionally, the loss function L of the CornerNet-lo region proposal generation module CornerNet For: L CornerNet =L det +αL pull +βL push +γL off ; among which, L det It is a focus-shift loss function used for classification tasks; L offIt is a smoothL1 Loss function used for regression tasks; L pull It is the loss function used to group vertices during network training; L push The loss function used to separate vertices; α, β, and γ are respectively the loss functions used to separate vertices. pull ,L push ,L off The weights;

[0023] The loss function L of the Fast R-CNN module FRCN for: Where N is the number of regional proposals, N pos This is the number of positive regional recommendations, p i and p i * It is the predicted probability label relative to the ground truth, L cls It is a cross-entropy loss used for classification tasks, t j and t j * These are the predicted offset and the actual offset relative to the ground truth, respectively. reg It is an L1 loss used for regression tasks;

[0024] The loss function L of the table detection algorithm is: L = λ·L CornerNet +L FRCN ; where λ is the balance coefficient.

[0025] Optionally, the text detection algorithm is the DBNet algorithm.

[0026] Optionally, the text recognition algorithm is a CRNN algorithm.

[0027] Optionally, the encoder module uses the lightweight network PP-LCNet as the backbone network to extract table features; and uses the FPN+PAN network to fuse features at different levels.

[0028] Optionally, the decoder module uses a Transformer to decode the predicted cell structure.

[0029] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the method.

[0030] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] This invention provides a table parsing method and device for heterogeneous document images. Based on the extraction of tables from standard documents, it improves upon current table parsing models. By proposing a lightweight table detection model, it enhances the efficiency and accuracy of table localization, thereby preserving more table feature information, such as strong localization features and enhanced semantic information, ultimately leading to more accurate table structure prediction. This method improves the efficiency and accuracy of table recognition, especially in complex table scenarios (such as scenarios with many borderless cells or many blank cells), where it demonstrates superior performance compared to traditional methods. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 The present invention provides a table parsing method and device for heterogeneous document images;

[0034] Figure 2 This is a flowchart illustrating the basic algorithm of the intelligent table parsing method for heterogeneous document images proposed in this invention.

[0035] Figure 3 This is a diagram of the table detection algorithm mentioned in this invention;

[0036] Figure 4 The diagram shows the text detection algorithm of this invention (DB-Net algorithm);

[0037] Figure 5 This is the network diagram of the table structure prediction model of the present invention;

[0038] Figure 6 This is a diagram of the Transformer architecture of the present invention;

[0039] Figure 7 This is a schematic diagram illustrating an example of input for the present invention (where the bolded box represents the result obtained by the table detection algorithm);

[0040] Figure 8 This is an example image showing the results of the text detection algorithm of this invention;

[0041] Figure 9 This is an example diagram of the cell aggregation result of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The purpose of this invention is to provide a table parsing method and device for heterogeneous document images, which can make full use of deep learning to enable the model to reasonably learn the feature information of document images, thereby obtaining more accurate tables, and thus improving the efficiency and accuracy of information entry and reducing overhead.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] like Figure 1 and Figure 2 As shown, the present invention provides a table parsing method for heterogeneous document images, the method comprising:

[0046] S101, convert the document to be processed into a heterogeneous document image;

[0047] S102, Based on the heterogeneous document images, extract the coordinates of the four vertices of the table using the table detection algorithm determined by the PP-LCNet network, CornerNet-lo region proposal generation module, and Fast R-CNN module;

[0048] like Figure 3 As shown, the steps of the table detection algorithm to extract the coordinates of the four vertices of the table are as follows:

[0049] S201 uses the PP-LCNet network as the backbone network to compute shared convolutional feature maps of heterogeneous document images;

[0050] S202 uses the CornerNet-lo region proposal generation module to detect the top left and bottom right corners of the table in the shared convolutional feature map, thereby generating table proposals and counting all potential table proposals;

[0051] S202 specifically includes:

[0052] Use convolutional networks to predict two sets of heatmaps, representing the top-left and bottom-right corner positions of different object categories, respectively;

[0053] Simultaneously predict the embeddings of each detected vertex, so that the distance between the embeddings of two vertices of the same object is small;

[0054] To generate tighter bounding boxes, the network also predicts offsets to slightly adjust the position of the vertices;

[0055] The final bounding boxes are obtained by applying a simple post-processing algorithm using the obtained heatmaps, embeddings, and offsets.

[0056] S203 uses the Fast R-CNN model to prune non-table proposals and refine the bounding boxes of retained table proposals.

[0057] S203 specifically includes:

[0058] Each region proposal is pooled using the ROIALIGN algorithm to obtain a 64-channel 7*7 feature map;

[0059] The system uses two 1024-dimensional fully connected layers, each followed by a ReLU activation function.

[0060] The probability distribution of each category is obtained through a fully connected layer and a softmax activation function. At the same time, bounding box regression is performed on each candidate region to adjust its position to better match the actual location of the target.

[0061] The loss function L of the CornerNet-lo region proposal generation module CornerNet For: L CornerNet =L det +αL pull +βL push +γL off ; among which, L det It is a focus-shift loss function used for classification tasks; L off It is a smoothL1Loss function used for regression tasks; L pull It is the loss function used to group vertices during network training; L push The loss function used to separate vertices; α, β, and γ are respectively the loss functions used to separate vertices. pull ,L push ,L off The weights are set; α and β are set to 0.1, and γ is set to 1.

[0062] The loss function L of the Fast R-CNN module FRCN for: Where N is the number of regional proposals, N pos This is the number of positive regional recommendations, p i and p i *It is the predicted probability label relative to the ground truth, L cls It is a cross-entropy loss used for classification tasks, t j and t j * These are the predicted offset and the actual offset relative to the ground truth, respectively. reg It is an L1 loss used for regression tasks;

[0063] The loss function L of the table detection algorithm is: L = λ·L CornerNet +L FRCN Where λ is the balance coefficient, which is set to 0.2.

[0064] S103. Based on the coordinates of the four vertices of the table, a text detection algorithm is used to extract the coordinates of the four points of the text boxes in the table; the text detection algorithm is the DBNet algorithm.

[0065] The specific network structure of the DBNet algorithm is as follows: Figure 4 As shown, the formula for difference binarization is as follows:

[0066]

[0067] Where B is the binarized image of the result, P (i，j) To obtain the probability map, T (i，j) The obtained threshold map is shown, where k is the gain factor, set to 50 here. Differential binarization refers to binarizing each pixel using a different threshold. This different threshold matrix is ​​learned by the network. To ensure gradient propagation throughout the optimization process, the difference between the probability map and the threshold map is passed to the sigmoid function. Through gradient optimization, different threshold matrices are used for different images, achieving the best binarization effect.

[0068] S104. Based on the four coordinates of the text box, a text recognition algorithm is used to extract the content of the text box; the text recognition algorithm is the CRNN algorithm.

[0069] The CTC loss function formula is as follows:

[0070] CTCLoss = -log(P(Y|X));

[0071] Among them, B -1Let (Y) represent all possible label sequences that can be mapped to the target sequence Y, and P(π|X) be the probability of the label sequence π given the input sequence X. To compute P(π|X), CTC introduces an alignment function B(π) to map the label sequences to the input sequence, where B(π) is a sequence of length equal to the input sequence. In B(π), adjacent identical labels are merged, and whitespace is removed.

[0072] Define the forward probability α t Let (s) be the probability of being in state s at time step t, where s represents the position in the label sequence. Then the recursive relationship for the forward probability is:

[0073]

[0074] Wherein, P(x t |π t This indicates that, given a sequence of labels, the input at time step t is x. t The probability of.

[0075] S105, based on the coordinates of the four vertices of the table, a table structure prediction model is used to obtain the coordinates of the four points of the cell and the table structure information; the table structure prediction model includes an encoder module and a decoder module; the table structure prediction model is as follows: Figure 5 As shown, the encoder module extracts the features of the table image into a sequence, followed by a position encoding module to retain the corresponding position information of the table. The resulting feature sequence is then input into the decoder module for decoding and prediction of the coordinates of the four points of the cells and the structural information of the table.

[0076] The encoder module uses the lightweight network PP-LCNet as its backbone to extract table features. It then uses an FPN+PAN network to fuse features from different levels, ensuring that the N6 data has bidirectional features. FPN works top-down, passing down strong semantic features from higher levels to enhance the entire feature pyramid; however, it only enhances semantic information and does not pass on localization information. PAN, on the other hand, adds a bottom-up pyramid after FPN, complementing it by passing up strong localization features from lower levels, thus preserving both local and global information features. When dealing with complex table scenarios, retaining more features is more beneficial for table prediction.

[0077] The decoder module uses a Transformer to decode and predict cell structures; the specific process is as follows. The sequence of the Encoder's results is combined with the positional encoding module to obtain a sequence with positional information. This sequence with positional information is used as the input to the Decoder. It first passes through a Transformer Layer, and then splits into two branches. One branch performs supervised learning of the table structure sequence, and the other branch performs supervised learning of cell position regression in the table. The reason for splitting into two branches after passing through a Transformer Layer, instead of passing through three Transformer Layers and adding two heads for different tasks, is to better decouple the features. To align the length of the predicted table sequence with the number of regressed cells, non-cell portions in the cell position regression branch are removed during training and not included in the table frame regression loss. Similarly, during prediction, non-cell portions are removed based on the predicted structure sequence (HTML sequence), and the regression results of other positions are taken as the output.

[0078] The structure of the Transformer Layer is as follows: Figure 6 As shown, feature enhancement is performed through two multi-head attention modules, followed by a feed-forward network to add more non-linear features. In the prediction phase, the Masked Multi-Head Attention network can only access the output information from the previous time step (t-1) during the prediction phase. However, during the training phase, by creating a lower triangular mask matrix, the decoder can output predictions from all time points simultaneously, instead of sequentially. This makes the training process highly parallel. The Point-wise Feed-Forward Network (FFN) consists of two fully connected layers. Between these two layers is a ReLU activation function. The formula for FFN is shown below:

[0079] FFN(x)=max(0,xW1+b1)W2+b2;

[0080] Where the weight is Bias is b2∈R d ;d ff It is the internal dimension of two linear transformations.

[0081] S106 performs an IOU (Intersection over Union) check between the four coordinates of the text box and the four coordinates of the cell to aggregate the cell coordinates. IOU is calculated by dividing the overlapping portion of the two regions by the sum of the values ​​of the two regions. The specific calculation formula is as follows:

[0082]

[0083] Here, A and B are two different sets of regions.

[0084] S107, aggregate cells and text based on the content of the text box and the coordinates of the aggregated cells;

[0085] S108 combines the aggregated text and cells with the table structure information to obtain the complete table structure.

[0086] For example, a sample document is provided, such as Figure 7 As shown:

[0087] First, the input document is converted into image format, providing heterogeneous document image input for subsequent table recognition algorithms.

[0088] Next, a table detection algorithm will be used. By detecting the coordinates of the top left and bottom right corners of the table, a complete table area can be obtained. Then, based on the vertex coordinates, the table area in the document image will be cropped out, such as... Figure 7 As shown in the bold box.

[0089] After obtaining the image of the table area, we input the image into two branches. The upper branch performs text recognition and extraction, while the lower branch predicts the table structure. First, in the upper branch, the image undergoes a text detection algorithm to obtain the text boxes located within the table, as shown in the image. Figure 8 As shown, the text boxes are then input into the subsequent text recognition algorithm (CRNN+CTC), ultimately yielding the text output for each text box. In the second branch, the text first passes through a PP-LCNet+FPN+PAN network to extract table features. The FPN and PAN networks then bidirectionally preserve these table features, enhancing both strong semantic information and strong localization features. Finally, the table's positional encoding information is added, and the data is passed through three transformers to obtain the final cell regression coordinates and the predicted table structure (HTML sequence).

[0090] Secondly, cell coordinate aggregation is performed based on the text box coordinates obtained from the text detection algorithm and the cell coordinates obtained from the table structure prediction algorithm. This is mainly achieved by calculating the Intersection over Union (IOU) and vertex distance between the text box coordinates and cell coordinates to aggregate single-line to multi-line data. IOU is used for judgment to achieve an aggregation effect between text boxes and cell boxes. The specific results are as follows: Figure 9 As shown.

[0091] Then, the text recognition results are concatenated based on the aggregation results, so that the cell contents of multi-line text can be concatenated into a single string.

[0092] Finally, the cells containing text information are arranged and combined according to the predicted table structure to obtain a complete table sequence (HTML format). The HTML format table can then be converted into Excel format, resulting in an Excel table.

[0093] In one embodiment, a computer device is provided, which may be a database. The computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores transactions to be processed. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0094] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A table parsing method for heterogeneous document images, characterized in that, The method includes: Convert the document to be processed into a heterogeneous document image; Based on heterogeneous document images, the coordinates of the four vertices of the table are extracted using a table detection algorithm determined by the PP-LCNet network, the CornerNet-lo region proposal generation module, and the FastR-CNN module. Based on the coordinates of the four vertices of the table, a text detection algorithm is used to extract the coordinates of the four points of the text boxes in the table; Based on the four coordinates of the text box, a text recognition algorithm is used to extract the content of the text box; Based on the coordinates of the four vertices of the table, a table structure prediction model is used to obtain the coordinates of the four points of the cell and the table structure information; the table structure prediction model includes an encoder module and a decoder module; The coordinates of the four points of the text box and the four points of the cell are compared using the Interchange of Union (IOU) method to aggregate the coordinates of the cells. The IOU is the result of dividing the overlapping part of the two regions by the sum of the two regions. Aggregate cells and text based on the content of the text box and the coordinates of the aggregated cells; The aggregated text and cells are combined with the table structure information to obtain the complete table structure.

2. The table parsing method for heterogeneous document images according to claim 1, characterized in that, The steps for the table detection algorithm to extract the coordinates of the four vertices of the table are as follows: The PP-LCNet network is used as the backbone network to compute shared convolutional feature maps of heterogeneous document images; Using CornerNet-lo's region proposal generation module, the top-left and bottom-right corners of the table in the shared convolutional feature map are detected to generate table proposals, and all potential table proposals are counted. The FastR-CNN model is used to prune non-table proposals and refine the bounding boxes of retained table proposals.

3. A table parsing method for heterogeneous document images according to claim 1 or claim 2, characterized in that, The loss function L of the CornerNet-lo region proposal generation module CornerNet For: L CornerNet =L det +αL pull +βL push +γL off ; among which, L det It is a focus-shift loss function used for classification tasks; L off It is a smoothL1Loss function used for regression tasks; L pull It is the loss function used to group vertices during network training; L push The loss function used to separate vertices; α, β, and γ are respectively the loss functions used to separate vertices. pull ,L push ,L off The weights; The loss function L of the Fast R-CNN module FRCN for: Where N is the number of regional proposals, N pos This is the number of positive regional recommendations, p i and p i * It is the predicted probability label relative to the ground truth, L cls A cross-entropy loss is used for classification tasks, t j and t j * These are the predicted offset and the actual offset relative to the ground truth, respectively. reg It is an L1 loss used for regression tasks; The loss function L of the table detection algorithm is: L = λ·L CornerNet +L FRCN ; where λ is the balance coefficient.

4. The table parsing method for heterogeneous document images according to claim 1, characterized in that, The text detection algorithm is the DBNet algorithm.

5. The table parsing method for heterogeneous document images according to claim 1, characterized in that, The text recognition algorithm is the CRNN algorithm.

6. The table parsing method for heterogeneous document images according to claim 1, characterized in that, The encoder module uses the lightweight network PP-LCNet as the backbone network to extract table features; and uses the FPN+PAN network to fuse features at different levels.

7. The table parsing method for heterogeneous document images according to claim 1, characterized in that, The decoder module uses Transformer to decode and predict cell structures.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

Images