A table structure recognition method based on cascade decoding

Abstract

The present invention relates to a table structure recognition method based on cascade decoding, which belongs to the technical field of semi-structured text generation. First, a table image is input into an encoder for encoding to capture the visual features of the input table image; then the encoding results are input into a row decoder and a cell decoder respectively; wherein, first, the encoding result is input into the row decoder alone for decoding to obtain table header, table body, row and other tags; then the encoding result and the decoding result of the row decoder are input into the cell decoder for decoding to obtain cell tags and cell row and column span attributes; finally, the decoding results of the two decoders are fused to obtain a structured HTML sequence representation of the corresponding table image. The present invention decodes the encoding results at the row and cell levels respectively. The hierarchical decoding structure ensures that each decoder will not generate a long HTML sequence, solves the error accumulation problem when generating a long sequence, and can more accurately identify table structure information.

Description

Technical Field

[0001] The invention relates to a table structure recognition method based on cascade decoding, and belongs to the technical field of semi-structured text generation. Background Art

[0002] As a commonly used data carrier, tables are easy to compare and analyze, and have high information density. They are often used to display important experimental and production data. In today's intelligent era, the ability to quickly and accurately identify and analyze this data will bring strong competitiveness to social enterprises and research institutions. The table structure recognition task aims to obtain the row and column information of cells from the table image. Accurately identifying the table structure is an important basis for analyzing the table content.

[0003] Current table structure recognition methods that recover the HTML sequence corresponding to a table from a table image typically use a single-stage decoder to decode the HTML tags corresponding to all elements in the table. As the complexity of the table increases during inference, these methods generate longer HTML sequences, which can lead to more errors. Therefore, it is important to develop a method that avoids generating long HTML sequences and accurately recognizes table structure information. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and provide a table structure recognition method based on cascade decoding. The encoding results are decoded in two stages. The first stage decodes the table header, table body, and rows according to the encoding results. The second stage decodes the cell labels and cell attributes according to the decoding results of the first stage and the encoding results. The design of the cascade decoding structure shortens the length of the labels generated by each level of decoder during decoding, and to a certain extent solves the problem of low accuracy when generating longer HTML sequences.

[0005] The technical solution of the present invention is: a table structure recognition method based on cascade decoding, first inputting a table image into an encoder for encoding to capture the visual features of the input table image; then inputting the encoding results into a row decoder and a cell decoder respectively; wherein, firstly, the encoding results are inputted into the row decoder for decoding to obtain table header, table body, row and other tags; then, the encoding results and the decoding results of the row decoder are inputted into the cell decoder for decoding to obtain cell tags and cell row and column span attributes; finally, the decoding results of the two decoders are fused to obtain a structured HTML sequence representation of the corresponding table image.

[0006] The specific steps are:

[0007] Step 1: Use the encoder to extract visual features from the table image.

[0008] Step 2: The visual features extracted by the encoder are separately input into the row decoder for decoding. The training process of the row decoder is as follows:

[0009] Training data: Process the public dataset annotation files and retain only the header labels, body labels, and row labels.

[0010] Model training and model inference: During model training, a recurrent unit module with an attention mechanism is used to learn table row features from the visual features extracted by the encoder; during inference, only header labels, title labels, and row labels are generated.

[0011] Step 3: The decoding results of the row decoder are input into the cell decoder together with the visual features extracted by the encoder to decode the cell labels and cell attributes corresponding to all cells in each row.

[0012] Training model: During model training, a recurrent unit module with an attention mechanism is used to learn cell features from the visual features extracted by the encoder and the hidden layer features of the row decoder.

[0013] Model inference: During model inference, the row decoder decoding result determines whether the cell decoder generates cell labels and cell row and column span attributes.

[0014] Step 4: Combine the decoding results of the row decoder and the cell decoder to obtain the complete HTML sequence corresponding to the table image.

[0015] In Step 1, the encoder uses ResNet18 as the backbone network to extract visual features and uses FPN to enhance the visual features.

[0016] During the training data processing in Step 2, the public dataset PubTabNet annotation file is processed, and only the header labels ('', ''), body labels ('', ''), and row labels ('', '') in the annotation file are retained.

[0017] The row decoder in Step 2 is composed of a GRU module based on the attention mechanism and a GRU hidden layer feature decoder module; during the training process, the GRU learns the corresponding features of the header, body, and row from the visual features of the table image and the hidden layer features of the previous layer; during model inference, the GRU hidden layer feature decoder is used to generate the header label ('', ''), body label ('', ''), and row label ('', '').

[0018] The loss function l of the row decoder that generates the header label, body label, and row label in Step 2 rd Using cross entropy loss function:

[0019]

[0020] Among them, N represents the total number of generated tags, i represents the i-th tag generated, and the six categories refer to the start and end tags corresponding to the header, body, and row. ic Represents the category label of each sample, p ic represents the probability that the i-th label belongs to category c.

[0021] The cell decoder in Step 3 is composed of a GRU module based on the attention mechanism and a GRU hidden layer feature decoder module; during the model training process, the GRU learns the cell label ('', '') features and cell attribute ('colspan', 'rowspan') features from the table image visual features and the row decoder hidden layer features.

[0022] The loss function l of the cell decoder that generates cell labels and cell attributes in Step 3 cd Using cross entropy loss function:

[0023]

[0024] Among them, N represents the total number of generated labels, i represents the i-th label generated, and the four categories refer to the start label and end label and row and column span attributes corresponding to the cell. ic represents the category label of each sample, p ic represents the probability that the i-th label belongs to category c.

[0025] In the cell decoder in Step 3, in addition to decoding the cell label and cell attribute label, the cell coordinate decoder is also used to regress the cell coordinates and regress the cell bounding box loss function l b-box Using Smooth L1 Loss:

[0026]

[0027] Where x = |y i -f(x i )|,y i represents the true bounding box value, f(x i ) represents the predicted bounding box value.

[0028] Finally, the overall network loss l all The calculation method is as follows:

[0029] l all =λ1l rd +λ2l cd +λ3l b-box

[0030] Among them, λ1,λ2,λ3∈[0,1] are hyperparameters.

[0031] During the model inference process in Step 3, if the decoding result of a slice in the row decoder is a sequence starting with the '' label, the cell decoder combines the row decoder hidden layer features corresponding to this sequence with the visual features of the table image to infer and generate the cell labels and cell attributes in the sequence. If it is a sequence that does not start with the '' label, all labels in the sequence except the beginning label and the corresponding end label are deleted.

[0032] The beneficial effects of the present invention are as follows: based on the existing table image to semi-structured text generation model, the present invention adopts a cascade decoder to perform row decoding and cell decoding on the visual features of the table image respectively; wherein, the row decoder only decodes the table header, table body, and row areas, and the cell decoder only decodes the cell area. Different decoders are used for decoding different areas, which can effectively solve the problem of low accuracy in the longer HTML sequence generated when decoding complex tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an overall framework diagram of the table structure recognition method based on cascade decoding in the present invention;

[0034] Figure 2 Flowchart of the table structure recognition method based on cascade decoding in the present invention;

[0035] Figure 3 FIG. 4 is a schematic diagram of a cascade decoding module in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] Figure 1 This is the overall framework diagram of the table structure recognition method based on cascade decoding in the present invention. The present invention consists of an encoder for extracting visual features of table images, a cascade decoder for decoding rows and cells in stages, and a two-stage decoding result fusion module.

[0038] Figure 2This is a flow chart of the table structure recognition method based on cascade decoding in the present invention. For each input table image, the encoder module extracts visual features, and then uses the row decoder to decode the visual features to obtain the corresponding tokens of the table header, table body, and row. The visual features extracted by the encoder are combined with the hidden layer features of the row decoder and input into the cell decoder. The cell decoder decoding result is determined according to the output token of the row decoder. Finally, the decoding results of the row decoder and the cell decoder are fused to output the complete HTML sequence of the table. The specific implementation steps are as follows:

[0039] Step 1: Use the encoder to extract visual features from the table image.

[0040] In an embodiment of the present application, a ResNet-based neural network can be used as the backbone network to extract visual features, and then the visual features can be enhanced through the FPN network.

[0041] Step 2: The visual features extracted by the encoder are input into the cascade decoder to decode the rows and cells respectively. The cascade decoding module model is shown in the figure below: Figure 3 shown.

[0042] Step 2.1: Input the row decoder separately for decoding. The training process of the row decoder is as follows:

[0043] Step 2.1.1: Training data: Process the public dataset annotation file and only retain the header label, table body label, and row label.

[0044] In an embodiment of the present application, specifically in the process of processing training data for the row decoder, the public dataset PubTabNet annotation file is processed, and only the header labels ('', ''), table body labels ('', ''), and row labels ('', '') in the annotation file are retained.

[0045] Step 2.1.2: Model training and model inference: During model training, the recurrent unit module with integrated attention mechanism is used to learn table row features from the visual features extracted by the encoder; during inference, header labels, title labels, and row labels are generated.

[0046] In an embodiment of the present application, the cyclic unit module can be a cyclic neural network based on GRU, LSTM, etc.

[0047] In an embodiment of the present application, specifically during the training process, GRU is used to learn the corresponding features of the header, body, and row from the visual features of the table image and the features of the previous hidden layer; during model inference, the GRU hidden layer feature decoder is used to generate the header label ('', ''), body label ('', ''), and row label ('', '').

[0048] Step 2.1.3: Generate the loss function l of the row decoder for the header label, body label, and row label rd Using cross entropy loss function:

[0049]

[0050] Among them, N represents the total number of generated tags, i represents the i-th tag generated, and the six categories refer to the start and end tags corresponding to the header, body, and row. ic represents the category label of each sample, p ic represents the probability that the i-th label belongs to category c.

[0051] Step 2.2: The decoding results of the row decoder are input into the cell decoder together with the visual features extracted by the encoder to decode all cell labels and cell attributes in each row.

[0052] Step 2.2.1: Model training and model inference: Use the recurrent unit module with integrated attention mechanism to learn cell features from the visual features extracted by the encoder and the hidden layer features of the row decoder; generate cell labels and cell row and column span attributes during the inference process.

[0053] In an embodiment of the present application, the cyclic unit module can be a cyclic neural network based on GRU, LSTM, etc.

[0054] In an embodiment of the present application, specifically during the model training process, GRU is used to learn cell label ('', '') features and cell attribute ('colspan', 'rowspan') features from the table image visual features and the row decoder hidden layer features.

[0055] Step 2.2.2: Generate the loss function l of the cell decoder of cell labels and cell attributes cd Using cross entropy loss function:

[0056]

[0057] Among them, N represents the total number of generated labels, i represents the i-th label generated, and the four categories refer to the start label and end label and row and column span attributes corresponding to the cell. ic represents the category label of each sample, p ic represents the probability that the i-th label belongs to category c.

[0058] Step 2.2.3: In addition to decoding the cell label and cell attribute label, the cell decoder also uses the cell coordinate decoder to regress the cell coordinates and regress the cell bounding box loss function lb-box Using Smooth L1Loss:

[0059]

[0060] Where x = |y i -f(x i )|,y i represents the true bounding box value, f(x i ) represents the predicted bounding box value.

[0061] Step 2.2.4: Finally, the overall network loss l all The calculation method is as follows:

[0062] l all =λ1l rd +λ2l cd +λ3l b-box

[0063] Among them, λ1,λ2,λ3∈[0,1] are hyperparameters.

[0064] Step 2.2.5: Model inference: During the model inference process, the decoding result of the cell decoder is determined based on the decoding result of the row decoder;

[0065] During the model inference process, if the decoding result of a slice in the row decoder is a sequence starting with the '' label, the cell decoder combines the row decoder hidden layer features corresponding to this sequence with the visual features of the table image to infer and generate the cell labels and cell attributes in the sequence. If it is a sequence that does not start with the '' label, all labels in the sequence except the beginning label and the corresponding end label are deleted.

[0066] Step 3: Combine the decoding results of the row decoder and the cell decoder to obtain the complete HTML sequence corresponding to the table image.

[0067] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Images