Table analysis method based on image-to-sequence technology and introducing cell alignment information

Abstract

The invention discloses a table analysis method based on an image-to-sequence technology and introducing cell alignment information, relates to the field of character recognition and information extraction, and designs a table analysis model capable of simultaneously analyzing a table structure, character recognition and cell alignment information by introducing the cell alignment information. Through the cell text matching algorithm added with the alignment information, the occurrence frequency of the wrong distribution problem of the partial texts is reduced, the accuracy of table structure analysis is improved, and therefore the model effect of table analysis is optimized. Meanwhile, a set of table random generation algorithm is designed, and the algorithm can generate table pictures of which the cells are randomly aligned and labeling information corresponding to the table pictures with alignment information. The algorithm can provide a data set for training for the table analysis algorithm provided by the invention.

Description

Technical Field

[0001] The present invention belongs to the field of text recognition and information extraction, and more specifically, particularly relates to a table parsing method based on image-to-sequence technology that introduces cell alignment information. Background Art

[0002] In recent years, with the continuous improvement of technologies such as text recognition and information extraction, table parsing technology has also received more and more attention; table parsing algorithms can analyze and extract the structure and information in table images, and store the extracted structured tables in corresponding formats; most of the current table parsing models use table structure information and text recognition accuracy as the main training targets; however, the models trained based solely on these two factors have not yet reached perfection in table parsing results, and some parsing results will still have deviations; the present invention notes that some of the parsing deviations are related to "cell alignment" to some extent; the present invention believes that this part of the result deviation caused by the misallocation of cell text can be optimized by introducing an additional model training target "cell alignment information" that has not yet received attention;

[0003] At present, the measurement criteria for table parsing effects can be roughly divided into two parts, namely, the accuracy of table structure parsing and the accuracy of text recognition within cells; most of the current table parsing models are trained based on these two parts, and the image annotation information in many large public data sets on which they are based is also mostly composed of these two parts. Other attributes expressed in the original table image have not been extracted and paid attention to in the existing models; however, the models currently trained solely based on these two still have room for improvement in practical parsing results; for example, the present invention observes that if a cell in the original table image is right-aligned and the next cell is left-aligned, the parsing model sometimes incorrectly assigns the text of one of the cells to the other cell; for example, for some table columns with larger column widths, if there are several rows that are aligned in the opposite direction to the rest of the cells in the column, they will sometimes be parsed as the content of another column; it can be noted that these parsing deviations are all related to "cell alignment" to some extent, among which the errors caused by the difference in left and right alignment are more obvious. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings existing in the prior art, and a table parsing method based on image-to-sequence technology that introduces cell alignment information is proposed. By introducing cell alignment information, a table parsing model that can simultaneously parse table structure, text recognition and cell alignment information is designed. The frequency of the above-mentioned partial text misallocation problem is reduced by adding a cell text matching algorithm with alignment information, and the accuracy of table structure parsing is improved, thereby optimizing the model effect of table parsing. At the same time, the present invention designs a set of table random generation algorithms, which can generate table images with randomly aligned cells and their corresponding annotation information with alignment information. The algorithm can provide a training data set for the table parsing algorithm proposed by the present invention.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A table parsing method based on image-to-sequence technology that introduces cell alignment information includes the following steps:

[0007] S1. The method includes a table generation algorithm: generating parameters according to user input, the parameters determine the number and size of the generated tables, whether there are cells across rows and columns, the length range of cells across rows and columns, and the complexity of the table structure; the table generation algorithm generates the data set required for training;

[0008] S2, establishing a table parsing model and training the model, wherein the table parsing model includes a text line detection model, a text line recognition model, and a table structure recognition model;

[0009] After the training of S3, text line detection, text line recognition, and table structure recognition, model inference is performed separately;

[0010] S4. After the inference results of the three parts are obtained, first, the merging algorithm will receive the structure sequence given by the table structure recognition and use it as the framework of the HTML table; secondly, the position prediction results of the table structure recognition and text line detection parts will be handed over to the cell text matching algorithm to obtain pairing information; finally, the merging algorithm will match the text lines into the corresponding positions in the table structure according to certain rules through pairing;

[0011] S5. For the input table image, the trained model will obtain an HTML table with alignment information and convert it into the corresponding table format through conversion code.

[0012] Preferably, in step S1, generating the table is specifically:

[0013] S11, generating a two-dimensional identification structure based on the required table size;

[0014] S12, selecting a cell within a certain range according to the parameters and marking it as the starting point of the cross-row and cross-column cell, and marking its spanning length and direction;

[0015] S13, the span length is randomized within a certain range according to the parameters, while ensuring that no other cross-row or cross-column cells are overlapped;

[0016] S14. Several cells in the spanning range following the marked cell will be marked as covered at the same time. These positions are invisible in the generated table and will not participate in the subsequent randomization of the positions of cells across rows and columns.

[0017] Preferably, for a table with complexity equal to n, the algorithm will first randomly find a table block and cut it into two parts, with the cutting direction being random in horizontal and vertical directions. After cutting, any part of the table block can be used as the object for the next cutting, and the cutting operation continues n times.

[0018] Preferably, in step S2, establishing the table parsing model is specifically as follows:

[0019] S21, Text Line Detection TLD is responsible for detecting the position where the text appears in the image. It reads in a table image and outputs text line coordinates and a text line image intercepted based on the coordinates. The coordinates will be used together with the coordinate prediction in the table structure recognition to match the cell text. The intercepted text line image will be handed over to the text recognition model TLR for further processing.

[0020] S22, Text Line Recognition TLR is responsible for detecting the content in the text image and converting it into text, reading in as a text image and outputting as a text recognition result; the text information will be subsequently handed over to the merging optimization algorithm to be embedded into the corresponding table position;

[0021] S23, table structure recognition TSR is responsible for predicting the overall table structure, cell alignment information and the approximate position of the corresponding cell; it reads in a table image and outputs a table structure + cell alignment information sequence and cell coordinate prediction; the cell coordinate prediction will be used together with the result of the text detection model for cell text matching; the structure plus the cell alignment information sequence will be handed over to the merge optimization algorithm as the framework of the table;

[0022] S24, the cell text matching algorithm is responsible for matching the text position output by the text line detection TLD to the cell position output by the table structure recognition TSR, the input is the text line coordinates of the text detection and the cell coordinate prediction of the structure recognition, and the output is the pairing relationship between the cell and the text;

[0023] S25, the merge optimization algorithm is responsible for receiving the output of all the above parts and integrating them into an HTML table, reading in the table structure + cell alignment information sequence given by the table structure recognition TSR, the cell text pairing relationship given by the cell text matching algorithm, and the cell text given by the text line recognition, embedding the text into the table structure framework through the pairing relationship, and further optimizing the accuracy of text recognition through cell alignment information, statistical information and some preset rules; the output is a synthesized HTML table;

[0024] S26, the format conversion code is responsible for converting the HTML table into the required table.

[0025] Preferably, in step S2, the model architecture is:

[0026] Arrow: refers to the transfer of the output of the previous part to the input of the part pointed by the arrow. If the previous part is not explicitly described, it means that the data is transferred from outside the component / model;

[0027] Linear layer: performs a linear calculation on the input;

[0028] Addition layer: adds two inputs;

[0029] Normalization layer: normalize the input;

[0030] Convolutional layer: performs convolution on the input;

[0031] Matrix multiplication layer: performs matrix multiplication on the input;

[0032] Mask layer: masks part of the input;

[0033] Scaling layer: performs scaling on the input;

[0034] Connection layer:: connects the inputs together;

[0035] ReLU: A rectified linear function with the formula:

[0036] f(x)=max(0,x);

[0037] Softmax: A probability conversion function with the formula:

[0038]

[0039] Embedding / vectorization layer: maps the input to a vector space;

[0040] The combination of layers can form basic modules. The following modules will be used in the model:

[0041] Scaled dot product attention module: The input is Queue, Key and Value respectively, and the output formula is:

[0042]

[0043] Multi-head attention module: that is, multi-head SDPA module + connection layer + linear layer;

[0044] One of the modules of the forward propagation module structure is a combination of a set of linear layers, ReLU linear layers, and its formula is:

[0045] Feedforward(x)=ReLU(xW1+b1)W2+b2.

[0046] Preferably, in step S2, the text line detection model is responsible for detecting the position where the text appears in the picture, reading in a table picture, and outputting text line coordinates and a text line picture intercepted based on the coordinates;

[0047] The model uses the PSENet text detection algorithm, uses the deep residual network ResNet as the backbone, and uses feature pyramid networks to generate several feature layers. It connects feature samples of several scales and performs convolution processing to complete the prediction.

[0048] Preferably, in step S2, the text line recognition model is responsible for detecting the content in the text image and converting it into text, reading in as a text image and outputting as a text recognition result;

[0049] The text line recognition model uses image-to-sequence img2seq, supports single-line and multi-line recognition, and can directly use table data and cell coordinate information to intercept text parts for training;

[0050] The encoding module Encoder Block adopts the MASTER structure, which consists of a ResNet residual module ResidualBlock connected to a multi-faceted global context attention module MAGC and a convolutional layer;

[0051] The formula of the multi-aspect global attention module MAGC of the residual module is as follows:

[0052] MAGC(x)=Concat(gc1,gc2...gc h )

[0053] gc i =Σα j x j

[0054]

[0055] The functions of concat and softmax in the formula are consistent with those of the model layers with the same name.

[0056] Preferably, in step S2, the table structure recognition model is responsible for predicting the overall table structure, cell alignment information and the approximate position of the corresponding cell. The input is a table image, and the output is a table structure + cell alignment information sequence and cell coordinate prediction.

[0057] Preferably, in step S2, the cell text matching algorithm is responsible for matching the text position output by the text line detection TLD to the cell position output by the table structure recognition TSR, reading in the text line coordinates of text detection and the cell coordinate predictions of structure recognition, and outputting the pairing relationship between the cell and the text.

[0058] Technical effects and advantages of the present invention: The present invention provides a table parsing method based on image-to-sequence technology that introduces cell alignment information. Compared with traditional technologies, the present invention introduces cell alignment information to design a table parsing model that can simultaneously parse table structure, text recognition and cell alignment information. The frequency of the above-mentioned partial text misallocation problem is reduced by adding a cell text matching algorithm with alignment information, and the accuracy of table structure parsing is improved, thereby optimizing the model effect of table parsing. At the same time, the present invention designs a set of table random generation algorithms, which can generate table images with randomly aligned cells and their corresponding annotation information with alignment information. The algorithm can provide a training data set for the table parsing algorithm proposed by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of complex table cutting in an embodiment of the present invention;

[0060] Figure 2 It is a flow chart of a table parsing algorithm in an embodiment of the present invention;

[0061] Figure 3 This is a diagram of the architecture of the scaled dot product attention SDPA module in an embodiment of the present invention;

[0062] Figure 4 This is a diagram of the architecture of a multi-head attention module in an embodiment of the present invention;

[0063] Figure 5 This is a diagram of the residual module architecture in an embodiment of the present invention;

[0064] Figure 6 This is a diagram of the text line detection model architecture in an embodiment of the present invention;

[0065] Figure 7 This is a diagram of the text recognition model encoding module architecture in an embodiment of the present invention;

[0066] Figure 8 This is a diagram of the text recognition model decoding module architecture in an embodiment of the present invention;

[0067] Fig. 9 This is a TLR architecture diagram of a text recognition model in an embodiment of the present invention;

[0068] Fig.10 This is a diagram of the TSR architecture of a text recognition model in an embodiment of the present invention;

[0069] Fig.11 4 is a flow chart of a cell matching algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0071] The present invention provides Figure 2 A table parsing method based on image-to-sequence technology that introduces cell alignment information is designed based on image-to-sequence technology. By introducing cell alignment information, a table parsing model that can simultaneously parse table structure, text recognition and cell alignment information is designed. The design includes a table structure recognition model that can simultaneously predict table structure, cell position and cell alignment; a set of Fig.11 The matching algorithm shown is based on the position information and cell alignment information to pair text with cells; a set of optimization rules that enable a specific table to further modify the text content inside the cell based on the cell alignment information; a text detection model and a text recognition model. The functions of the above parts are orderly combined through the model structure to complete the table parsing task.

[0072] Compared with the previous table parsing model, the table parsing model of the present invention will additionally learn the cell alignment information, and then reduce the frequency of the problem of partial text misassignment through the matching algorithm, improve the accuracy of table structure parsing, and thus optimize the model effect of table parsing. At the same time, the present invention designs a set of table random generation algorithms, which can generate table images with randomly aligned cells and their corresponding annotation information with alignment information. The algorithm can provide a training data set for the table parsing algorithm proposed by the present invention;

[0073] The steps include:

[0074] S1. Generate parameters based on user input. The parameters determine the number and size of the generated table, whether there are cells across rows and columns, the length range of cells across rows and columns, and the complexity of the table structure. Generate the data set required for training by the table generation algorithm. In step S1, the generated table is specifically as follows:

[0075] S11, generating a two-dimensional identification structure based on the required table size;

[0076] S12, selecting a cell within a certain range according to the parameters and marking it as the starting point of the cross-row and cross-column cell, and marking its spanning length and direction;

[0077] S13, the span length is randomized within a certain range according to the parameters, while ensuring that no other cross-row or cross-column cells are overlapped;

[0078] S14. Several cells in the spanning range following the marked cell will be marked as covered at the same time. These positions are invisible in the generated table and will not participate in the subsequent randomization of the positions of cells across rows and columns.

[0079] Among them, for a table with a complexity equal to n, the algorithm will first randomly find a table block and cut it into two parts. The cutting direction is random from horizontal to vertical. After cutting, any part of the table block can be used as the object of the next cutting. The cutting operation continues n times. In this case, cross-row cells can be randomly generated in the first row of any table block, and the same is true for cross-column cells. The design idea of ​​this table complexity is based on the simulation of different table types, where a complexity of 1 represents a simple table, in which row and column headers are more likely to have cross-row and cross-column elements. High-complexity tables can simulate table structure types such as registration forms or application forms, which do not necessarily have table headers in the usual sense, and the structure is more complex and diverse;

[0080] Furthermore, the table structure complexity is a positive integer. It is set to allow the algorithm to generate more complex structures for targeted training of the model. For ordinary tables, its value is 1. For each table, the algorithm first generates a two-dimensional identification structure based on the required table size. The algorithm will select cells within a certain range according to the parameters and identify them as the starting point of the cross-row and cross-column cells, and also identify their span length and direction. The span length is random within a certain range according to the parameters, while ensuring that it does not cover any other cross-row and cross-column cells. Several cells in the span range following the identified cell will also be marked as covered at the same time. The above positions are not visible in the generated table and will not participate in the subsequent randomization of the cross-row and cross-column cell positions.

[0081] In addition, the generated 2D identifier represents the spatial layout of the cells. Next, the algorithm will randomly assign an alignment identifier to each uncovered cell. The algorithm will determine whether to generate both left and right alignment and top and bottom alignment information based on the input parameters. The left and right alignment information will be randomized for all cells, while the top and bottom alignment information will only be randomized for cells spanning multiple rows (the top and bottom alignment information of cells with a height of 1 will not affect the text output position). The left and right alignment is controlled by the align attribute to align the cells, and it is randomized among three left and right alignment identifiers (left alignment, center alignment, right alignment); the top and bottom alignment is controlled by the valign attribute to align the cells, and it is randomized among three top and bottom alignment identifiers (top alignment, center alignment, bottom alignment). The proportion of the three alignment methods in the two alignment directions can be adjusted by weights. By default, the center alignment accounts for the majority. After the alignment identifiers are also generated, the algorithm will traverse each cell in turn and generate the corresponding html tags according to the identifier information. Specifically, the general table tag is generated first, and then the 2D identifier structure is traversed. The and tags are added before and after each row traversal, and the and tags are added before and after each cell. When encountering cells spanning multiple rows and columns, the colspan=n / rowspan=n tags are added, and the cell alignment information is added as the align=left / center / right or valign=top / middle / bottom tag. After traversal, the entire generated table structure and alignment information are losslessly converted into html information. This information is in duplicate. One copy is added to the annotation file as the table structure information, and the other copy is randomly generated from the corpus and added before each cell tag as the text information for generating the table picture. Finally, the algorithm will call the browser driver, render the randomly generated HTML table with alignment information into a web page through the browser rendering mechanism, and call the corresponding screenshot function to capture and save the table. During rendering, the algorithm will simultaneously use DOM to monitor whether the elements corresponding to the cell IDs appear, and record the coordinates of each cell based on the position, length, and width of the elements after detecting their appearance. After the recording is completed, it will be processed and summarized into the corresponding annotation information, which is stored in the dataset together with the rendered table picture.

[0082] S2. Establish a table parsing model and train the model. The table parsing model includes a text line detection model, a text line recognition model, and a table structure recognition model. In step S2, establishing the table parsing model specifically includes:

[0083] S21. The text line detection TLD is responsible for detecting the positions where text appears in the picture. The input is the table picture, and the output is the text line coordinates and the text line pictures intercepted based on the coordinates. Among them, the coordinates will be used for cell text matching together with the coordinate prediction in the table structure recognition, and the intercepted text line pictures will be handed over to the text recognition model TLR for further processing.

[0084] S22, Text Line Recognition TLR is responsible for detecting the content in the text image and converting it into text, reading in as a text image and outputting as a text recognition result; the text information will be subsequently handed over to the merging optimization algorithm to be embedded into the corresponding table position;

[0085] S23, table structure recognition TSR is responsible for predicting the overall table structure, cell alignment information and the approximate position of the corresponding cell; it reads in a table image and outputs a table structure + cell alignment information sequence and cell coordinate prediction; the cell coordinate prediction will be used together with the result of the text detection model for cell text matching; the structure plus the cell alignment information sequence will be handed over to the merge optimization algorithm as the framework of the table;

[0086] S24, the cell text matching algorithm is responsible for matching the text position output by the text line detection TLD to the cell position output by the table structure recognition TSR, the input is the text line coordinates of the text detection and the cell coordinate prediction of the structure recognition, and the output is the pairing relationship between the cell and the text;

[0087] S25, the merge optimization algorithm is responsible for receiving the output of all the above parts and integrating them into an HTML table, reading in the table structure + cell alignment information sequence given by the table structure recognition TSR, the cell text pairing relationship given by the cell text matching algorithm, and the cell text given by the text line recognition, embedding the text into the table structure framework through the pairing relationship, and further optimizing the accuracy of text recognition through cell alignment information, statistical information and some preset rules; the output is a synthesized HTML table;

[0088] S26, the format conversion code is responsible for converting the HTML table into the required table.

[0089] Furthermore, the three model parts of text line detection, text line recognition and table structure recognition need to use data sets to train the models respectively, and the other parts are rule-based algorithms. The input table image will first be parsed by the table structure recognition model and the text line detection model. The results of the two will be matched by the cell text matching algorithm, and the results of the text line detection model will be additionally processed by the text line recognition model. The table structure recognition results, cell text matching results and text line recognition results will be uniformly combined into an HTML table by the merging optimization algorithm, and output as an Excel spreadsheet file through the format conversion algorithm.

[0090] The model architecture is as follows: Arrow: refers to the output of the previous part being passed to the input of the part pointed by the arrow. If the previous part is not explicitly described, it means that the data is passed from outside the component / model.

[0091] Linear layer: performs a linear calculation on the input

[0092] Addition layer (Add / Addition / +): adds two inputs

[0093] Normalization: Normalize the input

[0094] Convolution layer (Conv. / convolution): performs convolution on the input

[0095] Matrix Multiplication Layer (MatMul): performs matrix multiplication on the input

[0096] Mask layer: mask part of the input

[0097] Scale layer: performs scaling on the input

[0098] Connection layer (Concat. / Concatenate): connect the inputs together

[0099] ReLU: A rectified linear function with the formula:

[0100] f(x)=max(0,x)

[0101] Softmax: A probability conversion function with the formula:

[0102]

[0103] Embedding / Vectorization Layer: Mapping the input to a vector space

[0104] The combination of layers can form basic modules (Blocks). The following modules will be used in the model:

[0105] Residual module: One of the modules of ResNet structure, the structure is as follows Figure 5 Shown

[0106] Scaled dot product attention module SDPA: One of the modules of Transformer structure, the structure is as follows Figure 3 As shown, the inputs are Queue, Key and Value respectively, and the output formula is:

[0107]

[0108] MultiHead Attention: One of the modules in the Transformer structure, namely, the multi-head SDPA module + connection layer + linear layer, with the following structure: Figure 4 Shown

[0109] Feed forward module: One of the modules that make up the Transformer structure, that is, a combination of a set of linear layers ReLU linear layers, and its formula is:

[0110] Feedforward(x)=ReLU(xW1+b1)W2+b2

[0111] MASTER Encoder Block: One of the modules that make up the MASTER encoding structure. Its architecture is as follows: Figure 7 shown.

[0112] Transformer Decoder Block: One of the modules that make up the Transformer decoding structure. Its architecture is as follows: Figure 8 shown.

[0113] It should be noted that the text line detection model is responsible for detecting the position where the text (line) appears in the picture. It reads in a table image and outputs text line coordinates and text line images intercepted based on the coordinates. The model uses the PSENet text detection algorithm, uses the deep residual network ResNet as the backbone, and uses the feature pyramid structure feature pyramid networks (FPN) to generate several feature layers. After connecting the feature samples of several scales, convolution processing is performed to complete the prediction. The model training uses table images and text line coordinates, and only a small amount (thousands) of data is required to complete the training. PSENet is a common text detection algorithm, and the present invention basically follows its original architecture. The structure of the (basic) residual module Residual Block in the backbone ResNet is:

[0114] The modules represent the intermediate results of the model, with ResNet on the left and FPN in the middle. The vertical connection is the (up / down) sampling layer, and the horizontal connection is the convolution layer supplemented by the corresponding batch normalization layer and ReLU layer;

[0115] The text line recognition TLR is responsible for detecting the content in the text (line) image and converting it into text. It reads in as a text image and outputs as a text recognition result (text). The text line recognition model uses the image-to-sequence img2seq technology and supports single-line and multi-line recognition. Therefore, the text part can be directly intercepted using table data and cell coordinate information for training. The model part is divided into an encoding part and a decoding part. The encoding module Encoder Block adopts a MASTER structure, which consists of a ResNet residual module ResidualBlock connected to a multi-aspect global context attention module Multi-Aspect Global ContextAttention (MAGC) and a convolutional layer;

[0116] The structure of the residual module is as shown above, and the public

[0117] The formula is as follows:

[0118] MAGC(x)=Concat(gc1,gc2...gc h )

[0119] gc i =∑α j x j

[0120]

[0121] The functions of concat and softmax in the formula are consistent with those of the model layers with the same name.

[0122] The decoding module Decoder Block uses the Transformer Decoder Block structure, which is composed of a multi-head attention module with a mask in which the output part participates, a multi-head attention module in which the input and output parts participate, and a forward propagation module in series. The modules are connected using addition layers and normalization layers.

[0123] The entire model uses 4 MASTER encoding modules and 3 Transformer decoding modules, both of which require adding positional encoding. The input image will pass through two additional convolutional layers, and the output of the final decoding block will pass through a linear layer and a Softmax layer.

[0124] The model uses the Adam optimizer and the loss uses the cross-entropy loss function.

[0125] measure:

[0126] loss TLR =mean(CrossEntropyLoss(sequence))

[0127] The Table Structure Recognition (TSR) is responsible for predicting the overall table structure, cell alignment information, and the approximate positions of corresponding cells. It takes a table image as input and outputs a sequence of table structure + cell alignment information and cell coordinate predictions. The training starting point for the table structure sequence is that since a table and its HTML format representation are in one-to-one correspondence, the tag sequence in HTML format should be able to perfectly reflect its table structure. Therefore, learning the table structure is simplified to learning the HTML tag sequence. If the set of HTML table tags is regarded as a kind of dictionary, then learning the table structure is essentially a task of converting an image to a sequence, that is, img2seq.

[0128] In this solution, the cell alignment information is embedded as a cell label attribute into the annotation of the table structure sequence. Therefore, when training the model, the tags of the alignment information can be regarded as a special kind of table structure tokens (etymology). By expanding the structure sequence dictionary used by the model, the model can additionally carry out the alignment information in the table while still taking the training to parse the table structure sequence as the training target. To enable this embedding to obtain relatively good training results, the model also needs to make corresponding adjustments. Embedding the alignment information will significantly increase the length of the structure sequence and increase the difficulty of model training. To minimize this impact as much as possible, the model will try to merge the tokens describing the left and right alignment information with the subsequent closing cell tokens. Or, in other words, the model will reduce all tokens. Due to the existence of the table generation algorithm, the order of the training set annotations can be controlled without affecting the correctness. By always controlling the alignment information at the end of the cell label description, the cell alignment tokens and the closing cell tokens can always be adjacent to each other in the structure sequence. Therefore, this combination of merging is feasible. On the other hand, since the alignment information exists in parallel with other structure information inside the cell, swapping the order before the cell information does not affect the description effect. Therefore, the model will also sort the tokens before processing the structure sequence to ensure the premise of this combination merge. The same technique can be used for the up and down alignment information. Since this type of alignment information in the generated data can only appear in multi-row cells, the starting token of its cell is always . By adjusting the order, the up and down alignment information can be bound to the cell opening tag . In this way, the length increase caused by the embedding of new tokens can be almost compensated (this compression can be regarded as embedding the cell alignment information into the opening / closing cell tags in the original sequence).

[0129] The model is divided into encoding and decoding. The encoding part uses 4 MASTER encoding modules, which is consistent with the text line recognition model. The model divides the decoding part into two paths to train the table structure sequence prediction and the cell position prediction respectively (due to the previous adjustment, the cell alignment information can be brought out through the table structure sequence prediction). The two paths use 3 Transformer decoding modules respectively. The first decoding module is shared by both, and the subsequent decoding modules will operate independently.

[0130] When training the structure sequence, the tokens expressing the structure and the tokens expressing the alignment are not distinguished. The prediction result is the structure sequence composed of the two and the position prediction of the cell. Before calculating the loss, the compression step of the table structure sequence needs to be restored to facilitate the subsequent separation of the structure information and the alignment information. The calculation of the loss is divided into three parts. The first part is the Ioss of the table structure prediction, which is measured by the cross-entropy loss function Cross-Entropy Loss. It should be noted that when calculating the Ioss of the table structure, all tokens expressing the alignment information will be skipped. This is achieved by masking the irrelevant parts with a mask. The second part is the Ioss of the cell position prediction, which calculates the L1Ioss of the cell position coordinates, which is further divided into two parts, namely the Ioss of the horizontal coordinate and the Ioss of the vertical coordinate. This part of the position calculation will skip the blank cells and is achieved by the same mask technique. The third part is the Ioss of the cell alignment information, which also uses the cross-entropy loss function and uses a similar mask technique to mask the parts that are not related to the alignment. All the above Ioss will calculate their average mean, and the final Ioss of this part is their weighted sum.

[0131] loss TsR =loss structure +γloss align +loss position

[0132] loss structure =mean(CrossEntropyLoss(sequence structure ))

[0133] loss align =mean(CrossEntropyLoss(sequence align ))

[0134] loss position =mean(L1loss x )+mean(L1loss y ).

[0135] The cell text matching algorithm is responsible for matching the text (row) position output by the text line detection TLD to the cell position output by the table structure recognition TSR. The input is the text line coordinates of text detection and the cell coordinate prediction of structure recognition, and the output is the pairing relationship between the cell and the text (row).

[0136] The matching rules between cells and text lines are mainly divided into three categories: center point matching, IoU (Intersection over Union) overlap matching, and distance matching. Center point matching gives priority to pairs whose center points of text lines are located inside the cell, IoU overlap matching gives priority to pairs whose overlapping area accounts for the largest ratio of the total area of ​​the two, and distance matching gives priority to pairs whose distance is closest. The three rounds of matching are performed in sequence.

[0137] In particular, for cells with alignment information in the prediction results, the predicted cell range will be additionally corrected. For example, for left- and right-aligned cells, the predicted range will be appropriately expanded toward the alignment side, and text lines that deviate from the original prediction range to a certain extent in this direction will be tolerated for matching; no additional corrections are made for cells that are centered or have no alignment information. Such corrections are due to the fact that the imperfect prediction of the cell position in the table structure prediction is always more inclined to the center of the original cell. When the cell is too large or the aligned text line is relatively biased, the position of the text line may exceed the predicted range of the cell, resulting in errors in the matching results.

[0138] A side effect of the above expansion correction is that it will generate many intersections of cell prediction intervals. This intersection will reduce the accuracy of the center point matching rule. Therefore, in the center point matching stage, the selected pairing will be temporarily recorded. When the stage is over, all text lines that only match a single cell will be assigned, and text lines that match multiple or zero cells will be left for subsequent IoU rules to judge.

[0139] The judgment criterion of the IoU rule is the ratio of the intersection area of ​​the cell and the text line divided by the joint area of ​​the two. The two with the highest scores are paired according to the value.

[0140] If the center point and IoU overlap cannot be matched, the text lines will be paired based on the distance matching principle: the cell will be matched with the text closest to the example. In addition, a rule for alignment information is added in this stage: if the relative direction of the text line and the cell is opposite to the alignment information predicted by the cell, the pairing will be prioritized in this stage. This is based on a model premise that the text within a single cell should maintain the same alignment.

[0141] After all matches are completed, the cell will be checked to determine whether the position of the text line inside it does not deviate from its predicted alignment. Cells without alignment information are considered centered. If the difference is greater than the threshold, the corresponding cell alignment information is modified to match its content.

[0142] After the training of S3, text line detection, text line recognition, and table structure recognition is completed, model inference is performed separately; for table structure recognition and text detection, the input is a table data image. Table structure recognition will output a predicted sequence (structure + alignment) and the predicted coordinates corresponding to the cells in the sequence. The text detection part will output the coordinates of all text lines, and the image of the coordinate part can be captured for subsequent text recognition. The text recognition part inputs a text image to give the corresponding text output. After the reasoning results of the three parts are obtained, first, the merging algorithm will receive the structure sequence given by the table structure recognition and use it as the framework of the HTML table. Secondly, the position prediction results of the table structure recognition and text line detection parts will be handed over to the cell text matching algorithm to obtain pairing information. Finally, the merging algorithm will match the text lines to the corresponding positions in the table structure through pairing according to certain rules. During the merging process, additional optimization algorithms can be added for different table types based on alignment information. For example, if it is detected that most of the text in the right-aligned table is numeric, it can be considered that the alignment information of the table has a certain correlation with the text type (such as the numbers in some Excel screenshots are automatically right-aligned). At this time, if numbers and other characters are detected mixed in the right-aligned cells, the non-numeric characters can be corrected according to the rules (such as the confusion between the number 6 and the euro symbol, and numbers and English letters at low resolution), thereby further improving the accuracy of the text recognition part.

[0143] S4. After the inference results of the three parts are obtained, first, the merging algorithm will receive the structure sequence given by the table structure recognition and use it as the framework of the HTML table; secondly, the position prediction results of the table structure recognition and text line detection parts will be handed over to the cell text matching algorithm to obtain pairing information; finally, the merging algorithm will match the text lines into the corresponding positions in the table structure according to certain rules through pairing;

[0144] S5. For the input table image, the trained model will obtain an HTML table with alignment information and convert it into the corresponding table format through conversion code;

[0145] For the input table image, the model will get an HTML table with alignment information. Finally, you only need to convert it into the corresponding table format (such as Excel) through the conversion code. The text in the header tag () is bold by default, and the subscript tag ( ,)、文本样式标签(粗体、斜体、下划线)内的文本将设置为对应的字体格式，对齐信息将设置为对应的单元格格式，跨行跨列单元格可以通过合并单元格进行。

[0146] 最后应说明的是：以上所述仅为本发明的优选实施例而已，并不用于限制本发明，尽管参照前述实施例对本发明进行了详细的说明，对于本领域的技术人员来说，其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换,凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims

1. A table parsing method based on image-to-sequence technology that introduces cell alignment information, characterized in that: The steps include: S1. The method includes a table generation algorithm: generating parameters according to user input, the parameters determine the number and size of the generated tables, whether there are cells across rows and columns, the length range of cells across rows and columns, and the complexity of the table structure; Generate the data set required for training by the table generation algorithm; S2, establishing a table parsing model and training the model, wherein the table parsing model includes a text line detection model, a text line recognition model, and a table structure recognition model; After the training of S3, text line detection, text line recognition, and table structure recognition, model inference is performed separately; S4. After the inference results of the three parts are obtained, first, the merging algorithm will receive the structure sequence given by the table structure recognition and use it as the framework of the HTML table; secondly, the position prediction results of the table structure recognition and text line detection parts will be handed over to the cell text matching algorithm to obtain pairing information; finally, the merging algorithm will match the text lines into the corresponding positions in the table structure according to certain rules through pairing; S5. For the input table image, the trained model will obtain an HTML table with alignment information and convert it into the corresponding table format through conversion code.

2. The table parsing method based on image-to-sequence technology that introduces cell alignment information according to claim 1, characterized in that: In step S1, the table is generated as follows: S11, generating a two-dimensional identification structure based on the required table size; S12, selecting a cell within a certain range according to the parameters and marking it as the starting point of the cross-row and cross-column cell, and marking its spanning length and direction; S13, the span length is randomized within a certain range according to the parameters, while ensuring that no other cross-row or cross-column cells are overlapped; S14. Several cells in the spanning range following the marked cell will be marked as covered at the same time. These positions are invisible in the generated table and will not participate in the subsequent randomization of the positions of cells across rows and columns.

3. The table generation algorithm included in the table parsing method based on image-to-sequence technology that introduces cell alignment information according to claim 2 is characterized in that: For a table with complexity equal to n, the algorithm will first randomly find a table block and cut it into two parts. The cutting direction is random from horizontal to vertical. After cutting, any part of the table block can be used as the object of the next cutting. The cutting operation continues n times.

4. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 1, characterized in that: In step S2, the table parsing model is established as follows: S21, Text Line Detection TLD is responsible for detecting the position where the text appears in the image. It reads in a table image and outputs text line coordinates and a text line image intercepted based on the coordinates. The coordinates will be used together with the coordinate prediction in the table structure recognition to match the cell text. The intercepted text line image will be handed over to the text recognition model TLR for further processing. S22, Text Line Recognition TLR is responsible for detecting the content in the text image and converting it into text, reading in as a text image and outputting as a text recognition result; the text information will be subsequently handed over to the merging optimization algorithm to be embedded into the corresponding table position; S23, table structure recognition TSR is responsible for predicting the overall table structure, cell alignment information and the approximate position of the corresponding cell; the input is a table image, and the output is a table structure + cell alignment information sequence and cell coordinate prediction; The cell coordinate prediction will be used together with the results of the text detection model to match the cell text; the structure plus the cell alignment information sequence will be handed over to the merge optimization algorithm as the framework of the table; S24, the cell text matching algorithm is responsible for matching the text position output by the text line detection TLD to the cell position output by the table structure recognition TSR, the input is the text line coordinates of the text detection and the cell coordinate prediction of the structure recognition, and the output is the pairing relationship between the cell and the text; S25, the merge optimization algorithm is responsible for receiving the output of all the above parts and integrating them into an HTML table, reading in the table structure + cell alignment information sequence given by the table structure recognition TSR, the cell text pairing relationship given by the cell text matching algorithm, and the cell text given by the text line recognition, embedding the text into the table structure framework through the pairing relationship, and further optimizing the accuracy of text recognition through cell alignment information, statistical information and some preset rules; the output is a synthesized HTML table; S26, the format conversion code is responsible for converting the HTML table into the required table.

5. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 1, characterized in that: In step S2, the model architecture is: Arrow: refers to the transfer of the output of the previous part to the input of the part pointed by the arrow. If the previous part is not explicitly described, it means that the data is transferred from outside the component / model; Linear layer: performs a linear calculation on the input; Addition layer: adds two inputs; Normalization layer: normalize the input; Convolutional layer: performs convolution on the input; Matrix multiplication layer: performs matrix multiplication on the input; Mask layer: masks part of the input; Scaling layer: performs scaling on the input; Connection layer:: connects the inputs together; ReLU: A rectified linear function with the formula: f(x)=max(0,x); Softmax: A probability conversion function with the formula: Embedding / vectorization layer: maps the input to a vector space; The combination of layers can form basic modules. The following modules will be used in the model: Scaled dot product attention module: The input is Queue, Key and Value respectively, and the output formula is: Multi-head attention module: that is, multi-head SDPA module + connection layer + linear layer; One of the modules of the forward propagation module structure is a combination of a set of linear layers, ReLU linear layers, and its formula is: Feedforward(x)=ReLU(xW1+b1)W2+b2.

6. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 1, characterized in that: In step S2, the text line detection model is responsible for detecting the position where the text appears in the image, reading in a table image, and outputting text line coordinates and a text line image intercepted based on the coordinates; The model uses the PSENet text detection algorithm, uses the deep residual network ResNet as the backbone, and uses feature pyramid networks to generate several feature layers. It connects feature samples of several scales and performs convolution processing to complete the prediction.

7. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 6, characterized in that: In step S2, the text line recognition model is responsible for detecting the content in the text image and converting it into text, reading in as a text image and outputting as a text recognition result; The text line recognition model uses the image-to-sequence img2seq, supports single-line and multi-line recognition, and can directly use table data and cell coordinate information to intercept text parts for training; The encoding module Encoder Block adopts the MASTER structure, which consists of a ResNet residual module Residual Block connected to a multi-faceted global context attention module MAGC and a convolutional layer; The formula of the multi-aspect global attention module MAGC of the residual module is as follows: MAGC(x)=Concat(gc1,gc2...gc h ) gc i =∑α j x j The functions of concat and softmax in the formula are consistent with those of the model layers with the same name.

8. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 1, characterized in that: In step S2, the table structure recognition model is responsible for predicting the overall table structure, cell alignment information and the approximate position of the corresponding cell. The input is a table image, and the output is a table structure + cell alignment information sequence and cell coordinate prediction.

9. The table parsing method based on image-to-sequence technology with cell alignment information introduced according to claim 1, characterized in that: In step S2, the cell text matching algorithm is responsible for matching the text position output by the text line detection TLD to the cell position output by the table structure recognition TSR, reading in the text line coordinates of text detection and the cell coordinate prediction of structure recognition, and outputting the pairing relationship between the cell and the text.

Images