A method and system for recognizing table-type text images

Abstract

The present invention discloses a method for recognizing a table-type text image. First, the table-type text image is grayed to obtain a grayscale image, the grayscale image is binarized using an automatic threshold, and then the horizontal and vertical line structure of the table is extracted using a Hough transform to obtain horizontal and vertical line coordinates and cells; then the text content in the table-type text image is detected and a text image with a label is obtained, the label having coordinates; character recognition is performed on the text image; then the cell of step S1 is automatically matched with the label of step S2, and the character recognition result of step S2 is output to the corresponding cell. The method is used to realize the recognition of the content of a financial information scanned file, and can prepare to recognize the table and the text content therein.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, pattern recognition, and image processing technology, and relates to a method and system for identifying the content of tabular text images suitable for identifying financial information scanned files, etc. Background Art

[0002] With the rapid development of computer vision, optical character recognition (OCR), one of the classic problems of computer vision, has quickly broken through the bottleneck of the traditional technical framework and has been widely used in many fields such as natural scene text recognition, license plate recognition, and document and bill recognition. Traditional optical character recognition technology is a classic pattern recognition problem, including several steps such as image preprocessing, feature extraction, and classification using classifiers, and has formed a relatively complete technical system. However, with the development of deep learning, the traditional optical character recognition framework has been broken. As a new research hotspot, optical character recognition based on deep learning has shown more vitality and broader application scenarios. Therefore, optical character recognition technology is currently facing huge opportunities and challenges. Most of the existing file content recognition methods use traditional scanning matching algorithms, which lack special processing of complex structures such as tables and paragraphs in files, and secondary calibration of erroneous content generated by detection and recognition. Summary of the invention

[0003] The purpose of the present invention is to provide a method for recognizing table-type text images, which is used to realize the recognition of contents such as financial information scanned documents, and can be prepared to recognize tables and the text contents therein, and can recognize layouts and headers and footers with high recognition accuracy.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for recognizing a table-type text image comprises the following steps:

[0006] S1: Convert the table text image to grayscale to obtain a grayscale image, apply automatic threshold binarization to the grayscale image, and then use Hough transform to extract the horizontal and vertical line structure of the table to obtain the horizontal and vertical line coordinates and cells;

[0007] S2: Detecting the text content in the table-type text image and obtaining a text image with a label, wherein the label has coordinates; performing character recognition on the text image;

[0008] S3: Automatically match the coordinates of the cells of S1 with the labels of S2, and output the character recognition results of S2 to the corresponding cells.

[0009] Furthermore, the method further comprises the following steps:

[0010] S4: After character recognition is performed on the text image in S2, the similarity of the table-type text images of two adjacent pages is calculated in units of rows using a calculation function of text sequence similarity based on intersection-over-union ratio, and all table-type text images are traversed to obtain the global text similarity;

[0011] S5: Perform a global search on all tabular text images based on global text similarity, and then extract the header and footer.

[0012] Furthermore, the following steps are included:

[0013] S6: Perform a global traversal on the labels of the text image in step S2 to determine the value range of the horizontal coordinate;

[0014] S7: Use adaptive thresholds to segment the main text content. If the horizontal coordinates of both endpoints of the current line are greater than the threshold coordinates, the line is a title; if the horizontal coordinate of only one endpoint of the current line is greater than the threshold coordinate, it is determined to be a new paragraph, and typesetting output is performed accordingly.

[0015] Preferably, step S2 is specifically implemented by the following steps:

[0016] S21: inputting the tabular text image into a feature pyramid network, and the feature pyramid network samples and cascades at the same ratio to generate feature F;

[0017] S22: Using the feature F to predict the probability map P and the threshold map T;

[0018] S23: Compute approximate binary graph using probability graph P and threshold graph T

[0019]

[0020] Where P∈R H*W is a probability map generated by the segmentation network, H represents the image height, W represents the image width, T is the adaptive threshold map learned by the network; K represents the magnification factor, (i, j) represents the coordinate point in the map;

[0021] S24: Deformable convolution is used to provide the model with a flexible receptive field to assist text instances with extreme aspect ratios;

[0022] S25: Generate labels for text images: Describe each polygon G of the text image region with a set of line segments:

[0023]

[0024] Among them, S k is a set of line segments, n is the number of vertices,

[0025] The positive area is then generated by reducing polygon G to a first polygon Gs using the Vatti clipping algorithm, where the reduction offset D is calculated based on the perimeter L and area A of the original polygon:

[0026] r is the shrinkage ratio;

[0027] S26: Generate labels for threshold map T: with the same offset D to the second polygon G d Expanding the polygon G, and then generating a label of the threshold map T by calculating the distance to the closest line segment in the polygon G;

[0028] S27: Use CNN to extract the convolution feature map of the input text image, use the deep bidirectional LSTM network to continue to extract the text sequence features based on the convolution features, and then perform softmax on the output to obtain the character output.

[0029] Preferably, in step S3, after automatically matching the coordinates of the cell of S1 with the label of S2 and outputting the character recognition result of S2 to the corresponding cell, a re-recognition technology is used to perform a secondary calibration on the erroneous content generated by the detection and recognition, and the re-recognition technology is to cut out the cells recognized as empty and then recognize them.

[0030] At the same time, the present invention also provides a table-type text image recognition system, and the recognition system can execute the above-mentioned table-type text image recognition method.

[0031] In the above technical solution, the recognition of cells is realized by identifying the horizontal and vertical lines and their coordinates in the table, and labels with coordinates are added to the text content. The text content in the cell is restored by automatic coordinate matching, and a method for recognizing headers, footers and layouts is also provided. The present invention can be applied to recognition scenarios of a large number of table-type text images such as financial information scanned files, saving manpower while improving recognition accuracy. DETAILED DESCRIPTION

[0032] The present invention will be further described below:

[0033] The object recognized by the present invention is a table-type text image. If a PDF file is obtained by scanning, it must first be converted into an image.

[0034] S1: Table recognition technology for structured data recovery. The specific steps are as follows:

[0035] The text image in the table text picture is grayed out, and the automatic threshold is used for binarization based on the grayscale image. Then, the Hough transform is used to extract the horizontal and vertical line structure of the table and determine the coordinates of the horizontal and vertical lines, thereby dividing each cell.

[0036] S2: Text detection and recognition, the specific steps are:

[0037] The text image of the tabular text image to be detected is input into the feature pyramid network, which includes a roll operator and two deconvolution operators; the feature pyramid network is used to sample and cascade at the same ratio to generate feature F.

[0038] The feature F is used to predict the probability map P and the threshold map T.

[0039] Compute the approximate binary graph using the probability graph P and the threshold graph T

[0040]

[0041] Where P∈R H*W It is a probability map generated by the segmentation network. H represents the image height and W represents the image width. is an approximate binary graph, T is the adaptive threshold mapping learned by the network; K represents the amplification factor, which is set to 50 based on experience, and (i, j) represents the coordinate point in the graph.

[0042] Deformable convolution is used to provide the model with a flexible receptive field to assist text instances with extreme aspect ratios;

[0043] Generate labels for text images: Describe each polygon in the text image region with a set of line segments:

[0044]

[0045] Among them, S k is a group of segments, G is each polygon in the text area, and n is the number of vertices, which may be different in different datasets.

[0046] The positive region is then generated by reducing the polygon G to Gs using the Vatti clipping algorithm, where the reduction offset D is calculated based on the perimeter L and area A of the original polygon:

[0047]

[0048] Among them, D is the reduced offset, L is the perimeter, A is the area, and r is the shrinkage ratio, which is set to 0.4 based on experience.

[0049] Generate labels for the threshold map T: Extend the text polygon G to a new polygon with the same offset D, denoted as G d , and then generate the labels of the threshold map by computing the distance to the closest line segment in G.

[0050] Then perform text recognition. The specific steps are:

[0051] CNN is used to extract the convolution feature map of the input image, and a deep bidirectional LSTM network is used to further extract the text sequence features based on the convolution features. The output is then softmaxed to obtain the character output.

[0052] Then, Connectionist Temporal Classification (CTC) is used to adjust the parameters of LSTM through gradients, so that the training samples do not need to be aligned. The gradients are quickly calculated through recursion to solve the problem of character alignment during training.

[0053] S3: Automatically match the coordinates of the text labels obtained in S2 and the cells of the table obtained in S1, thereby recovering the structured text content of the table. Furthermore, the re-recognition technology is used to perform secondary calibration on the erroneous content generated by the detection and recognition, thereby improving the recognition precision and accuracy of the table data.

[0054] Then, identify the header and footer, including:

[0055] S4: Header and footer processing solution based on global text similarity search:

[0056] A calculation function for text sequence similarity based on intersection-over-union ratio is designed. The similarity of two adjacent pages of text files is calculated in units of lines. After traversing the entire text, the global text similarity is obtained.

[0057] S5: Perform a global search on all tabular text images to be recognized based on the similarity of the global text, and then extract the header and footer of the entire text.

[0058] Further, typesetting format recognition is performed:

[0059] S6: Adaptive text layout segmentation: Globally traverse the detection box of the text image to determine the value range of the horizontal coordinate;

[0060] S7: Use adaptive thresholds to segment the main text content. If the horizontal coordinates of both endpoints of the current line are greater than the threshold coordinates, it is determined to be the title part. If the horizontal coordinate of only one vertex of the current line is greater than the threshold coordinates, it is determined to be a new paragraph and typeset output is performed accordingly.

[0061] The present invention also provides a system for implementing the above method, such as a computer system according to the above program and having corresponding hardware.

[0062] This embodiment is only an illustration of the concept and implementation of the invention, and does not limit it. Under the concept of the present invention, technical solutions that have not been substantially changed are still within the scope of protection.

Claims

1. A method for recognizing a table-type text image, characterized in that The steps include: S1: Convert the tabular text image into a grayscale image and apply automatic The threshold is binarized, and then the Hough transform is used to extract the horizontal and vertical line structure of the table to obtain the horizontal and vertical line coordinates. Label and cell; S2: Detect the text content in the tabular text image and obtain a labeled text image The label has coordinates; character recognition is performed on the text image; and the specific implementation is through the following steps: S21: Input the tabular text image into a feature pyramid network. Sampling at the same ratio and concatenating to generate feature F; S22: Using the feature F to predict the probability map P and the threshold map T; S23: Compute approximate binary graph using probability graph P and threshold graph T : ， in, is the probability map generated by the segmentation network, H represents the image height, and W represents the image The width of the slice, T is the adaptive threshold map learned by the network; K represents the amplification factor, and (i, j) represents the map The coordinate points in ; S24: Using deformable convolution to provide the model with a flexible receptive field to help documents with extreme aspect ratios This example; S25: Generate labels for text images: Describe each polygon in the text image region with a set of line segments G： ， Among them, S K is a set of line segments, n is the number of vertices, Then the polygon G is reduced to the first polygon Gs by using the Vatti clipping algorithm to generate the positive area, where the reduced offset D is calculated based on the perimeter L and area A of the original polygon: , r is the shrinkage ratio; S26: Generate labels for threshold map T: with the same offset D to the second polygon G d Expanding on the polygon G, and then generate the labels of the threshold map T by calculating the distance to the closest line segment in the polygon G; S27: Use CNN to extract the convolutional feature map of the input text image, and use the deep bidirectional LSTM network The network continues to extract text sequence features based on the convolution features, and then performs softmax on the output to obtain Character output; then use Connectionist Temporal Classification (CTC) to adjust the LSTM parameters through gradients, so that the training samples do not need to be aligned. Through recursion, the gradients are quickly calculated to solve the problem of character alignment during training; S3: Automatically match the coordinates of the cells in step S1 with the labels in step S2, and After the character recognition result of S2 is output to the corresponding cell, the erroneous content generated by the detection and recognition is recalibrated using the re-recognition technology, wherein the cells recognized as empty are cut out and then recognized; S4: After character recognition is performed on the text image in step S2, the text sequence is The column similarity calculation function calculates the similarity of two adjacent pages of table text images in units of rows. Traverse all tabular text images to obtain global text similarity; S5: Perform a global search on all tabular text images based on global text similarity, and then Remove the header and footer; S6: Perform a global traversal on the labels of the text image in step S2 to determine the value range of the horizontal coordinate; S7: Use the adaptive threshold to segment the text content. If the horizontal coordinates of both endpoints of the current line are greater than the threshold coordinate, the line is a title; if the horizontal coordinate of only one endpoint of the current line is greater than the threshold coordinate, the line is a title. The threshold coordinates are determined as new paragraphs and formatted and output accordingly.

2. A table-type text image recognition system, characterized in that: The recognition system is capable of performing The method for recognizing a tabular text image as claimed in claim 1.

Images