Table data recovery method based on OCR model

Abstract

The invention discloses a table data recovery method based on an OCR model, and the method comprises the steps: S1, recognizing table types through employing a lightweight model, and enabling the table types to comprise a wired table and a wireless table; s2, respectively acquiring a logic coordinate and a physical coordinate of a detection frame of the wireless table and a cell physical coordinate of the wired table; s3, recovering a logical structure of the table according to the detected table coordinates, and determining a row-column relationship of the table; s4, performing character recognition on the image by using a text recognition model, obtaining the position and content of a character box, performing accurate matching of characters and cells by calculating the coordinate overlap ratio of the OCR recognition character box and the cell box, and determining the character content in each cell; and S5, for the cells which are not matched with the characters, cutting the cell areas, carrying out secondary OCR identification on the image areas of the cells, and supplementing missing character contents. According to the method, the automatic identification and analysis capability of the complex table data is improved.

Description

Technical Field

[0001] The invention relates to a table data recovery method based on an OCR model. Background Art

[0002] In today's information age, companies and individuals are faced with a large amount of tabular data, such as financial statements, survey results, and statistical data. These tabular data usually exist in images, PDFs, or other unstructured formats. How to efficiently and accurately identify and extract these data for further analysis and use has become an important task. However, due to the diversity and complexity of tabular data, traditional manual entry or rule-based parsing methods are time-consuming and prone to errors.

[0003] Optical character recognition (OCR) technology is an important research direction in the field of computer vision, which aims to convert scanned or image-formatted documents into editable and searchable text. Traditional OCR technology usually relies on fixed templates and simple character matching algorithms. These methods are often unable to handle complex and format-changing tabular data, and the recognition accuracy and efficiency cannot meet actual needs.

[0004] In recent years, with the rapid development of artificial intelligence technology, OCR models such as RAG (Retrieval-Augmented Generation) and deep learning have made significant progress in the field of information extraction. RAG achieves accurate information extraction and generation by combining document retrieval and generation models; while modern OCR technology combines deep learning methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) to recognize and parse complex structured table data. Through these advanced technologies, the system can not only recognize the characters in the table, but also parse the row and column structure and semantic information of the table.

[0005] At present, some existing table recognition methods have been applied in practice. Traditional methods include rule-based table parsing and deep learning-based table detection. Rule-based parsing methods detect straight lines and borders in images and use preset rules to identify the structure and cell position of the table. Deep learning-based table detection methods use models such as convolutional neural networks to automatically detect table areas and structures.

[0006] However, these methods face challenges in dealing with diverse table types, especially wireless tables and complex wired tables. Since the table structure is complex and changeable, there may be blurred, missing lines or irregular table formats. Traditional methods often find it difficult to accurately capture the global structure and detailed information of the table, resulting in inaccurate, incomplete and inconsistent recognition results.

[0007] Therefore, a table data recovery method based on OCR model is proposed. Summary of the invention

[0008] The purpose of the present invention is to overcome the existing defects and provide a table data recovery method based on an OCR model, thereby improving the automatic recognition and parsing capabilities of complex table data.

[0009] The technical solution to achieve the above purpose is:

[0010] A table data recovery method based on an OCR model, comprising:

[0011] Step S1, using a lightweight model to identify table types, the table types include wired tables and wireless tables;

[0012] Step S2, respectively obtaining the logical coordinates and physical coordinates of the detection frame of the wireless table and the physical coordinates of the cells of the wired table;

[0013] Step S3, based on the detected table coordinates, restore the logical structure of the table and determine the row and column relationship of the table;

[0014] Step S4, using a text recognition model to perform text recognition on the image, obtain the position and content of the text box, and accurately match the text and the cell by calculating the coordinate overlap between the OCR recognized text box and the cell box to determine the text content in each cell;

[0015] Step S5, for cells that do not match text, the cell area is cropped, and the image area of ​​these cells is subjected to secondary OCR recognition to supplement the missing text content;

[0016] Step S6, combining the matched OCR recognition result with the physical coordinates and logical coordinates of the cell and converting them into a unified result format;

[0017] Step S7, generating a table in HTML format according to the logical cell position and text content, and completely presenting the structure and content of the table.

[0018] Preferably, in step S1, the table image is used as the input of the lightweight model, and the output is the table type probability.

[0019] Preferably, in step S2, if the table type is identified as a wireless table, then:

[0020] Use the Detection model to infer the heat map, size, angle, and offset features of the table in the image;

[0021] Use a series of algorithms to decode the feature map output by the model into specific information of the detection box;

[0022] Then, NMS (non-maximum suppression) is used to suppress duplicate detections, and the top-K most likely detection results are selected according to the confidence scores. The detection results are transformed, merged, and filtered to obtain the logical and physical coordinates of the detection box.

[0023] Finally, the logical coordinates are corrected and filtered;

[0024] If the identification table type is a wired table, then:

[0025] Use a pre-trained ONNX (an open file format for machine learning) model to detect table lines in the image and obtain the physical structure of the table;

[0026] The image is preprocessed, and the detection results are post-processed, and finally the physical coordinates of the cell are obtained through coordinate transformation and sorting.

[0027] Preferably, in step S2, the specific information of the detection box includes but is not limited to the bounding box coordinates, confidence score, and category.

[0028] Preferably, in step S2, the physical structure of the table includes polygonal coordinates of table cells.

[0029] Preferably, in step S2, the preprocessing includes:

[0030] Resizing: scaling the image to a suitable size, maintaining resolution while reducing computational effort;

[0031] Color space conversion: convert the image to grayscale or binary to highlight line features;

[0032] Normalization: Normalize the image to eliminate the effects of lighting and contrast.

[0033] Preferably, in step S2, post-processing of the detection result includes:

[0034] Separate row and column lines;

[0035] Image Resizing

[0036] Morphological operations: Use morphological operations such as corrosion and dilation to enhance lines and connect broken line segments;

[0037] Line extraction: Accurately extract straight lines through Hough transform or other algorithms, and correct and align them;

[0038] Line adjustment;

[0039] Cell generation: Generate the polygon coordinates of the cell based on the intersection of row and column lines.

[0040] Preferably, step S3 comprises:

[0041] Step S31, firstly divide the cells into corresponding rows according to the physical coordinates of the cells, i.e., the vertical positions;

[0042] Step S32, determining the table column structure and the total number of columns by confirming the reference column and adjusting all column split points to align the columns;

[0043] Step S33, determining the reference row, determining the initial row segmentation point, and calculating the height of each row to determine the table row structure and the total number of rows;

[0044] Step S34, finally determining the starting row, ending row, starting column, ending column in the logical table, and whether it is a merged cell according to the physical size of the cell.

[0045] The beneficial effects of the present invention are:

[0046] 1) Accuracy and reliability of wireless table detection: This invention uses the Detection model to extract features such as the heat map, size, angle, and offset of the table in the image, and adopts algorithms such as Top-K screening to improve the accuracy and reliability of wireless table detection and ensure the validity and correctness of the logical and physical coordinates of the detection box;

[0047] 2) Accuracy and completeness of wired table line detection: Pre-processing and post-processing of images are performed to accurately extract table lines and obtain the polygonal coordinates of cells to ensure the accuracy and completeness of wired table line detection;

[0048] 3) Accuracy and completeness of table structure recovery: The present invention aims to accurately restore the logical structure of the table, determine the starting row, ending row, starting column and ending column of each cell in the logical table, and correctly identify merged cells by dividing the corresponding rows and columns according to the vertical and horizontal positions of the cells, determining the reference rows and columns, and adjusting the row and column division points;

[0049] 4) Accuracy of text recognition and cell matching: The present invention uses a text recognition model to perform text recognition, obtain the position and content of the text box, and calculate the coordinate overlap between the OCR recognized text box and the cell box, aiming to achieve accurate matching of text and cells and ensure that the text content in each cell is correct;

[0050] In summary, the present invention realizes the complete extraction from image to table content through table line detection, table structure recovery and OCR recognition. In addition, it supports the use of externally provided OCR results or internal OCR recognition. At the same time, for cells that fail to match text, a re-recognition mechanism is provided to improve the integrity of recognition. The generated HTML table can fully present the structure and content of the original table; it can accurately restore the physical and logical structure of the table, accurately identify the text content in the cell, generate a unified result format and high-quality table presentation, improve the quality and efficiency of table recognition, and provide reliable technical support for the automatic processing of table information. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a table data recovery method based on an OCR model of the present invention;

[0052] Figure 2 It is a specific flow chart of restoring the logical structure of the table and determining the row-column relationship of the table according to the detected table coordinates in the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0054] The present invention will be further described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, a table data recovery method based on an OCR model includes:

[0056] Step S1, using a lightweight model to identify table types, the table types include wired tables and wireless tables.

[0057] In the embodiment, the table image is used as the input of the lightweight model, and the output is the table type probability.

[0058] In the embodiment, the lightweight models include: MobileNet and EffectiveNet.

[0059] In another embodiment, a deep learning model ResNet or DenseNet is used to identify the table type. These models have good performance in feature extraction and classification accuracy, and can further improve the accuracy of table type judgment. According to actual application requirements and computing resources, a suitable model can be selected to balance performance and efficiency.

[0060] Step S2, respectively obtaining the logical coordinates and physical coordinates of the detection frame of the wireless table and the physical coordinates of the cells of the wired table.

[0061] In the embodiment, if the identification table type is a wireless table, then:

[0062] Use the Detection model to infer the heat map, size, angle, and offset features of the table in the image.

[0063] In another embodiment, a Transformer-based detection model can be used to infer the heat map, size, angle, and offset features of the table in the image. The self-attention mechanism can be used to better capture the complex structure and long-distance dependencies in the table, which helps to improve the accuracy of wireless table detection, especially when processing complex table layouts.

[0064] A series of algorithms are used to decode the feature map output by the model into specific information of the detection box.

[0065] Then, NMS (non-maximum suppression) is used to suppress duplicate detections, and the Top-K most likely detection results are selected according to the confidence scores. The detection results are transformed, merged, and filtered to obtain the logical coordinates and physical coordinates of the detection box.

[0066] Finally, the logical coordinates are corrected and filtered.

[0067] If the identification table type is a wired table, then:

[0068] Use the pre-trained ONNX model to detect table lines in the image and obtain the physical structure of the table.

[0069] In another embodiment, a classic edge detection algorithm (such as Canny, Sobel) is used to assist in enhancing the edges of the image and highlighting the features of the table lines. Combined with morphological operations and line extraction algorithms, the table lines can be detected and extracted more accurately, especially when processing images with more noise or thinner lines.

[0070] The image is preprocessed, and the detection results are post-processed, and finally the physical coordinates of the cell are obtained through coordinate transformation and sorting.

[0071] In an embodiment, the specific information of the detection box includes but is not limited to the bounding box coordinates, confidence score, and category.

[0072] In an embodiment, the physical structure of the table includes polygonal coordinates of table cells.

[0073] In an embodiment, the pretreatment comprises:

[0074] Resizing: scaling the image to a suitable size, maintaining resolution while reducing computational effort;

[0075] Color space conversion: convert the image to grayscale or binary to highlight line features;

[0076] Normalization: Normalize the image to eliminate the effects of lighting and contrast.

[0077] In the embodiment, the post-processing of the detection result includes:

[0078] Separate row and column lines;

[0079] Image Resizing

[0080] Morphological operations: Use morphological operations such as corrosion and dilation to enhance lines and connect broken line segments;

[0081] Line extraction: Accurately extract straight lines through Hough transform or other algorithms, and correct and align them;

[0082] Line adjustment;

[0083] Cell generation: Generate the polygon coordinates of the cell based on the intersection of row and column lines.

[0084] Step S3, based on the detected table coordinates, restore the logical structure of the table and determine the row and column relationship of the table.

[0085] like Figure 2 As shown, step S3 includes:

[0086] Step S31, firstly divide the cells into corresponding rows according to the physical coordinates of the cells, that is, the vertical positions.

[0087] Step S32, determining the table column structure and the total number of columns by confirming the reference column and adjusting all column division points to align the columns.

[0088] Step S33, by determining the reference row, determining the initial row segmentation point, and calculating the height of each row to determine the table row structure and the total number of rows.

[0089] Step S34, finally determining the starting row, ending row, starting column, ending column in the logical table, and whether it is a merged cell according to the physical size of the cell.

[0090] In another embodiment, a deep learning model, such as a model based on Graph Neural Networks, is used to predict the logical structure of a table; by learning a large amount of table data, the model can automatically infer the row and column relationships and merging conditions of cells, thereby improving the automation and accuracy of table structure recovery.

[0091] Step S4, using the text recognition model to perform text recognition on the image, obtain the position and content of the text box, and accurately match the text and the cell by calculating the coordinate overlap between the OCR recognized text box and the cell box to determine the text content in each cell.

[0092] In another embodiment, in addition to using the existing text recognition model, a self-supervised learning method can be used to train an OCR model specifically optimized for table scenarios; by using unlabeled data for pre-training, the model can better adapt to special characters, fonts, and layouts in the table, thereby improving the accuracy of text recognition.

[0093] Another embodiment is that in addition to coordinate coincidence, a content-based matching method can be introduced; for example, natural language processing technology is used to perform semantic analysis on the text content, and the cell position and context information are combined to determine the cell to which the text belongs; this method can improve the accuracy of text matching when the text box position is inaccurate or the cell boundary is blurred.

[0094] Step S5, for cells that do not match text, the cell area is cropped, and the image area of ​​these cells is subjected to secondary OCR recognition to supplement the missing text content to ensure that no key information is lost.

[0095] Step S6, combining the matched OCR recognition results with the physical coordinates and logical coordinates of the cells, and converting them into a unified result format to facilitate subsequent table content processing and exporting.

[0096] Step S7, generating a table in HTML format according to the logical cell position and text content, and completely presenting the structure and content of the table.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A table data recovery method based on an OCR model, characterized in that: include: Step S1, using a lightweight model to identify table types, the table types include wired tables and wireless tables; Step S2, respectively obtaining the logical coordinates and physical coordinates of the detection frame of the wireless table and the physical coordinates of the cells of the wired table; Step S3, based on the detected table coordinates, restore the logical structure of the table and determine the row and column relationship of the table; Step S4, using a text recognition model to perform text recognition on the image, obtain the position and content of the text box, and accurately match the text and the cell by calculating the coordinate overlap between the OCR recognized text box and the cell box to determine the text content in each cell; Step S5, for cells that do not match text, the cell area is cropped, and the image area of ​​these cells is subjected to secondary OCR recognition to supplement the missing text content; Step S6, combining the matched OCR recognition result with the physical coordinates and logical coordinates of the cell and converting them into a unified result format; Step S7, generating a table in HTML format according to the logical cell position and text content, and completely presenting the structure and content of the table.

2. The table data recovery method based on the OCR model according to claim 1, characterized in that: In step S1, the table image is used as the input of the lightweight model, and the output is the table type probability.

3. The table data recovery method based on the OCR model according to claim 1, characterized in that: In step S2, if the table type is identified as a wireless table, then: Use the Detection model to infer the heat map, size, angle, and offset features of the table in the image; Use a series of algorithms to decode the feature map output by the model into specific information of the detection box; Then, NMS is used to suppress duplicate detections, and the top-K most likely detection results are selected according to the confidence scores. The detection results are transformed, merged, and filtered to obtain the logical and physical coordinates of the detection box. Finally, the logical coordinates are corrected and filtered; If the identification table type is a wired table, then: Use the pre-trained ONNX model to detect table lines in the image and obtain the physical structure of the table; The image is preprocessed and the detection results are post-processed, and finally the physical coordinates of the cell are obtained through coordinate transformation and sorting.

4. The table data recovery method based on the OCR model according to claim 3 is characterized in that: In step S2, the specific information of the detection box includes but is not limited to the bounding box coordinates, confidence score, and category.

5. The table data recovery method based on the OCR model according to claim 3 is characterized in that: In step S2, the physical structure of the table includes polygonal coordinates of table cells.

6. The method for recovering table data based on an OCR model according to claim 3, characterized in that: In step S2, the preprocessing includes: Resizing: scaling the image to a suitable size, maintaining resolution while reducing computational effort; Color space conversion: convert the image to grayscale or binary to highlight line features; Normalization: Normalize the image to eliminate the effects of lighting and contrast.

7. The method for recovering table data based on an OCR model according to claim 3, characterized in that: In step S2, post-processing of the detection results includes: Separate row and column lines; Image Resizing Morphological operations: Use morphological operations such as corrosion and dilation to enhance lines and connect broken line segments; Line extraction: Accurately extract straight lines through Hough transform or other algorithms, and correct and align them; Line adjustment; Cell generation: Generate the polygon coordinates of the cell based on the intersection of row and column lines.

8. The method for recovering table data based on an OCR model according to claim 1, characterized in that: The step S3 comprises: Step S31, firstly divide the cells into corresponding rows according to the physical coordinates of the cells, i.e., the vertical positions; Step S32, determining the table column structure and the total number of columns by confirming the reference column and adjusting all column split points to align the columns; Step S33, determining the reference row, determining the initial row segmentation point, and calculating the height of each row to determine the table row structure and the total number of rows; Step S34, finally determining the starting row, ending row, starting column, ending column in the logical table, and whether it is a merged cell according to the physical size of the cell.

Images