Document table content extraction method and system based on semantic segmentation

Abstract

According to the semantic segmentation-based document table content extraction method and system disclosed by the invention, through a semantic segmentation-based document table content extraction algorithm, the defects in the aspects of table format adaptability, complex table processing capability, data annotation cost, universality, automation degree and the like in the prior art are overcome; the overall performance and the application value of document table content extraction are remarkably improved; according to the method, the accuracy and robustness of table content extraction are improved, the data labeling cost is reduced, the universality of the method is enhanced, and efficient and automatic processing of document table content extraction is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and more specifically, to a method and system for extracting document table content based on semantic segmentation. Background Art

[0002] With the advent of the digital age, the demand for document processing automation is increasing. Traditionally, the extraction of table content usually relies on manual operations or rule-based automation tools. These methods have solved the needs of table data extraction to a certain extent, but there are still many shortcomings, such as strong dependence on table format, limited ability to process complex tables, high data annotation costs, and poor versatility.

[0003] Therefore, the prior art has defects and needs to be improved urgently. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a document table content extraction method and system based on semantic segmentation, which can more effectively improve the accuracy and robustness of table content extraction and reduce data annotation costs.

[0005] A first aspect of the present invention provides a method for extracting document table content based on semantic segmentation, comprising:

[0006] Get table image information;

[0007] Preprocess the table image to obtain a standard table image;

[0008] Based on the preset feature extraction and semantic segmentation methods, the table area and its area position and image features in the standard table image are extracted;

[0009] Slice the table area into rows and columns to get rows and columns;

[0010] The image features are averaged according to rows and columns to obtain row features and column features;

[0011] Merge row features and column features to obtain the overall structural features of the table;

[0012] According to the overall structural characteristics of the table, determine the label of each pixel in the table;

[0013] Based on the label of each pixel in the table, the table area is divided into multiple cell areas containing labels;

[0014] The text content in the cell area containing the label is extracted, and the text content is sorted according to the table structure to obtain structured table data.

[0015] In this solution, the step of preprocessing the table image to obtain a standard table image specifically includes:

[0016] Preprocessing the table image to obtain a preprocessed table image;

[0017] Extract the size and resolution of the preprocessed table image;

[0018] If the size of the preprocessed table image is within a preset size range, and the resolution of the preprocessed table image is within a preset resolution range, then the corresponding preprocessed table image is set as a standard table image;

[0019] If the size of the table image after preprocessing is not within the preset size range, the size of the table image after preprocessing is adjusted;

[0020] If the resolution of the preprocessed table image is not within the preset resolution range, the resolution of the preprocessed table image is adjusted;

[0021] The preprocessed table image is revised by adjusting the size of the preprocessed table image or adjusting the resolution of the preprocessed table image to obtain a standard table image.

[0022] In this solution, the step of adjusting the size of the preprocessed table image specifically includes:

[0023] When the size of the preprocessed table image is smaller than a preset size range, extracting the middle point of the preprocessed table image;

[0024] A middle point of a blank table image based on a preset first size;

[0025] The blank table image of the preset first size and the preprocessed table image are merged at an intermediate point to complete the adjustment of the preprocessed table image;

[0026] When the size of the preprocessed table image is larger than a preset size range, the preprocessed table image is cropped according to the preset first size until a remaining table image smaller than or equal to the preset first size is cropped.

[0027] In this solution, the step of adjusting the resolution of the preprocessed table image specifically includes:

[0028] Extract the outline of the table image;

[0029] Preprocessing the outline of the table image to obtain the outline of the image after preprocessing;

[0030] The preprocessed image contour and the preprocessed table image are fused to obtain a fused image;

[0031] Extracting the resolution of the fused image, if the resolution of the fused image is within a preset resolution range, setting the fused image to the adjusted preset processed table image;

[0032] If the resolution of the fused image is not within the preset resolution range, the corresponding fused image will be deleted, and the shooting parameters will be adjusted to re-shoot the table image of the target document.

[0033] In this solution, the step of determining the label of each pixel in the table according to the overall structural characteristics of the table specifically includes:

[0034] Extract the pixels and corresponding pixel positions in the table;

[0035] Based on a preset pixel point gap, pixel points that are smaller than the preset pixel point gap are connected to obtain the overall structural features of the pixel points;

[0036] Compare and analyze the overall structural features of the table and the overall structural features of the pixels to obtain similarity values;

[0037] If the similarity value is greater than or equal to the preset first similarity threshold, the corresponding pixel point is a cell boundary pixel point; if the similarity value is less than the preset first similarity threshold, the corresponding pixel point is a cell content pixel point;

[0038] The labels of the pixel points include cell boundaries and cell contents.

[0039] In this solution, after obtaining the structured table data, the process further includes:

[0040] Extract the text content in the cell;

[0041] Classify the text content in the cells by rows or columns, and obtain the text content in the cells in the same row or column;

[0042] Compare and analyze the text contents in the same row or column to obtain the relevance of the text contents;

[0043] Determine whether the text content relevance is greater than a preset relevance threshold, if so, the text content in the corresponding row or column cell is accurate; if not, mark the current row or column to obtain a marked row or column;

[0044] Send the marked row or column to the preset management end for prompting.

[0045] The second aspect of the present invention provides a document table content extraction system based on semantic segmentation, comprising: a data preprocessing module, a table area detection module, a table structure recognition module, a table content extraction module and a post-processing and result output module;

[0046] The data preprocessing module is used to preprocess the input table image, including image input, image enhancement and data standardization;

[0047] The table area detection module is used to detect the table area in the table image and perform area positioning by using feature extraction and semantic segmentation technology;

[0048] The table structure recognition is used to slice the table area into rows and columns and fuse features, and label each pixel;

[0049] The table content extraction module is used to identify the content of the table area containing cell content pixels, extract the text content, and organize it into structured data;

[0050] The post-processing and result output module verifies and corrects the structured data and outputs the final result in a preset format.

[0051] In this solution, the data preprocessing module includes: an image input unit, an image enhancement unit and a data standardization unit;

[0052] The image input unit is used for inputting a table image;

[0053] The image enhancement unit is used to perform a preprocessing operation on the input table image to obtain a preprocessed table image;

[0054] The data standardization unit adjusts the size and resolution of the preprocessed table image within a preset size range and a preset resolution range to ensure the consistency of the input image.

[0055] In this solution, the table structure recognition module specifically includes: a cell segmentation unit, a content recognition unit and a data sorting unit;

[0056] The cell segmentation unit is used to divide the table area into a plurality of cell areas containing labels according to the label of each pixel point in the table;

[0057] The content recognition unit is used to perform content recognition on the table area containing cell content pixels to obtain the text content in the cell;

[0058] The data sorting unit is used to sort the recognized text content according to the table structure to obtain structured table data.

[0059] In this solution, the post-processing and result output module specifically includes: a data verification unit and a result output unit;

[0060] The data verification unit is used to verify the structured table data;

[0061] The result output unit outputs the structured table data verified to be accurate according to a preset format.

[0062] The present invention discloses a method and system for extracting document table content based on semantic segmentation. Through a document table content extraction algorithm based on semantic segmentation, the deficiencies of the prior art in terms of adaptability to table formats, complex table processing capabilities, data annotation costs, versatility and degree of automation are solved, and the overall performance and application value of document table content extraction are significantly improved. The present invention not only improves the accuracy and robustness of table content extraction, but also reduces data annotation costs, enhances the versatility of the method, and realizes efficient and automated processing of document table content extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a method for extracting document table content based on semantic segmentation according to the present invention is shown;

[0064] Figure 2 A block diagram of a document table content extraction system based on semantic segmentation of the present invention is shown. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0067] Figure 1 A flow chart of a method for extracting document table content based on semantic segmentation of the present invention is shown.

[0068] like Figure 1 As shown, the present invention discloses a document table content extraction method based on semantic segmentation, comprising:

[0069] S101, obtaining table image information;

[0070] S102, preprocessing the table image to obtain a standard table image;

[0071] S103, extracting the table area and its area position and image features in the standard table image based on a preset feature extraction and semantic segmentation method;

[0072] S104, slicing the table area into rows and columns to obtain rows and columns;

[0073] S105, averaging the image features according to rows and columns to obtain row features and column features;

[0074] S106, fusing the row features and column features to obtain the overall structural features of the table;

[0075] S107, determining the label of each pixel in the table according to the overall structural characteristics of the table;

[0076] S108, based on the label of each pixel in the table, dividing the table area into a plurality of cell areas containing labels;

[0077] S109, extracting the text content in the cell area containing the label, and organizing the text content according to the table structure to obtain structured table data.

[0078] According to an embodiment of the present invention, the preprocessing includes graying, denoising, binarization, etc., to preprocess the table image, improve the accuracy of the corresponding table image in subsequent processing, and extract the image features in each image of the standard table through a preset feature extraction method, such as a pre-trained ResNet convolutional neural network; extract the table area and its area position in the standard table image through a preset semantic segmentation method, such as a fully convolutional network (FCN).

[0079] According to an embodiment of the present invention, the step of preprocessing the table image to obtain a standard table image specifically includes:

[0080] Preprocessing the table image to obtain a preprocessed table image;

[0081] Extract the size and resolution of the preprocessed table image;

[0082] If the size of the preprocessed table image is within a preset size range, and the resolution of the preprocessed table image is within a preset resolution range, then the corresponding preprocessed table image is set as a standard table image;

[0083] If the size of the table image after preprocessing is not within the preset size range, the size of the table image after preprocessing is adjusted;

[0084] If the resolution of the preprocessed table image is not within the preset resolution range, the resolution of the preprocessed table image is adjusted;

[0085] The preprocessed table image is revised by adjusting the size of the preprocessed table image or adjusting the resolution of the preprocessed table image to obtain a standard table image.

[0086] It should be noted that the input table image is preprocessed, including grayscale, denoising, binarization, etc., to obtain the preprocessed table image, and then the preprocessed table image is unified according to the preset size range and resolution range to ensure the consistency of the standard table image.

[0087] According to an embodiment of the present invention, the step of adjusting the size of the preprocessed table image specifically includes:

[0088] When the size of the preprocessed table image is smaller than a preset size range, extracting the middle point of the preprocessed table image;

[0089] A middle point of a blank table image based on a preset first size;

[0090] The blank table image of the preset first size and the preprocessed table image are merged at an intermediate point to complete the adjustment of the preprocessed table image;

[0091] When the size of the preprocessed table image is larger than a preset size range, the preprocessed table image is cropped according to the preset first size until a remaining table image smaller than or equal to the preset first size is cropped.

[0092] It should be noted that the preset first size is within a preset size range. When the size of the table image after preprocessing is larger than the preset size range, the corresponding table image after preprocessing will be cropped according to the preset first size, and cropped into two table sub-images, one of which is of the preset first size and the other is of the remaining size. The table sub-image of the remaining size is set as the remaining table image. If the size of the remaining table image is still larger than the preset first size, the remaining table image will continue to be cropped according to the preset first size until the size of the corresponding remaining table image is less than or equal to the preset first size.

[0093] According to an embodiment of the present invention, the step of adjusting the resolution of the preprocessed table image specifically includes:

[0094] Extract the outline of the table image;

[0095] Preprocessing the outline of the table image to obtain the outline of the image after preprocessing;

[0096] The preprocessed image contour and the preprocessed table image are fused to obtain a fused image;

[0097] Extracting the resolution of the fused image, if the resolution of the fused image is within a preset resolution range, setting the fused image to the adjusted preset processed table image;

[0098] If the resolution of the fused image is not within the preset resolution range, the corresponding fused image will be deleted, and the shooting parameters will be adjusted to re-shoot the table image of the target document.

[0099] It should be noted that the resolution of the preprocessed table image is increased preferentially by fusing images. If the resolution of the preprocessed table image is still not within the preset resolution range, the shooting parameters are adjusted and the table image of the target document is re-shot.

[0100] According to an embodiment of the present invention, the step of determining the label of each pixel in the table according to the overall structural characteristics of the table specifically includes:

[0101] Extract the pixels and corresponding pixel positions in the table;

[0102] Based on a preset pixel point gap, pixel points that are smaller than the preset pixel point gap are connected to obtain the overall structural features of the pixel points;

[0103] Compare and analyze the overall structural features of the table and the overall structural features of the pixels to obtain similarity values;

[0104] If the similarity value is greater than or equal to the preset first similarity threshold, the corresponding pixel point is a cell boundary pixel point; if the similarity value is less than the preset first similarity threshold, the corresponding pixel point is a cell content pixel point;

[0105] The labels of the pixel points include cell boundaries and cell contents.

[0106] It should be noted that, according to the labels of the pixels, the pixels are divided into cell boundary pixels and cell content pixels.

[0107] According to an embodiment of the present invention, after obtaining the structured table data, the method further includes:

[0108] Extract the text content in the cell;

[0109] Classify the text content in the cells by rows or columns, and obtain the text content in the cells in the same row or column;

[0110] Compare and analyze the text contents in the same row or column to obtain the relevance of the text contents;

[0111] Determine whether the text content relevance is greater than a preset relevance threshold, if so, the text content in the corresponding row or column cell is accurate; if not, mark the current row or column to obtain a marked row or column;

[0112] Send the marked row or column to the preset management end for prompting.

[0113] It should be noted that after obtaining the structured table data, the correlation of the text contents in the cells of the same row or column is used to determine whether the text contents in the corresponding rows or columns are related, so as to improve the accuracy of the text contents.

[0114] Figure 2 A block diagram of a document table content extraction system based on semantic segmentation of the present invention is shown.

[0115] like Figure 2 As shown, the second aspect of the present invention provides a document table content extraction system based on semantic segmentation, comprising: a data preprocessing module, a table area detection module, a table structure recognition module, a table content extraction module and a post-processing and result output module;

[0116] The data preprocessing module is used to preprocess the input table image, including image input, image enhancement and data standardization;

[0117] The table area detection module is used to detect the table area in the table image and perform area positioning by using feature extraction and semantic segmentation technology;

[0118] The table structure recognition is used to slice the table area into rows and columns and fuse features, and label each pixel;

[0119] The table content extraction module is used to identify the content of the table area containing cell content pixels, extract the text content, and organize it into structured data;

[0120] The post-processing and result output module verifies and corrects the structured data and outputs the final result in a preset format.

[0121] The data preprocessing module includes: an image input unit, an image enhancement unit and a data standardization unit;

[0122] The image input unit is used for inputting a table image;

[0123] The image enhancement unit is used to perform a preprocessing operation on the input table image to obtain a preprocessed table image;

[0124] The data standardization unit adjusts the size and resolution of the preprocessed table image within a preset size range and a preset resolution range to ensure the consistency of the input image.

[0125] The table structure recognition module specifically includes: a cell segmentation unit, a content recognition unit and a data sorting unit;

[0126] The cell segmentation unit is used to divide the table area into a plurality of cell areas containing labels according to the label of each pixel point in the table;

[0127] The content recognition unit is used to perform content recognition on the table area containing cell content pixels to obtain the text content in the cell;

[0128] The data sorting unit is used to sort the recognized text content according to the table structure to obtain structured table data.

[0129] The post-processing and result output module specifically includes: a data verification unit and a result output unit;

[0130] The data verification unit is used to verify the structured table data;

[0131] The result output unit outputs the structured table data verified to be accurate according to a preset format.

[0132] It should be noted that the executor of the data preprocessing module is the terminal device. For example, the user terminal scans or uploads the table image of the target document, the terminal device receives the image input, and then preprocesses the input table image through the image enhancement unit to improve the accuracy of subsequent processing; the executor of the table area detection module, the table structure recognition module and the table content extraction module is the server, which first determines the table area, and then performs image feature extraction and table area extraction on the table area, and then slices the table area according to rows or columns, and then averages the image features according to rows and columns to determine the row features and column features, and then merges the row features and column features to determine the overall structural features of the table.

[0133] The present invention discloses a method and system for extracting document table content based on semantic segmentation. Through a document table content extraction algorithm based on semantic segmentation, the deficiencies of the prior art in terms of adaptability to table formats, complex table processing capabilities, data annotation costs, versatility and degree of automation are solved, and the overall performance and application value of document table content extraction are significantly improved. The present invention not only improves the accuracy and robustness of table content extraction, but also reduces data annotation costs, enhances the versatility of the method, and realizes efficient and automated processing of document table content extraction.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and other media that can store program codes.

[0138] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A document table content extraction method based on semantic segmentation, characterized in that: include: Get table image information; Preprocess the table image to obtain a standard table image; Based on the preset feature extraction and semantic segmentation methods, the table area and its area position and image features in the standard table image are extracted; Slice the table area into rows and columns to get rows and columns; The image features are averaged according to rows and columns to obtain row features and column features; Merge row features and column features to obtain the overall structural features of the table; According to the overall structural characteristics of the table, determine the label of each pixel in the table; Based on the label of each pixel in the table, the table area is divided into multiple cell areas containing labels; The text content in the cell area containing the label is extracted, and the text content is sorted according to the table structure to obtain structured table data.

2. The method for extracting document table content based on semantic segmentation according to claim 1, characterized in that: The step of preprocessing the table image to obtain a standard table image specifically includes: Preprocessing the table image to obtain a preprocessed table image; Extract the size and resolution of the preprocessed table image; If the size of the preprocessed table image is within a preset size range, and the resolution of the preprocessed table image is within a preset resolution range, then the corresponding preprocessed table image is set as a standard table image; If the size of the table image after preprocessing is not within the preset size range, the size of the table image after preprocessing is adjusted; If the resolution of the preprocessed table image is not within the preset resolution range, the resolution of the preprocessed table image is adjusted; The preprocessed table image is revised by adjusting the size of the preprocessed table image or adjusting the resolution of the preprocessed table image to obtain a standard table image.

3. The method for extracting document table content based on semantic segmentation according to claim 2, characterized in that: The step of adjusting the size of the preprocessed table image specifically includes: When the size of the preprocessed table image is smaller than a preset size range, extracting the middle point of the preprocessed table image; A middle point of a blank table image based on a preset first size; The blank table image of the preset first size and the preprocessed table image are merged at an intermediate point to complete the adjustment of the preprocessed table image; When the size of the preprocessed table image is larger than a preset size range, the preprocessed table image is cropped according to the preset first size until a remaining table image smaller than or equal to the preset first size is cropped.

4. The method for extracting document table content based on semantic segmentation according to claim 2, characterized in that: The step of adjusting the resolution of the preprocessed table image specifically includes: Extract the outline of the table image; Preprocessing the outline of the table image to obtain the outline of the image after preprocessing; The preprocessed image contour and the preprocessed table image are fused to obtain a fused image; Extracting the resolution of the fused image, if the resolution of the fused image is within a preset resolution range, setting the fused image to the adjusted preset processed table image; If the resolution of the fused image is not within the preset resolution range, the corresponding fused image will be deleted, and the shooting parameters will be adjusted to re-shoot the table image of the target document.

5. The method for extracting document table content based on semantic segmentation according to claim 1, characterized in that: The step of determining the label of each pixel in the table according to the overall structural characteristics of the table specifically includes: Extract the pixels and corresponding pixel positions in the table; Based on a preset pixel point gap, pixel points that are smaller than the preset pixel point gap are connected to obtain the overall structural features of the pixel points; Compare and analyze the overall structural features of the table and the overall structural features of the pixels to obtain similarity values; If the similarity value is greater than or equal to the preset first similarity threshold, the corresponding pixel point is a cell boundary pixel point; if the similarity value is less than the preset first similarity threshold, the corresponding pixel point is a cell content pixel point; The labels of the pixel points include cell boundaries and cell contents.

6. The method for extracting document table content based on semantic segmentation according to claim 1, characterized in that: After obtaining the structured table data, the method further includes: Extract the text content in the cell; Classify the text content in the cells by rows or columns, and obtain the text content in the cells in the same row or column; Compare and analyze the text contents in the same row or column to obtain the relevance of the text contents; Determine whether the text content relevance is greater than a preset relevance threshold, if so, the text content in the corresponding row or column cell is accurate; if not, mark the current row or column to obtain a marked row or column; Send the marked row or column to the preset management end for prompting.

7. A document table content extraction system based on semantic segmentation, characterized in that: include: Data preprocessing module, table area detection module, table structure recognition module, table content extraction module and post-processing and result output module; The data preprocessing module is used to preprocess the input table image, including image input, image enhancement and data standardization; The table area detection module is used to detect the table area in the table image and perform area positioning by using feature extraction and semantic segmentation technology; The table structure recognition is used to slice the table area into rows and columns and fuse features, and label each pixel; The table content extraction module is used to identify the content of the table area containing cell content pixels, extract the text content, and organize it into structured data; The post-processing and result output module verifies and corrects the structured data and outputs the final result in a preset format.

8. A document table content extraction system based on semantic segmentation according to claim 7, characterized in that: The data preprocessing module includes: an image input unit, an image enhancement unit and a data standardization unit; The image input unit is used for inputting a table image; The image enhancement unit is used to perform a preprocessing operation on the input table image to obtain a preprocessed table image; The data standardization unit adjusts the size and resolution of the preprocessed table image within a preset size range and a preset resolution range to ensure the consistency of the input image.

9. A document table content extraction system based on semantic segmentation according to claim 7, characterized in that: The table structure recognition module specifically includes: a cell segmentation unit, a content recognition unit and a data sorting unit; The cell segmentation unit is used to divide the table area into a plurality of cell areas containing labels according to the label of each pixel point in the table; The content recognition unit is used to perform content recognition on the table area containing cell content pixels to obtain the text content in the cell; The data sorting unit is used to sort the recognized text content according to the table structure to obtain structured table data.

10. A document table content extraction system based on semantic segmentation according to claim 7, characterized in that: The post-processing and result output module specifically includes: a data verification unit and a result output unit; The data verification unit is used to verify the structured table data; The result output unit outputs the structured table data verified to be accurate according to a preset format.

Images