A method and apparatus for table automatic recognition and structural conversion

Abstract

The application discloses a table automatic recognition and structural conversion method and device, extracts elements and element position information in a document page, wherein the elements include text, lines and images; determines table candidate regions in the document page according to text density, line distribution and image edge strength of the document page; carries out isolation processing on the images according to the overlapping degree of each image and the table candidate regions and the image type; generates a table frame based on line fitting in the table candidate regions, and carries out merged cell detection based on the table frame to determine the logical relationship between the cells; fills the text and the images into corresponding cells according to the logical relationship of the cells, the position of the text and the position of the unisolated images, to generate structured table data, and the method realizes high-precision recognition and logical restoration of a complex table structure, can intelligently distinguish and isolate interference elements, and outputs semantic complete and high-quality structured table data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a table automatic recognition and structured conversion method and device. BACKGROUND

[0002] With the increasing emphasis of global enterprises on operational efficiency and process automation, traditional paper document management methods have been difficult to meet the needs of modern organizations for rapid information acquisition, structured processing, and long-term secure storage. In this context, digital transformation has become an inevitable development trend in various industries, and digital document management systems are gradually replacing traditional paper office models, becoming an important means to improve work efficiency and optimize resource allocation. In this process, efficient recognition and structured processing of document content is a key link, especially for table content containing a large amount of structured information, the extraction and restoration quality directly affects the utilization value of subsequent data.

[0003] In related technologies, typical table extraction and restoration schemes include the following two:

[0004] One scheme is based on PDFBox for text extraction and line detection, combined with contour recognition and deep learning models to achieve integrated table extraction. However, this scheme has the following defects: it relies on the text extraction capability of PDFBox for non-scanned PDFs, and cannot effectively process mixed PDF documents containing scanned images; for complex layout, cross-page tables, or table structures with multiple levels of merged cells, the grid division logic has certain limitations; the output format is single, only supporting HTML, lacking support for Markdown, Excel, and other diversified formats, limiting its flexibility in actual enterprise application scenarios.

[0005] Another scheme is a table recognition technology based on CNN and OCR joint processing, which inputs the PDF document into a pre-trained convolutional neural network after converting it into an image, identifies the table region using the RPN region candidate network, and further extracts the text and its coordinate information using OCR technology, and finally reconstructs the table structure according to the text distribution. This scheme also has several defects: it completely relies on image processing procedures, resulting in the loss of structured text information originally present in the PDF, increasing the OCR recognition burden and possibly introducing recognition errors; the table structure restoration process only divides rows and columns based on text position information, lacking the use of the original table line structure, and is prone to structural disorder when facing merged cells, cross-row and cross-column situations; it does not involve semantic understanding and text correction mechanisms for table content, making it difficult to ensure the accuracy and consistency of the extraction results; the output method is basic and does not provide structured data format export capabilities, affecting the efficiency of subsequent data processing.

[0006] Therefore, there is an urgent need for an efficient, intelligent and highly compatible technical solution to comprehensively extract and structure the table information in various documents. SUMMARY

[0007] The present application provides a table automatic recognition and structured conversion method and device, which can solve the technical problems of limited adaptability to document types and inaccurate table structure analysis in the prior art during the table structured conversion process.

[0008] In a first aspect, the present application provides a table automatic recognition and structured conversion method, which comprises:

[0009] extracting elements and element position information in a document page, wherein the elements include text, lines and images;

[0010] determining table candidate regions in the document page according to the text density, line distribution and image edge strength of the document page;

[0011] isolating images according to the overlapping degree of each image with the table candidate regions and the image type;

[0012] generating a table border based on line fitting in the table candidate regions, and performing merged cell detection based on the table border to determine the logical relationship between cells;

[0013] filling text and images into corresponding cells according to the logical relationship of cells, the position of text and the position of non-isolated images to generate structured table data.

[0014] In combination with the first aspect, in an implementation mode, before determining the table candidate regions in the document page according to the text density, line distribution and image edge strength of the document page, the method further comprises:

[0015] dividing the document page into multiple windows;

[0016] taking the text coverage area or the number of text blocks in each window as the text density of each window;

[0017] taking the number or total length of straight line segments in each window as the line distribution of each window;

[0018] taking the degree of image gray scale change in each window as the image edge strength of each window.

[0019] In an implementation mode, the determination of the table candidate regions in the document page according to the text density, line distribution and image edge strength of the document page comprises:

[0020] adding the product of the text density of each window and the first weighting coefficient, the product of the line distribution and the second weighting coefficient, and the product of the image edge strength and the third weighting coefficient, to obtain a probability that each window is a table region;

[0021] determining a window as a table candidate region if the probability that the window is a table region is greater than a preset probability threshold.

[0022] In an embodiment, the images are isolated according to the degree of overlap of each image with the table candidate region and the type of image, including:

[0023] calculating the degree of overlap of each image with the table candidate region;

[0024] if the degree of overlap is less than or equal to a preset overlap degree threshold, determining the image as an independent image;

[0025] if the degree of overlap is greater than the overlap degree threshold, determining the image as a table-in image;

[0026] determining the category of the table-in image, wherein the category includes a data image and a non-data image;

[0027] isolating the independent image and the non-data image.

[0028] In an embodiment, the table border is generated based on line fitting in the table candidate region, including:

[0029] identifying and classifying the lines in the table candidate region to obtain a horizontal line group and a vertical line group;

[0030] cross-combining the horizontal line group and the vertical line group to generate a table cell border.

[0031] In an embodiment, the merged cell detection is performed based on the table border to determine the logical relationship between cells, including:

[0032] defining a basic cell based on the table cell border, and assigning a unique identifier to each basic cell;

[0033] constructing an adjacency graph with each basic cell as a node, and establishing a connection edge between two corresponding nodes in the adjacency graph if there is no line between the two adjacent basic cells;

[0034] traversing the adjacency graph to identify all connected components, wherein each connected component represents a logical cell;

[0035] For a connected component composed of multiple nodes connected by connection edges, all base cells covered by the connected component are aggregated and identified as a logical merged cell, and its bounding rectangle boundary is recorded;

[0036] For a connected component including only a single node, its corresponding base cell is identified as a logical independent cell.

[0037] In an embodiment, the text and image are filled into the corresponding cell according to the logical relationship of the cell, the position of the text, and the position of the image not isolated by the image, to generate structured table data, including:

[0038] All logical cells, including logical independent cells and logical merged cells, are obtained;

[0039] For each text and image not isolated, all logical cells are traversed to determine whether the position of the text and image falls within the physical boundary of the current logical cell;

[0040] If the logical cell in which the text or image falls is a logical independent cell, the text or image is assigned to the logical independent cell;

[0041] If the logical cell in which the text or image falls is a logical merged cell, the text or image is assigned to the logical merged cell to generate structured table data.

[0042] In an embodiment, the method further includes:

[0043] Determine whether the geometric structure features and text content distribution features between the table at the beginning of the current document page and the table at the end of the previous document page are consistent;

[0044] If consistent, the table at the end of the previous document page is taken as a first segment, and the table at the beginning of the current document page is taken as a subsequent segment;

[0045] Align the column width of the subsequent segment with the column structure of the first segment, and inherit the table header of the first segment to the subsequent segment to complete the table reconstruction.

[0046] In an embodiment, the method further includes:

[0047] Using a large language model, a semantic label vector is generated for the text of each cell;

[0048] For each row of the table, the error between each cell and the semantic label vector mean of all cells in the row is calculated;

[0049] For each column of the table, calculate the error between each cell and the mean of the semantic label vectors of all cells in the same column;

[0050] If all row and column errors are below the preset error threshold, the table is determined to have passed the consistency check; otherwise, the table is determined to have failed the consistency check.

[0051] Secondly, embodiments of this application provide an apparatus for automatic table recognition and structure conversion, the apparatus comprising:

[0052] An extraction module is used to extract elements and element location information from a document page, wherein the elements include text, lines, and images;

[0053] The determination module is used to determine candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page;

[0054] An isolation module is used to isolate images based on the degree of overlap between each image and the candidate region of the table, as well as the image type.

[0055] The detection module is used to generate table borders based on line fitting in the candidate area of ​​the table, and to perform merged cell detection based on the table borders to determine the logical relationship between cells.

[0056] The generation module is used to fill text and images into corresponding cells based on the logical relationship between cells, the position of text, and the position of unisolated images, in order to generate structured tabular data.

[0057] The beneficial effects of the technical solutions provided in this application include:

[0058] By extracting elements and their positions from a document page, including text, lines, and images, candidate table regions are determined based on the text density, line distribution, and image edge strength. Images are isolated based on their overlap with these candidate regions and their type. Table borders are generated by fitting lines within the candidate regions, and cell merging is performed to determine the logical relationships between cells. Text and images are then filled into corresponding cells based on these logical relationships, text positions, and the positions of unisolated images, generating structured table data. This automated conversion from unstructured document images to highly structured table data achieves high-precision recognition of complex table structures, effectively avoiding common issues of missed and false detections when detecting complex pages, thus significantly improving adaptability to diverse real-world document scenarios. Furthermore, this method intelligently distinguishes and isolates decorative images within tables from the data images that need to be retained, greatly reducing interference from non-table elements in the structure recognition process and ensuring the consistency and purity of subsequent data processing. By achieving a deep understanding and accurate reconstruction of the logical structure of tables, and further ensuring the integrity and accuracy of the semantic content of tables, this method ultimately outputs high-quality, reusable structured table data. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating an embodiment of the method for automatic form recognition and structured conversion in this application;

[0060] Figure 2 This is a schematic diagram of the functional modules of an embodiment of the form automatic recognition and structure conversion device of this application. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0063] In a first aspect, embodiments of this application provide a method for automatic table recognition and structured conversion.

[0064] In one embodiment, reference is made to Figure 1 ,Figure 1 This is a flowchart illustrating the first embodiment of the method for automatic form recognition and structured conversion in this application. Figure 1 As shown, the methods for automatic table recognition and structured conversion include:

[0065] Step S101: Extract elements and element location information from the document page, wherein the elements include text, lines, and images.

[0066] In one embodiment, before extracting elements and element location information from the document page, the method further includes: identifying the document type based on the document's extension or header information, wherein the document type includes: PDF document, Microsoft Word document, and PowerPoint slide document.

[0067] Perform format checks on the input file:

[0068]

[0069] It is worth noting that different types of documents have different internal structures and encoding methods. Therefore, to accurately and completely extract the content information from a document, it is necessary to first determine the document type so that the corresponding dedicated parser can be called for processing. The automatic table recognition and structured conversion method provided in this embodiment supports unified processing of documents of various formats (such as PDF, Word, PPT, etc.). By adapting to different types of dedicated parsers, it accurately extracts text, lines, images, and their corresponding position information from the document page, laying the foundation for subsequent table region detection and structural analysis. The text can specifically be text blocks.

[0070] As an example, when the document type is PDF, use libraries such as PyMuPDF or pdfplumber to read the content of a PDF document. These tools not only support extracting text content, but also obtain the precise coordinates of each character, font size, color, and layout information such as lines and images on the page. The extracted content includes:

[0071] Text and its location information on the document page:

[0072]

[0073] in, This represents the i-th page of the document; This represents the set of text information for the i-th page of the document. This represents the x-coordinate (X-axis position) of the top-left corner of the j-th text. This represents the ordinate (Y-axis position) of the top-left corner of the j-th text. This represents the width of j text elements; This represents the content of the j-th text, i.e., the text string.

[0074] Lines on a document page can be represented using the parameterized method of Hough Transform when detecting straight lines:

[0075]

[0076] in, This represents the set of line parameters for the i-th page of the document. This indicates the angle (degree) between the line and the horizontal axis, usually measured in radians or degrees. This represents the distance from the line to the origin (usually the top left corner of the page) (radial distance in polar coordinates).

[0077] Images on the document page:

[0078]

[0079] in, This represents the image on the i-th page of the document; Represents the x-coordinate of the top-left corner of the image; Represents the ordinate of the top-left corner of the image; Indicates the width of the image; Indicates the height of the image.

[0080] When the document type is DOCX, this library uses python-docx to parse Microsoft Word documents. It can read document content paragraph by paragraph and table by table, extracting formatting information such as text styles, fonts, bold, italics, paragraph indentation, and bullet points. It can also identify and extract images, charts, headers, and footers from the document.

[0081] When the document type is PPTX, python-pptx is used to parse the PowerPoint slide document. Each slide is treated as an independent page unit, and the parser extracts elements such as text boxes, shapes, charts, and images from the slide, and records their position, size, and hierarchy.

[0082] Step S102: Determine the candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page.

[0083] In one embodiment, before determining the candidate table region in the document page based on the text density, line distribution, and image edge intensity of the document page, the method further includes: dividing the document page into multiple windows; using the text coverage area or number of text blocks in each window as the text density of each window; using the number or total length of line segments in each window as the line distribution of each window; and using the degree of drastic change in image grayscale in each window as the image edge intensity of each window.

[0084] As an example, in order to identify potential table areas in a document, the system takes into account multiple visual features, including information such as text density, line distribution, and image edge intensity.

[0085] Text density is one of the core visual features in table recognition systems. It is used to quantify the information density of a specific area on a document page, providing a crucial basis for the accurate location of table areas. Since table areas contain structured text content, they typically have higher text density. Therefore, text density can effectively distinguish table areas from ordinary text areas during the table candidate area recognition process.

[0086] Text density represents the number of text blocks per unit area or the sum of the areas covered by text blocks within a local region (e.g., a 50×50 pixel window) centered at coordinates (x, y) on a document page. It quantifies the information density of that region. The formula for calculating text density is:

[0087]

[0088] in, Indicates the text density of the window; Represents the area of ​​the j-th text block ( × If only the number of text blocks is counted, then =1; W×H represents the window area, used for normalization calculations; This represents the sum of the areas (or the sum of the numbers) of all text blocks within the window.

[0089] Line distribution is a core visual feature used to quantify the structural regularity of local areas in a document, with a particular focus on the density of horizontal and vertical lines. In table areas, horizontal and vertical lines constitute the table borders and internal cell structure. Therefore, line distribution can effectively reflect the probability of table borders and the regularity of the area structure, providing a key basis for the accurate positioning of table areas.

[0090] Line distribution represents the number or total length of straight line segments (especially horizontal and vertical lines) detected per unit area within a local region centered at coordinates (x, y) and with a window size of W × H. It reflects the structural regularity of the region and the probability of table borders being present. The formula for calculating line distribution is:

[0091]

[0092] in, Indicates the line distribution of the window; Let represent the length of the k-th line segment. If only the number of line segments is counted (without considering the length), then... =1; W×H is the window area, used for normalization calculation; This represents the sum of the lengths (or the sum of the numbers) of all horizontal / vertical line segments within the window. The calculation formula is: .

[0093] Image edge intensity is a core visual feature used in table recognition systems to quantify the clarity of structural boundaries in local document regions, with a particular focus on structural boundaries such as table borders and cell outlines. In table regions, edge response intensity is significantly higher than in ordinary text regions; therefore, this feature can effectively distinguish between table and non-table content, providing crucial geometric basis for table localization.

[0094] Image edge intensity is a measure of the degree of drastic change in image grayscale within a local region centered at coordinates (x, y) and with a window size of W × H. It is also known as edge response intensity and is used to characterize whether there are obvious contours, boundaries, or structural abrupt changes in the region.

[0095]

[0096] in, Indicates the image edge intensity of the window; This represents the gradient magnitude of pixel (i, j) (e.g., calculated using the Sobel operator), and the summation range is all pixels within the current window.

[0097] Furthermore, in one embodiment, determining the candidate table region in the document page based on the text density, line distribution, and image edge intensity of the document page includes: adding the product of the text density of each window with a first weighting coefficient, the product of the line distribution with a second weighting coefficient, and the product of the image edge intensity with a third weighting coefficient to obtain the probability that each window is a table region; and determining the window whose probability of being a table region is greater than a preset probability threshold as a candidate table region.

[0098] As an example, this embodiment quantifies the likelihood that different window regions are tables by constructing a table region confidence map.

[0099]

[0100] in, This indicates the probability that the window is a table area; Indicates the text density of the window; Indicates the number of lines in the window; Indicates the image edge intensity of the window; , , These represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, respectively, which can be optimized through training. The higher the value of [0,1], the greater the probability that the window position belongs to the table area.

[0101] The probability of selecting a table region is higher than the probability threshold. The window area is used as the candidate area for the table:

[0102]

[0103] in, This is the set of candidate regions for the table.

[0104] Step S103: Isolate the images according to the degree of overlap between each image and the candidate region of the table and the image type.

[0105] In one embodiment, the degree of overlap between each image and the candidate region of the table is calculated; if the degree of overlap is less than or equal to a preset overlap threshold, the image is determined to be an independent image; if the degree of overlap is greater than the overlap threshold, the image is determined to be an image within the table; the category of the image within the table is determined, wherein the category includes data images and non-data images; the independent images and the non-data images are isolated.

[0106] As an example, YOLOv8 can be used for image detection on a document page. YOLOv8 is a high-efficiency object detection algorithm that can quickly and accurately identify various objects in an image and return the bounding box coordinates and category label for each object. In document image processing scenarios, images can be trained as targets of a specific category or pre-trained models can be directly applied to identify image elements in the document page.

[0107]

[0108] in, This represents the set of images on the i-th page of the document. This represents the k-th image, and each image... This includes the bounding box coordinates (x, y, w, h), representing the coordinates of the top-left corner and the width and height, respectively. During operation, the entire document page is first scanned, and then the YOLOv8 model is run to detect all possible image regions. Based on the model output, the coordinate information and dimensions of a series of images in the document can be obtained.

[0109] After determining the locations of all images in the document, it is then determined whether these images are within the candidate table area. The purpose of this determination is that if an image happens to be inside the table, it requires special handling to avoid being mistaken for part of the table; conversely, if an image is outside the table, it can be treated as an independent element and isolated.

[0110] Prior to this, the candidate areas for tables on the document pages had already been determined. Each candidate region in the table is defined by a set of bounding boxes, in the form of:

[0111]

[0112] in, This represents the candidate region of the m-th table. These represent the x-coordinate (X-axis position) of the top-left corner of the candidate area, the y-coordinate (Y-axis position) of the candidate area, the width of the candidate area, and the height of the candidate area, respectively.

[0113] Next, for each image Calculate it with each candidate region in the table. The degree of overlap between two rectangles can be measured by calculating the proportion of their overlapping area to their total area.

[0114]

[0115] in, Representing an image With table candidate area The degree of overlap between the images is a value between 0 and 1. 0 indicates no overlap (the image is completely outside the candidate region of the table); 1 indicates complete overlap (the image is completely within the candidate region of the table). The intersection area refers to the area of ​​overlap between the images. and table candidate area The area of ​​the overlapping region. The area of ​​the union refers to the area of ​​the image. and table candidate area The total area covered (i.e., the area after merging the two rectangles).

[0116] In this embodiment, an overlap threshold is set. If the overlap between an image and a candidate region in a table exceeds the overlap threshold... If the image is within the table, it is considered to be an independent image; otherwise, it is considered a separate image. For example: .

[0117] For images located within a table, their impact on the table structure needs further analysis. If they are decorative icons or watermarks that do not affect the table data, they can be ignored; however, if they are part of the table (such as data images), they need to be included in the table parsing process to ensure data integrity.

[0118] In this embodiment, a pre-trained image classification model (such as ResNet, ViT, or CLIP) is used to identify the image category within the table. If the image is determined to be decorative / non-data image (such as watermark or icon), it is isolated during the table reconstruction stage (such as ignored or marked separately) to avoid interfering with the logical structure of the table.

[0119] Step S104: Generate table borders based on line fitting in the candidate area of ​​the table, and perform merged cell detection based on the table borders to determine the logical relationship between cells.

[0120] In one embodiment, the step of generating table borders based on line fitting in the candidate area of ​​the table includes: identifying and classifying the lines in the candidate area of ​​the table to obtain horizontal line groups and vertical line groups; and combining the horizontal line groups and vertical line groups to generate table cell borders.

[0121] As an example, the Hough transform can be used to detect straight lines, and then combined with a clustering algorithm (such as DBSCAN) to group horizontal and vertical lines:

[0122]

[0123] in, The input is a set of lines, and the output is a clustered group of horizontal lines. , The input is a set of lines, and the output is a clustered group of vertical lines. .

[0124] Next, combine the horizontal and vertical line groups to create the table cell borders:

[0125]

[0126] in, Set the border for a table cell.

[0127] In one embodiment, the step of detecting merged cells based on the table cell borders to determine the logical relationships between cells includes: defining basic cells based on the table borders and assigning a unique identifier to each basic cell; constructing an adjacency graph with each basic cell as a node, and establishing a connecting edge between the corresponding two nodes in the adjacency graph if there is no line between two adjacent basic cells; traversing the adjacency graph to identify all connected components, where each connected component represents a logical cell; for a connected component consisting of multiple nodes connected by connecting edges, aggregating all basic cells covered by the connected component into a logical merged cell and recording its circumscribed rectangle boundary; for a connected component containing only a single node, identifying its corresponding basic cell as a logically independent cell.

[0128] As an example, define the set of all base cells based on the table cell borders as follows:

[0129]

[0130] in, This represents the set of basic cells on the i-th page of the document; This represents the l-th basic cell.

[0131] Each basic cell consists of four vertex coordinates:

[0132]

[0133] For cases involving merged cells, the boundaries of adjacent base cells are compared to determine if a cell spans multiple rows or columns. A graph-based algorithm can be used to aggregate physically adjacent but semantically connected base cells into a single logically merged cell. Specifically, the following steps can be taken:

[0134] First, construct the adjacency graph, connecting each basic cell c. i It is considered as a node in the graph.

[0135] For each pair of adjacent basic cells (c i , c j Check if there are "real dividing lines" between them: if they exist, do not add borders (consider them as independent cells); if they do not exist, add borders (consider them as belonging to the same logically merged cell).

[0136] An adjacency graph can be traversed using either Depth-First Search (DFS) or Breadth-First Search (BFS), starting from any unvisited node and iterating through all reachable nodes. Each traversal yields a connected component, and the following checks are performed:

[0137]

[0138] For each connected component, determine whether it spans multiple rows or columns, and for each connected component, record all the underlying cell indices it contains.

[0139] Furthermore, mark logically merged cells. For each connected component component = [c1, c2, ..., c k Calculate its bounding box:

[0140] x_min = min(cx for c in component)

[0141] y_min = min(cy for c in component)

[0142] x_max = max(cx + cw for c in component)

[0143] y_max = max(cy + ch for c in component)

[0144] Use this outer rectangle as the boundary of the "logically merged cells". Original cell c i Mark it as "merged" and record the ID of the merged cell to which it belongs.

[0145] In the final output table structure, each "cell" can be either a logically independent cell (not merged, consisting of only a single base cell) or a logically merged cell (composed of multiple base cells).

[0146] In this context, adjacent basic cells refer to basic cells that share a boundary in the table; connected components are extracted from the adjacency graph through depth-first search (DFS) or breadth-first search (BFS); spanning multiple rows or columns means that the connected component covers more than 1 row or more than 1 column.

[0147] By following the steps above, merged cells in a table can be effectively identified and marked, providing support for subsequent table structuring processing.

[0148] Step S105: Based on the logical relationship of the cells, the position of the text, and the position of the unisolated image, the text and image are filled into the corresponding cells to generate structured tabular data.

[0149] In one embodiment, all logical cells are obtained, including logically independent cells and logically merged cells; for each text and non-isolated image, all logical cells are traversed to determine whether the position of the text and image falls within the physical boundary of the current logical cell; if the logical cell into which the text or image falls is a logically independent cell, then the text or image is assigned to this logically independent cell; if the logical cell into which the text or image falls is a logically merged cell, then the text or image is assigned to this logically merged cell to generate structured tabular data.

[0150] As an example, considering the existence of logically merged cells, when assigning text and unisolated images to logical cells, it is necessary to consider not only physical location but also logical relationships. For instance, if a logical cell is marked as a logically merged cell, it is necessary to check all the base cells it covers and appropriately assign text to this logically merged cell. In the presence of logically merged cells, text should not be simply assigned to the smallest base cell based on physical location, but rather assigned to its corresponding "logically merged cell" (i.e., the outer frame of the logically merged cell) to ensure correct content attribution and complete structural restoration.

[0151] For each text and unisolated image, iterate through all logical cells and determine whether the position of the text and image falls within the physical boundary of the current logical cell:

[0152]

[0153] in, Representing text With cells The attribution function returns 1 for "belongs" and 0 for "does not belong". This represents the j-th text block, containing its content. and location ( , , ), text block, , text block The coordinates of the top left corner (or the coordinates of the center point, depending on the implementation).

[0154] It is worth noting that this allocation logic ensures that, in the presence of merged cells, text blocks are correctly allocated to their respective logical cells, thereby guaranteeing the integrity and accuracy of the table structure.

[0155] Furthermore, in one embodiment, the method further includes: determining whether the geometric structure features and text content distribution features of the table at the beginning of the current document page and the table at the end of the previous document page are consistent; if consistent, then the table at the end of the previous document page is taken as the first segment, and the table at the beginning of the current document page is taken as the subsequent segment; based on the column structure of the first segment, the column width of the subsequent segment is aligned, and the table header of the first segment is inherited to the subsequent segment to complete the table reconstruction.

[0156] As an example, when dealing with tables that span multiple pages, the first step is to identify the table's dividing points. When a table crosses the page boundaries, its start and end points must be accurately identified. This process can be achieved by analyzing the table's features on each page, including line patterns, text distribution, and so on. In particular, it's necessary to look for similar table features that abruptly stop at the end of a page and continue at the beginning of the next page. These "table features" refer to multi-dimensional, quantifiable attributes that can be used to determine the continuity of a table.

[0157] The geometric structure features include: column width distribution, row height distribution, and border pattern. The text content distribution features include: header text sequence, column data types, and keywords. By comparing the similarity of these features between the end of the current page and the beginning of the next page, if the similarity exceeds a threshold, it is determined to be a cross-page continuation of the same table.

[0158] Next, after determining the table's split points, these scattered sections need to be reassembled into a complete table. This typically involves aligning the column widths and row heights of the different sections and fixing any discontinuities caused by pagination. A template-matching method can be used to predict and fill in the missing sections based on the known table structure.

[0159] Specifically, a table reorganization function can be defined. This function is used to reconstruct a table fragment spanning multiple pages into a complete table. First, it aligns the column widths of subsequent fragments based on the column structure of the first fragment, ensuring consistency in column width across all fragments for easier merging. Second, it inherits the header from the first fragment to the fragment without a header, ensuring the header information of the entire table is complete and consistent. Third, it detects and repairs merged cells that break across pages, ensuring the logical relationships between merged cells are restored. Finally, it performs data continuity checks to ensure the integrity and consistency of table data across pages. The reconstructed table is as follows:

[0160]

[0161] in, For k segments of the same logical table on consecutive pages.

[0162] Internal table logic validation includes checking whether the data types of each row / column are consistent after the initial table reconstruction is completed; confirming whether the content in merged cells meets expectations; and verifying the continuity and integrity of the reconstructed table across pages, thereby ensuring the logical correctness of the table.

[0163] As a preferred implementation, the method further includes: using a large language model to analyze the generated table and generate a semantic label vector for the text of each cell; for each row of the table, calculating the error between each cell and the mean of the semantic label vectors of all cells in the same row; for each column of the table, calculating the error between each cell and the mean of the semantic label vectors of all cells in the same column; if all row errors and column errors are lower than a preset error threshold, the table is determined to have passed the consistency check; otherwise, the table is determined to have failed the consistency check.

[0164] As an example, to ensure that the information extracted from the table is not only structurally accurate but also semantically consistent, this embodiment introduces a text semantic understanding and logical verification module. This module mainly consists of two parts: text semantic label vector generation and row / column consistency verification. These two steps effectively improve data accuracy and uncover potential problems.

[0165] Advanced large-scale language models (LLMs), such as BERT or ChatGLM, can be used to perform in-depth semantic analysis and annotation of the text in each logical cell. This process goes beyond simple text recognition; it attempts to understand the actual meaning of the text within each cell to better support subsequent data processing and applications.

[0166] Before inputting the table into the large language model, the text within the cells needs to be preprocessed, including removing extra whitespace characters and standardizing punctuation marks, to ensure that the text input into the LLM is clean and consistent.

[0167]

[0168] in It is a logical cell semantic label vectors, This represents the text content within the cell.

[0169] After semantic annotation and semantic label vector generation, a consistency check is performed on each row and column of the table to ensure that the content of all cells is logically related and conflict-free. This is crucial for detecting misidentification due to OCR errors or other reasons. The semantic labels of each row / column are checked for consistency to avoid structural errors caused by misidentification. Taking a row as an example, for each row r, the semantic consistency error Erow(r) for that row is calculated:

[0170]

[0171] in, This represents the semantic consistency error of the r-th row; Let S(crl) represent the average semantic vector of row r; S(crl) represent the semantic label vector of cell l in row r; n represents the number of logical cells in the row. Then, calculate the error between the semantic vector of each logical cell and the average semantic vector of the row. The same method as above can be used for the semantic consistency error of each column.

[0172] Set a reasonable error threshold. If the error of a row or column exceeds this threshold, the table is considered to have a potential problem and needs to be manually reviewed. If the error does not exceed the threshold, the constructed table is considered to be qualified.

[0173] As a preferred implementation, after completing table structure parsing and semantic understanding, the identified table content can be transformed into a standard data structure format according to the table's usage requirements, facilitating subsequent system processing, display, or analysis. This process needs to consider complex situations such as cell hierarchy, logical relationships between merged cells, and row and column alignment to ensure that the output data is not only structurally clear but also semantically complete.

[0174] Data structure mapping to table expansion: Maps table structures to standard data structures, such as two-dimensional arrays or JSON format.

[0175]

[0176] Where n represents the number of rows in the table; m represents the maximum number of columns in each row (which may vary depending on merged cells). This represents the content of the cell in row i and column j. If this cell is merged, it is marked as merged or left blank. For merged cells, their coverage area is recorded as follows:

[0177]

[0178] For example, if a cell spans two rows and three columns, it is represented as

[0179] Standard data structure mapping methods include:

[0180] (1) Two-dimensional array (List of Lists): Suitable for simple table structures, especially suitable for internal processing of programs or as a basic data structure passed to downstream modules.

[0181] Example: A two-dimensional array structure table_array = [["Name", "Age", "Gender"], ["Zhang San", "28", "Male"], ["Li Si", "32", "Female"]]

[0182] It is worth noting that for merged cells, in a two-dimensional array, the covered sub-cells are usually filled with placeholders (such as None or "merged") to maintain the integrity of the array structure.

[0183] (2) JSON format (JavaScript Object Notation): Supports richer semantic expression, facilitating cross-platform transmission and integration. It can include information such as table headers, data rows, cell attributes (such as whether to merge), and semantic tags.

[0184] { "header": ["Name", "Age", "Gender"], "rows": [ {"Name": "Zhang San", "Age": "28", "Gender": "Male"}, {"Name": "Li Si", "Age": "32", "Gender": "Female"} ],"merged_cells": { "start_row": 0, "start_col": 0, "end_row": 1,"end_col": 1,"content": "Personal Information"} ]}

[0185] The JSON structure can also be extended with fields such as semantic tags, OCR confidence scores, and raw text coordinates, making it suitable for advanced data analysis and visualization needs.

[0186] (3) Pandas DataFrame: A data structure designed specifically for data analysis and widely used in data processing tasks in the Python ecosystem.

[0187] import pandas as pddf = pd.DataFrame(table_array[1:], columns=table_array[0])

[0188] It supports flexible row and column operations, missing value handling, type conversion, and other functions, making it very suitable for scenarios such as machine learning model training and BI report generation.

[0189] (4) XML format (eXtensible Markup Language): suitable for scenarios that require the preservation of strict format and hierarchical structure, and is often used for the exchange of standard documents in industries such as government and finance.

[0190] <row>

[0191] <cell>Name< / cell>

[0192] <cell>Age< / cell>

[0193] <cell>Gender< / cell>

[0194] < / row>

[0195] <row>

[0196] <cell>Zhang San< / cell>

[0197] <cell> 28< / cell>

[0198] <cell>Male< / cell>

[0199] < / row>

[0200]

[0201] Supports multiple output formats for easy integration with downstream systems:

[0202] in, .

[0203] This application proposes an automatic table recognition and structured conversion method, which constructs a fully automated processing workflow covering document type recognition, non-table content isolation, table structure parsing, semantic understanding and multi-format output, effectively solving several key challenges currently faced in the field of document processing.

[0204] Specifically, the document type is automatically identified by file extension or header information, and the corresponding dedicated parser (such as PyMuPDF / pdfplumber, python-docx, and python-pptx) is invoked to perform preliminary processing of the document. This process transforms the document content into an intermediate representation, while extracting basic information such as text and its coordinates, page lines, and images. Furthermore, a confidence map is constructed based on text density, line distribution, and image edge intensity to intelligently filter out candidate areas for tables.

[0205] Next, advanced image detection tools (such as YOLOv8 or OpenCV) are used to identify and isolate non-table content, such as images and protected areas, ensuring that these elements do not interfere with the subsequent table recognition process. Then, techniques such as Hough transform and DBSCAN clustering are used to accurately fit table borders, divide cells, and detect merged cells, thereby achieving accurate parsing and reconstruction of the table structure. Furthermore, for tables spanning multiple pages, the system employs methods such as template matching to ensure the integrity and continuity of the table data.

[0206] Building upon this foundation, large-scale language models (such as BERT and ChatGLM) are applied to perform semantic understanding and logical verification of the table content. This corrects misidentifications caused by OCR errors or other reasons, significantly enhancing data accuracy and consistency. Furthermore, it supports multiple structured output formats, including CSV, JSON, Excel, HTML, and Markdown, to meet the data processing and analysis needs of different application scenarios. This flexible output capability allows the system to be seamlessly integrated into various workflows, improving data usability and integrability.

[0207] The entire process is highly automated and robust, effectively handling various complex table structures, including merged cells and multi-page tables. This method is applicable to multiple document formats, such as PDF, Word, and PPT, and performs exceptionally well in various scenarios, including financial statements, government documents, and research papers, significantly improving the efficiency and accuracy of table information extraction.

[0208] Secondly, embodiments of this application also provide an apparatus for automatic table recognition and structured conversion.

[0209] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of an embodiment of the form automatic recognition and structured conversion device of this application. Figure 2 As shown, the automatic table recognition and structure conversion device includes:

[0210] An extraction module is used to extract elements and element location information from a document page, wherein the elements include text, lines, and images;

[0211] The determination module is used to determine candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page;

[0212] An isolation module is used to isolate images based on the degree of overlap between each image and the candidate region of the table, as well as the image type.

[0213] The detection module is used to generate table borders based on line fitting in the candidate area of ​​the table, and to perform merged cell detection based on the table borders to determine the logical relationship between cells.

[0214] The generation module is used to fill text and images into corresponding cells based on the logical relationship between cells, the position of text, and the position of unisolated images, in order to generate structured tabular data.

[0215] Furthermore, in one embodiment, the determining module is further configured to:

[0216] Divide the document page into multiple windows;

[0217] The text coverage area or number of text blocks in each window is used as the text density of each window;

[0218] The number or total length of straight line segments in each window is used as the line distribution of each window;

[0219] The degree of drastic change in grayscale in each window is used as the edge intensity of the image in each window.

[0220] Furthermore, in one embodiment, the determining module is further configured to:

[0221] The probability that each window is a table region is obtained by adding the product of the text density of each window with the first weighting coefficient, the product of the line distribution with the second weighting coefficient, and the product of the image edge intensity with the third weighting coefficient.

[0222] Windows whose probability of being a table region is greater than a preset probability threshold are identified as table candidate regions.

[0223] Furthermore, in one embodiment, the isolation module is also used for:

[0224] Calculate the degree of overlap between each image and the candidate regions in the table;

[0225] If the degree of overlap is less than or equal to the preset overlap threshold, the image is determined to be an independent image.

[0226] If the degree of overlap is greater than the overlap threshold, the image is determined to be an image within the table;

[0227] Determine the category of the images within the table, wherein the category includes data images and non-data images;

[0228] The independent images and the non-data images are isolated.

[0229] Furthermore, in one embodiment, the detection module is also used for:

[0230] Identify and classify the lines within the candidate area of ​​the table to obtain horizontal line groups and vertical line groups;

[0231] The horizontal and vertical line groups are combined to create table cell borders.

[0232] Furthermore, in one embodiment, the detection module is also used for:

[0233] Define basic cells based on the borders of the table cells, and assign a unique identifier to each basic cell;

[0234] Construct an adjacency graph by treating each basic cell as a node. If there is no line between two adjacent basic cells, then establish a connecting edge between the corresponding two nodes in the adjacency graph.

[0235] Traverse the adjacency graph to identify all connected components, where each connected component represents a logical cell;

[0236] For a connected component consisting of multiple nodes connected to each other by connecting edges, all the basic cells covered by the connected component are aggregated and identified as a logically merged cell, and its outer rectangle boundary is recorded.

[0237] For a connected component that includes only a single node, its corresponding base cell is identified as a logically independent cell.

[0238] Furthermore, in one embodiment, the generation module is further configured to:

[0239] Retrieve all logical cells, including logically independent cells and logically merged cells;

[0240] For each text and unisolated image, traverse all logical cells and determine whether the position of the text and image falls within the physical boundary of the current logical cell;

[0241] If the text or image falls into a logically independent cell, then the text or image is assigned to that logically independent cell.

[0242] If the text or image falls into a logically merged cell, then the text or image is assigned to that logically merged cell to generate structured tabular data.

[0243] Furthermore, in one embodiment, the device further includes a reconfiguration module, which is used to:

[0244] Determine whether the geometric structure features and text content distribution features of the table at the beginning of the current document page are consistent with those of the table at the end of the previous document page;

[0245] If they match, the table at the end of the previous document page will be taken as the first segment, and the table at the beginning of the current document page will be taken as the subsequent segment.

[0246] Based on the column structure of the first segment, align the column widths of the subsequent segments, and inherit the table header of the first segment to the subsequent segments to complete the table reconstruction.

[0247] Furthermore, in one embodiment, the device further includes a verification module, which is used to:

[0248] The generated table is analyzed using a large language model, and semantic label vectors are generated for the text of each cell.

[0249] For each row of the table, calculate the error between each cell and the mean of the semantic label vectors of all cells in the same row;

[0250] For each column of the table, calculate the error between each cell and the mean of the semantic label vectors of all cells in the same column;

[0251] If all row and column errors are below the preset error threshold, the table is determined to have passed the consistency check; otherwise, the table is determined to have failed the consistency check.

[0252] The functions of each module in the above-mentioned automatic table recognition and structure conversion device correspond to the steps in the above-mentioned automatic table recognition and structure conversion method embodiment, and their functions and implementation processes will not be described in detail here.

[0253] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0254] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0255] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0256] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0257] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0258] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0259] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for automatic table recognition and structured conversion, characterized in that, The automatic table recognition and structure conversion method includes: Extract elements and element location information from a document page, wherein the elements include text, lines, and images; Based on the text density, line distribution, and image edge intensity of the document page, candidate table regions in the document page are determined; Based on the degree of overlap between each image and the candidate region of the table, as well as the image type, the images are isolated. Table borders are generated by fitting lines in the candidate areas of the table, and merged cells are detected based on the table borders to determine the logical relationship between cells. Based on the logical relationships between cells, the position of text, and the position of unisolated images, text and images are filled into the corresponding cells to generate structured tabular data; The step of determining candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page includes: Divide the document page into multiple windows; The probability that each window is a table region is obtained by adding the product of the text density of each window with the first weighting coefficient, the product of the line distribution with the second weighting coefficient, and the product of the image edge intensity with the third weighting coefficient. Window whose probability of being a table region is greater than a preset probability threshold is identified as a table candidate region; Specifically, based on the logical relationships between cells, the position of text, and the position of unisolated images, text and images are filled into the corresponding cells to generate structured tabular data, including: Retrieve all logical cells, including logically independent cells and logically merged cells; If the text or image falls into a logically independent cell, then the text or image is assigned to that logically independent cell. If the text or image falls into a logically merged cell, then the text or image is assigned to this logically merged cell to generate structured tabular data.

2. The method for automatic table recognition and structured conversion as described in claim 1, characterized in that, Before determining the candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page, the method further includes: The text coverage area or number of text blocks in each window is used as the text density of each window; The number or total length of straight line segments in each window is used as the line distribution of each window; The degree of drastic change in grayscale in each window is used as the edge intensity of the image in each window.

3. The method for automatic table recognition and structured conversion as described in claim 1, characterized in that, Based on the degree of overlap between each image and the candidate region of the table, and the image type, the images are isolated, including: Calculate the degree of overlap between each image and the candidate regions in the table; If the degree of overlap is less than or equal to the preset overlap threshold, the image is determined to be an independent image. If the degree of overlap is greater than the overlap threshold, the image is determined to be an image within the table; Determine the category of the images within the table, wherein the category includes data images and non-data images; The independent images and the non-data images are isolated.

4. The method for automatic table recognition and structured conversion as described in claim 1, characterized in that, The process of generating table borders based on line fitting within the candidate regions of the table includes: Identify and classify the lines within the candidate area of ​​the table to obtain horizontal line groups and vertical line groups; The horizontal and vertical line groups are combined to create table cell borders.

5. The method for automatic table recognition and structured conversion as described in claim 4, characterized in that, The step of detecting merged cells based on the table borders to determine the logical relationship between cells includes: Define basic cells based on the borders of the table cells, and assign a unique identifier to each basic cell; Construct an adjacency graph by treating each basic cell as a node. If there is no line between two adjacent basic cells, then establish a connecting edge between the corresponding two nodes in the adjacency graph. Traverse the adjacency graph to identify all connected components, where each connected component represents a logical cell; For a connected component consisting of multiple nodes connected to each other by connecting edges, all the basic cells covered by the connected component are aggregated and identified as a logically merged cell, and its outer rectangle boundary is recorded. For a connected component that includes only a single node, its corresponding base cell is identified as a logically independent cell.

6. The method for automatic table recognition and structured conversion as described in claim 5, characterized in that, The process of filling text and images into corresponding cells based on the logical relationships between cells, the position of text, and the position of unisolated images to generate structured tabular data includes: For each text and unisolated image, iterate through all logical cells and determine whether the position of the text and image falls within the physical boundary of the current logical cell.

7. The method for automatic table recognition and structured conversion as described in claim 1, characterized in that, The method also includes: Determine whether the geometric structure features and text content distribution features of the table at the beginning of the current document page are consistent with those of the table at the end of the previous document page; If they match, the table at the end of the previous document page will be taken as the first segment, and the table at the beginning of the current document page will be taken as the subsequent segment. Based on the column structure of the first segment, align the column widths of the subsequent segments, and inherit the table header of the first segment to the subsequent segments to complete the table reconstruction.

8. The method for automatic table recognition and structured conversion as described in claim 1, characterized in that, The method also includes: The generated table is analyzed using a large language model, and semantic label vectors are generated for the text of each cell. For each row of the table, calculate the error between each cell and the mean of the semantic label vectors of all cells in the same row; For each column of the table, calculate the error between each cell and the mean of the semantic label vectors of all cells in the same column; If all row and column errors are below the preset error threshold, the table is determined to have passed the consistency check; otherwise, the table is determined to have failed the consistency check.

9. A device for automatic table recognition and structured conversion, characterized in that, The automatic form recognition and structure conversion device includes: An extraction module is used to extract elements and element location information from a document page, wherein the elements include text, lines, and images; The determination module is used to determine candidate table regions in the document page based on the text density, line distribution, and image edge intensity of the document page; An isolation module is used to isolate images based on the degree of overlap between each image and the candidate region of the table, as well as the image type. The detection module is used to generate table borders based on line fitting in the candidate area of ​​the table, and to perform merged cell detection based on the table borders to determine the logical relationship between cells. The generation module is used to fill text and images into corresponding cells based on the logical relationship between cells, the position of text, and the position of unisolated images, in order to generate structured tabular data. The determining module is further configured to: Divide the document page into multiple windows; The probability that each window is a table region is obtained by adding the product of the text density of each window with the first weighting coefficient, the product of the line distribution with the second weighting coefficient, and the product of the image edge intensity with the third weighting coefficient. Window whose probability of being a table region is greater than a preset probability threshold is identified as a table candidate region; The generation module is further configured to: Retrieve all logical cells, including logically independent cells and logically merged cells; If the text or image falls into a logically independent cell, then the text or image is assigned to that logically independent cell. If the text or image falls into a logically merged cell, then the text or image is assigned to this logically merged cell to generate structured tabular data.

Images