A heterogeneous table extraction method based on a hybrid visual strategy

Abstract

This invention relates to a heterogeneous table extraction method based on a hybrid visual strategy, belonging to the fields of artificial intelligence, big data, computer vision, and natural language processing. By improving existing table parsing techniques and combining deep learning and natural language processing technologies, this invention enhances the accuracy and efficiency of table data recognition and extraction, laying the foundation for building a high-quality military knowledge base. This invention addresses the current technical bottlenecks in table data recognition and provides data support for future military intelligence analysis and automated decision-making systems.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, big data, computer vision, and natural language processing, and specifically relates to a heterogeneous table extraction method based on a hybrid vision strategy. Background Technology

[0002] In modern military research and intelligence analysis, data is a core element supporting decision-making and strategy formulation. A vast amount of military data exists in unstructured forms, such as historical documents, intelligence reports, and training records. This data is often stored in paper form and digitized through scanning, resulting in massive amounts of image-formatted data. However, much crucial information, including structured tabular data, is hidden within these unstructured image files. How to efficiently extract and utilize this information has become one of the key technologies for building knowledge bases and intelligence analysis.

[0003] Tabular data possesses highly structured information, typically including data relationships, numerical statistics, and classification information, thus holding irreplaceable value in military research. However, due to the prevalence of scanned paper documents in real-world data, tables exhibit diverse formats, complex structures, and numerous text recognition challenges, traditional table recognition methods face significant difficulties in processing this type of data. Accurately extracting tables from these image files and transforming them into usable structured data is not only crucial for improving data utilization efficiency but also fundamental to building an intelligent military knowledge base. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The technical problem this invention aims to solve is to propose an automatic identification and extraction method for unstructured tabular data in the military field. Considering the complexity of tabular data in practical applications, this task faces two main challenges: First, the diversity of table regions and text content is significant, with substantial differences in layout, structure, and text density among different tables. Therefore, an adaptive table region detection and text extraction method is needed to ensure efficient table parsing. Second, in real-world environments, the correspondence between text and table regions may be inconsistent. Extracting text solely based on table regions may lead to information loss or redundancy; therefore, text extraction needs to be based on semantic understanding of the table content and the contextual dependencies of the information. This invention addresses both of these technical problems.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, this invention provides a heterogeneous table extraction method based on a hybrid visual strategy, comprising the following steps:

[0008] Step 1: Perform table region identification and structured analysis based on improved Mask R-CNN and Unet.

[0009] In table region recognition and structured analysis, an improved method combining Mask R-CNN and Unet models is used to achieve table region detection and structure parsing. The Mask R-CNN model is improved as follows: an edge-aware loss function is introduced, which gives higher weight to the pixels at the edge of the table during the learning process.

[0010] The specific method for improving the Mask R-CNN model by introducing an edge-aware loss function is as follows: First, an edge map E(x,y) is generated using the image gradient, representing whether each pixel in the image is a table edge pixel:

[0011]

[0012] For each pixel, the segmentation error is associated with its edge label, and the edge-aware loss is defined as:

[0013]

[0014] Where p(x,y) are the actual table pixel labels; ω(x,y) is the pixel segmentation probability predicted by the model; ω(x,y) is the weight function, defined based on the edge map E(x,y).

[0015] ω(x,y)=1+αE(x,y) (3)

[0016] Where α is a constant and satisfies α>1, so that edge pixels are given higher weights; the final loss function is expressed as:

[0017] L total =L mask +λL edge (4)

[0018] Among them, L mask It is the loss function of the original Mask R-CNN model, and λ is the importance coefficient that controls the edge loss;

[0019] Then, the improved Mask R-CNN and Unet models are combined. The table region detection results of the former are used to provide accurate initial regions for the structured analysis of the latter. The specific implementation process is as follows: First, the improved Mask R-CNN model is used to detect and locate the table regions and generate a pixel-level mask map of the table. Then, the pixel-level mask map is input into the Unet model to perform detailed segmentation of columns and cells to obtain a structured representation of the table.

[0020] Step 2: Based on IoT metrics, use the PP-OCRv4 model to extract text from the table.

[0021] First, using the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model, the text regions in the structured representation table obtained in step one of DBNet are detected. By binarizing the text regions in the table, the text regions are located in complex backgrounds, and the binarization threshold is adjusted through backpropagation. Then, using the lightweight convolutional recurrent neural network structure adopted by the text recognition module, features in the text regions are extracted, and sequence modeling is performed through recurrent neural networks to capture the temporal dependencies of the text.

[0022] Then, an IoT metric is introduced to measure the ratio of overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely inside the table cell. The formula is as follows:

[0023]

[0024] Set an IoT threshold to determine if the text is large enough to fit inside the table cell and adjust the table cell boundary. When the IoT value is too low, the table cell size is increased by automatically adjusting the table cell boundary so that more text is contained inside the table cell. If the text exceeds the table cell by a large margin, the text is split into multiple lines or pages to fit the table cell space.

[0025] Step 3: Based on BERT-BiLSTM-Attention-CRF, a model consisting of BERT network, BiLSTM network, Attention mechanism and CRF is used to understand the text content in the table and build a knowledge base.

[0026] The present invention also provides a system for implementing the method.

[0027] (III) Beneficial Effects

[0028] This invention provides a heterogeneous table extraction method based on a hybrid visual strategy, with the main advantages being as follows:

[0029] 1. A table detection method based on fusion deep learning algorithm is proposed: This invention addresses the shortcomings of existing OCR technology in recognizing real table data, especially the problems of image orientation distortion and text structure diversity. It proposes a strategy that integrates lightweight table detection, multi-scale text detection, and detection box fine-tuning technology to achieve accurate localization of table regions and structures. Furthermore, the overall structure of the table is analyzed and reconstructed through a multimodal learning method, enabling the extracted data to be stored in a standardized manner, thus ensuring data consistency and integrity.

[0030] 2. To address the issue of inconsistent correspondence between text and table areas in real-world data, a table text extraction algorithm is proposed. This algorithm incorporates IoT metrics to handle line breaks and page breaks, correcting text detection capabilities and identifying and associating multiple lines of text belonging to the same cell. Furthermore, it models the contextual information within the text, enhancing the recognition of line breaks, ensuring text coherence, and significantly improving the accuracy of table parsing.

[0031] 3. Building upon tabular data recognition and content understanding, this invention further explores how to apply the extracted structured data to the construction of a military knowledge base. Through an instanced military intelligence analysis scenario, it demonstrates the importance of efficient tabular data extraction in enhancing knowledge base construction and decision support capabilities. Attached Figure Description

[0032] Figure 1 This is the overall flowchart of the present invention;

[0033] Figure 2 This is a framework diagram of the table detection method based on a fusion deep learning algorithm in the present invention. Detailed Implementation

[0034] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0035] This invention proposes a method for identifying and extracting tabular data in the military field. By improving existing table parsing techniques and combining deep learning and natural language processing technologies, it enhances the accuracy and efficiency of tabular data identification and extraction, laying the foundation for building a high-quality military knowledge base. This invention overcomes the current technical bottlenecks in tabular data identification and provides data support for future military intelligence analysis and automated decision-making systems.

[0036] like Figure 1 The diagram shows the overall flowchart of a heterogeneous table extraction method based on a hybrid vision strategy according to the present invention. This method includes three steps: table extraction region detection, row, column, and cell segmentation, and table content understanding and knowledge base construction. First, a method based on improved Mask R-CNN and Unet for table region recognition and structured analysis is used to locate table regions and structures. Then, a method fusing PP-OCRv4 table text extraction based on IoT metrics and table content understanding based on BERT-BiLSTM-Attention-CRF is used to achieve text recognition and standardized storage of real-world table data.

[0037] The three steps are described in detail below.

[0038] Step 1: Perform table region identification and structured analysis based on improved Mask R-CNN and Unet.

[0039] In table region recognition and structured analysis, this invention employs a combination of an improved Mask R-CNN and Unet model to achieve efficient table region detection and structure parsing. First, Mask R-CNN, as the base model, possesses powerful object detection and instance segmentation capabilities, enabling accurate detection of table regions. Its principle is to add a branch to Faster R-CNN to generate pixel-level segmentation masks. However, Mask R-CNN is prone to false positives or false negatives when handling complex backgrounds or fine table lines, especially in cases of complex table structures and high background noise. To address this issue, this invention improves the Mask R-CNN model in three aspects, as shown in the technical framework diagram below. Figure 2 As shown.

[0040] (1) ResNeXt-101 was adopted as a new backbone network, and the ability of the model to capture details of the table region was improved by introducing grouped convolution.

[0041] (2) Introduce channel attention and spatial attention mechanisms into the FPN network to enable the model to focus more on table edges and structural features, thereby improving detection accuracy;

[0042] (3) An edge-aware loss function is introduced, which gives higher weight to the pixels at the edge of the table during the learning process, further improving the detection accuracy of the table region.

[0043] The specific method for improving the Mask R-CNN model by introducing an edge-aware loss function is as follows: First, an edge map is generated using image gradients or other edge detection algorithms to indicate whether each pixel in the image is a table edge pixel:

[0044]

[0045] For each pixel, the segmentation error is associated with its edge label, and the edge-aware loss is defined as:

[0046]

[0047] Where p(x,y) is the actual table pixel label (0 or 1); ω(x,y) is the pixel segmentation probability predicted by the model; ω(x,y) is the weight function, defined based on the edge map E(x,y).

[0048] ω(x,y)=1+αE(x,y) (3)

[0049] Here, α is a constant satisfying α>1, to assign higher weights to edge pixels. The final loss function can be expressed as:

[0050] L total =L mask +λL edge (4)

[0051] Among them, L mask It is the loss function of the original Mask R-CNN model, and λ is the importance coefficient that controls the edge loss.

[0052] Then, an improved Mask R-CNN and the Unet model are combined. The table region detection results from the former provide accurate initial regions for the latter's structured analysis. The specific implementation process is as follows: First, the improved Mask R-CNN model is used to detect and locate table regions, generating a pixel-level mask image of the table. Next, this pixel-level mask image is input into the Unet model for detailed segmentation of columns and cells, resulting in a structured representation of the table. This method effectively combines the region detection capabilities of Mask R-CNN with the pixel-level segmentation advantages of Unet, achieving high-precision analysis of complex table structures.

[0053] For special applications of scanning paper documents, such as broken lines, after obtaining the structured representation of the table, morphological operations (such as dilation and erosion) are needed to process the table lines to ensure their integrity. Then, Hough transform is used to detect straight lines and calculate the intersection of horizontal and vertical lines. By setting a threshold, noisy lines are removed, and only lines that conform to the table's framework structure are retained, thus enabling the positioning of cell areas in the table.

[0054] Step 2: Based on IoT metrics, use the PP-OCRv4 model to extract text from the table.

[0055] First, the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model is used to detect text regions in the structured representation table obtained in step one using DBNet (Detection Model). By binarizing the text regions in the table, the text regions can be located quickly and accurately in complex backgrounds. Furthermore, the binarization threshold is adjusted through backpropagation, which improves the flexibility and accuracy of detection. Then, the lightweight convolutional recurrent neural network structure used in the text recognition module is used to extract features from the text regions, and sequence modeling is performed through the recurrent neural network to capture the temporal dependencies of the text.

[0056] Then, to address the issue of text extending beyond the table area in some real-world data, the IoT (Intersection Area over Text Area) metric is introduced to measure the ratio of overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely within the table cell. The formula is as follows:

[0057]

[0058] An IoT threshold is set to determine if the text is large enough to fit within a table cell, allowing for adjustments to the cell boundaries. If the IoT value is too low, the cell size can be increased by automatically adjusting the cell boundaries, allowing more text to be contained within the cells. If the text significantly exceeds the cell boundaries, it can be split into multiple lines or pages to fit within the cell space. For text segments spanning multiple pages, the text can be split into upper and lower parts based on the IoT calculation results, and the content spanning multiple pages can be annotated in the recognition results of the text area, ensuring the text order remains unchanged.

[0059] Step 3: Implement text content understanding and knowledge base building in the table based on BERT-BiLSTM-Attention-CRF (i.e., a model composed of BERT network, BiLSTM network, attention mechanism, and CRF).

[0060] In the table content understanding and information extraction task, for the text content of the table text region detection that has been completed in step two, this invention first inputs the text extracted from the table into the BERT network to obtain the context feature representation of each word and output the embedding vector of each word. As a pre-trained language model, the BERT network can capture deep bidirectional contextual information.

[0061] Then, by inputting the embedding vector output from the BERT network into the BiLSTM network, the sequence features of the embedding vector can be further encoded, capturing forward and backward dependencies in the sequence and enhancing the ability to model text sequence information. The output of the BiLSTM network is a feature vector incorporating sequence features. Simultaneously, to highlight important information, an attention mechanism is added after the BiLSTM network. The attention mechanism generates a weight distribution by calculating the relevance between the hidden state and the global context, assigning higher weights to more important words. This allows the BiLSTM network to focus on key semantic information, improving the accuracy of information extraction.

[0062] Finally, a Conditional Random Field (CRF) is used to decode the sequence labels and output the final predicted text content of the cells. CRF can consider the dependencies between labels, ensuring that the predicted label sequence has global optimality.

[0063] Having completed the identification of unstructured to structured tabular data, the next step is to add the content extracted from various tables to a knowledge base, thus constructing the knowledge base. Since real-world data contains massive amounts of unstructured tabular data, automatically achieving its structured storage and forming a systematic knowledge data is crucial for building a knowledge base. To solve this problem and lay a solid data foundation for subsequent knowledge base construction, this invention explores an adaptive table format parsing and reconstruction technology.

[0064] First, based on the relevant knowledge system and existing tabular materials, the tabular materials are classified according to content and format. Tables in real-world data can be broadly categorized into the following two types:

[0065] (1) Data tables: used to display large amounts of structured data, such as project reports, financial statistics, and research data displays. Data tables usually have clear row and column relationships and detailed header information.

[0066] (2) Input Forms: Forms used for data input and information collection, such as personal applications, questionnaires, and meeting opinions. These forms focus more on information filling and collection, with headers consisting of titles and field names, and the content can be varied.

[0067] Then, the table storage framework is dynamically constructed according to different tables. Here, a dictionary between table types and main fields is constructed based on the characteristics of real data to provide multiple validations for the confirmation of the main fields of each type of table.

[0068] (1) For data tables, the table header fields are parsed based on the table area recognition technology in step one, and then the table data is stored in a "row-column" structure using a relational database. The main fields of the data can be set according to the specific identifier fields (such as project number) of the table header fields to facilitate subsequent queries and statistical analysis.

[0069] (2) For the input table, semantic analysis is performed on the title and fields of the input table to identify the main fields of the table (such as "applicant name", "project name" etc.). Combined with the semantic analysis of the title content and table fields, the main fields of the input table are determined and stored in the graph database according to the storage structure of [main field, secondary field, secondary field content] to form the relationship between the fields in the input table, which facilitates the generation of the knowledge base and subsequent semantic query.

[0070] Secondly, we handle unconventional situations such as multi-level headings and merged cells that may exist in various tables.

[0071] For multi-level headings, it is necessary to establish the hierarchical relationship between each heading and the data content to form a structured data tree. The specific steps are as follows:

[0072] (1) Preliminary identification of the hierarchical relationship of the headings through positional feature analysis:

[0073] By using the vertical and horizontal coordinates of the heading area, the relative positional relationship between different headings is determined. For example, if a heading is below another heading and horizontally centered, it is likely a subordinate heading. The width and height of the text boxes are used to determine the visual hierarchy of the headings within the table header; for instance, headings that are wider and located at the top are usually of a higher level. The detected header areas are then grouped according to their relative positions to initially establish a hierarchical distribution of headings.

[0074] (2) Determine the heading level through semantic relation parsing:

[0075] Semantic analysis is performed on the text content of each title to extract its main meaning and contextual relationships. Contextual embedding vectors are then used to analyze the similarity between titles, determining which titles belong to the same topic or category, thereby further identifying their hierarchical relationships.

[0076] If there are merged cells, the corresponding content cells are split based on the hierarchical field relationships. The content is then identified by OCR and semantic analysis is performed to bind the text content to the corresponding cell coordinates.

[0077] Finally, during the standardization process, the content information of the cells that have been split and processed is mapped to the reconstructed structured table framework according to the table category, and the data is stored in a standardized manner according to the table type-main field mapping dictionary.

[0078] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A heterogeneous table extraction method based on a hybrid visual strategy, characterized in that, Includes the following steps: Step 1: Perform table region identification and structured analysis based on improved Mask R-CNN and Unet. In table region recognition and structured analysis, an improved method combining Mask R-CNN and Unet models is used to achieve table region detection and structure parsing. The Mask R-CNN model is improved as follows: an edge-aware loss function is introduced, which gives higher weight to the pixels at the edge of the table during the learning process. The specific method for improving the Mask R-CNN model by introducing an edge-aware loss function is as follows: First, an edge map E(x,y) is generated using the image gradient, representing whether each pixel in the image is a table edge pixel: For each pixel, the segmentation error is associated with its edge label, and the edge-aware loss is defined as: Where p(x,y) are the actual table pixel labels; ω(x,y) is the pixel segmentation probability predicted by the model; ω(x,y) is the weight function, defined based on the edge map E(x,y). ω(x,y)=1+αE(x,y) (3) Where α is a constant and satisfies α>1, so that edge pixels are given higher weights; the final loss function is expressed as: THE total =L mask +λL edge (4) Among them, L mask It is the loss function of the original Mask R-CNN model, and λ is the importance coefficient that controls the edge loss; Then, the improved Mask R-CNN and Unet models are combined. The table region detection results of the former are used to provide accurate initial regions for the structured analysis of the latter. The specific implementation process is as follows: First, the improved Mask R-CNN model is used to detect and locate the table regions and generate a pixel-level mask map of the table. Then, the pixel-level mask map is input into the Unet model to perform detailed segmentation of columns and cells to obtain a structured representation of the table. Step 2: Based on IoT metrics, use the PP-OCRv4 model to extract text from the table. First, using the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model, the text regions in the structured representation table obtained in step one of DBNet are detected. By binarizing the text regions in the table, the text regions are located in complex backgrounds, and the binarization threshold is adjusted through backpropagation. Then, using the lightweight convolutional recurrent neural network structure adopted by the text recognition module, features in the text regions are extracted, and sequence modeling is performed through recurrent neural networks to capture the temporal dependencies of the text. Then, an IoT metric is introduced to measure the ratio of overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely inside the table cell. The formula is as follows: Set an IoT threshold to determine if the text is large enough to fit inside the table cell and adjust the table cell boundary. When the IoT value is too low, the table cell size is increased by automatically adjusting the table cell boundary so that more text is contained inside the table cell. If the text exceeds the table cell by a large margin, the text is split into multiple lines or pages to fit the table cell space. Step 3: Based on BERT-BiLSTM-Attention-CRF, a model consisting of BERT network, BiLSTM network, Attention mechanism and CRF is used to understand the text content in the table and build a knowledge base.

2. The method as described in claim 1, characterized in that, In step one, the Mask R-CNN model was further improved as follows: (1) ResNeXt-101 is used as the backbone network, and grouped convolution is introduced to improve the model’s ability to capture details of the table region; (2) Introduce channel attention and spatial attention mechanisms into the FPN network to enable the model to focus more on table edges and structural features.

3. The method as described in claim 1, characterized in that, For some special uses of paper document scanning, after obtaining the structured representation of the table in step one, morphological operations are used to process the table lines to ensure the integrity of the lines; then, Hough transform is used to detect straight lines and calculate the intersection positions of horizontal and vertical lines. By setting a threshold, noisy lines are removed, and only lines that conform to the table frame structure are retained, thus enabling the positioning of cell areas in the table.

4. The method as described in claim 1, characterized in that, In step two, for text segments that span multiple pages, the text is split into two parts based on the calculation results of IoT, and the content spanning multiple pages is marked in the recognition results of the text area to ensure that the text order remains unchanged.

5. The method as described in claim 1, characterized in that, In step three, for the table content understanding and information extraction task, the text content of the table text region detection completed in step two is first input into the BERT network to obtain the contextual feature representation of each word and output the embedding vector of each word. As a pre-trained language model, the BERT network can capture deep-level bidirectional contextual information. Then, by inputting the embedding vector output by the BERT network into the BiLSTM network, the sequence features of the embedding vector can be further encoded to capture the forward and backward dependencies in the sequence, enhancing the ability to model text sequence information. The output of the BiLSTM network is a feature vector with added sequence features. At the same time, an Attention mechanism is added after the BiLSTM network. The Attention mechanism generates a weight distribution by calculating the correlation between the hidden state and the global context, assigning higher weights to more important words. Finally, a Conditional Random Field (CRF) is used to decode the sequence labeling and output the final predicted content of the text in the cell.

6. The method as described in claim 1, characterized in that, Conditional random fields can take into account the dependencies between labels, ensuring that the predicted label sequence has global optimality.

7. The method as described in claim 1, characterized in that, After step three completes the identification of unstructured to structured tabular data, the next step is to add the content extracted from various tables to the knowledge base, thus realizing the construction of the knowledge base.

8. The method as described in claim 7, characterized in that, In the process of building the knowledge base, an adaptive table format parsing and reconstruction method is used to achieve structured storage and form a systematic knowledge data.

9. The method as described in claim 8, characterized in that, The following adaptive table format parsing and reconstruction method is used to achieve structured storage and form a systematic knowledge data: First, the table materials are divided into the following two categories based on content and format: (1) Data table: used to display structured data, and the data table has a clear row and column relationship; (2) Input Table: A table used for data input and information collection; Then, the table storage framework is dynamically constructed according to different tables, and a dictionary between table types and main fields is constructed according to data characteristics to provide multiple validations for the confirmation of main fields of each type of table; (1) For the data table, the table header field is parsed based on the table area recognition technology in step one, and then the table data is stored in a "row-column" structure using a relational database. The main field of the data is set according to the specific identifier field of the table header field. (2) For the input table, semantic analysis is performed on the title and fields of the input table to identify the main field of the table. Combined with the semantic analysis of the title content and table fields, the main field of the input table is determined and stored in the graph database according to the storage structure of [main field, secondary field, secondary field content] to form the association relationship between the fields in the input table; Secondly, we need to handle the unusual cases of multi-level headings and merged cells in various tables; For multi-level headings, establish the hierarchical relationship between each heading and the data content to form a structured data tree. The specific steps are as follows: (1) Preliminary identification of the hierarchical relationship of the headings through positional feature analysis: By using the vertical and horizontal coordinates of the title area, the relative positional relationship between different titles is determined; by using the width and height of the text box, the visual hierarchy of the titles in the table header is determined; then, the detected table header areas are grouped according to their relative positions to initially form the hierarchical distribution of the titles. (2) Determine the heading level through semantic relation parsing: Semantic analysis is performed on the text content of each title to extract its meaning and contextual relationships; through contextual embedding vectors, the similarity between titles is analyzed to determine which titles belong to the same topic or category, thereby further determining their hierarchical relationships; If there are merged cells, the corresponding content cells are split based on the hierarchical field relationship, and the content is identified by OCR and semantic analysis is performed to bind the text content to the corresponding cell coordinates. Finally, during the standardization process, the content information of the cells that have been split is mapped to the reconstructed structured table framework according to the table category, and the data is stored in a standardized manner according to the table type-main field mapping dictionary.

10. A system for implementing the method as described in any one of claims 1 to 9.

Images