Heterogeneous table extraction method based on hybrid vision strategy

Abstract

The invention relates to a heterogeneous table extraction method based on a hybrid vision strategy, and belongs to the technical field of artificial intelligence, big data, computer vision and natural language processing. According to the method, the existing table analysis technology is improved, and deep learning and natural language processing technologies are combined, so that the identification accuracy and extraction efficiency of table data are improved, and a foundation is laid for constructing a high-quality military knowledge base. According to the method, the technical bottleneck in current table data identification is improved, and data support is provided for future military intelligence analysis and an automatic decision-making system.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence, big data, computer vision and natural language processing, and specifically relates to a heterogeneous table extraction method based on a hybrid vision strategy. Background Art

[0002] In modern military research and intelligence analysis, data is the core element that supports decision-making and strategy formulation. A large amount of military data exists in unstructured form, such as historical documents, intelligence reports, training records, etc. These data are often stored in paper form and digitally preserved by scanning, forming a large amount of image format data. However, much key information, including structured table data, is hidden in these unstructured image files. How to efficiently extract and utilize them has become one of the key technologies for building knowledge bases and intelligence analysis.

[0003] Tabular data has highly structured information, usually including data relationships, numerical statistics, and classification information, so it has irreplaceable value in military research. However, since there are many documents in real data that are scanned from paper, the tables have diverse forms, complex formats, and many difficulties in text recognition. Traditional table recognition methods face great challenges in processing such data. Accurately extracting tables from these image files and converting them into usable structured data is not only the key to improving data utilization efficiency, but also the basis for building an intelligent military knowledge base. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is: the present invention proposes a method for automatic recognition and extraction of unstructured tabular data in the military field. Taking into account the complexity of tabular data in practical applications, this task mainly faces the following two challenges: First, the diversity of table areas and text contents is large, and the layout, structure and text density of different tables vary significantly. Therefore, an adaptive table area detection and text extraction method is required to ensure efficient table parsing. Secondly, the data in the real environment may have irregular correspondence between text and table areas. Extracting text only based on the table area may cause information loss or redundancy. It is necessary to extract text based on the semantic understanding of the table content and the contextual dependency of the information. The present invention can solve the above two technical problems.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention provides a heterogeneous table extraction method based on a hybrid visual strategy, comprising the following steps:

[0008] Step 1: Table area recognition and structural analysis based on improved Mask R-CNN and Unet

[0009] In table area recognition and structured analysis, a method combining the improved Mask R-CNN and Unet models is used to achieve table area detection and structure analysis. The Mask R-CNN model is improved as follows: an edge-aware loss function is introduced to enable the model to give higher weights to table edge pixels during the learning process;

[0010] The specific method of introducing the edge-aware loss function to improve the Mask R-CNN model is as follows: First, use the image gradient to generate the edge map E(x, y), which indicates whether each pixel in the image is a table edge pixel:

[0011]

[0012] For each pixel, the segmentation error is associated with its edge label and the edge-aware loss is defined as:

[0013]

[0014] Where: p(x,y) is the actual table pixel label; is the pixel segmentation probability predicted by the model; ω(x, y) is the weight function, which is defined according to the edge graph E(x, y):

[0015] ω(x,y)=1+αE(x,y) (3)

[0016] Where α is a constant and satisfies α>1 to give higher weights to edge pixels; the final loss function is expressed as:

[0017] L total =L mask +λL edge (4)

[0018] Among them, L mask is the loss function of the original Mask R-CNN model, and λ is the importance coefficient for controlling edge loss;

[0019] Then, the improved Mask R-CNN and Unet models are combined, and the table area detection results of the former are used to provide accurate initial areas for the structured analysis of the latter. The specific implementation process is as follows: first, the improved Mask R-CNN model is used to detect and locate the table area, and a pixel-level mask map of the table is generated; then, the pixel-level mask map is input into the Unet model for detailed segmentation of columns and cells to obtain a structured representation of the table;

[0020] Step 2: Based on IoT indicators, use the PP-OCRv4 model to extract table text

[0021] First, the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model is used to detect the text area in the table of the structured representation obtained in step 1 using DBNet. The text area in the table is binarized to locate the text area in the complex background, and the binarization threshold is adjusted through back propagation. The lightweight convolutional recurrent neural network structure adopted by the text recognition module is used to extract features in the text area, and sequence modeling is performed through the recurrent neural network to capture the temporal dependency of the text.

[0022] Then, the IoT indicator is introduced to measure the ratio of the overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely inside the table cell. The formula is:

[0023]

[0024] Set an IoT threshold to determine whether the text is large enough to fit inside the table cell to adjust the table cell border. When the IoT value is too low, the table cell size is increased by automatically adjusting the table cell border so that more text is included in the table cell. If the text is too large to fit inside the table cell, the text is split into multiple lines or displayed in pages to fit the space of the table cell.

[0025] Step 3: Based on BERT-BiLSTM-Attention-CRF, a model consisting of BERT network, BiLSTM network, Attention mechanism, and CRF is used to understand the text content in the table and establish a knowledge base.

[0026] The present invention also provides a system for implementing the method.

[0027] (III) Beneficial effects

[0028] The present invention provides a heterogeneous table extraction method based on a hybrid visual strategy, the main advantages of which are reflected in the following aspects:

[0029] 1. A table detection method based on a fusion deep learning algorithm is proposed: In view of the shortcomings of existing OCR technology in the recognition of real table data, especially the problems of image direction distortion and text structure diversity, the present invention proposes a strategy that integrates lightweight table detection, multi-scale text detection, and detection frame fine-tuning technology to achieve precise positioning of table areas and structures, and parses and reconstructs the overall structure of the table through a multimodal learning method, so that the extracted data can be stored in a standardized manner, ensuring the consistency and integrity of the data.

[0030] 2. In view of the phenomenon that the correspondence between text and table areas in real data is not standardized, a table text extraction algorithm is proposed, which introduces IoT indicators to deal with the problems of text line breaks and page breaks, can correct text detection capabilities, and identify and associate multiple lines of text belonging to the same cell. At the same time, the context information in the text is modeled, thereby enhancing the recognition ability of line-break text, ensuring the coherence of text content, and greatly improving the accuracy of table parsing.

[0031] 3. Based on the recognition and content understanding of tabular data, this paper further explores how to apply the extracted structured data to the construction of a military knowledge base. Through an instantiated military intelligence analysis scenario, it demonstrates the importance of efficient tabular data extraction in improving knowledge base construction and decision support capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall flow chart of the present invention;

[0033] Figure 2 This is a framework diagram of the table detection method based on the fusion deep learning algorithm in the method of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0035] This paper proposes a method for identifying and extracting tabular data in the military field. By improving the existing table parsing technology and combining deep learning and natural language processing technology, the recognition accuracy and extraction efficiency of tabular data are improved, laying the foundation for building a high-quality military knowledge base. This paper improves the current technical bottleneck in tabular data recognition and also provides data support for future military intelligence analysis and automated decision-making systems.

[0036] like Figure 1 The figure shows the overall flow chart of a heterogeneous table extraction method based on a hybrid visual strategy of the present invention, which includes three steps: table extraction area detection, row and column and cell segmentation, table content understanding and knowledge base construction. First, the table area and structure are located by using a table area recognition and structured analysis method based on improved Mask R-CNN and Unet, and then the text extraction of the real table data is realized by integrating the PP-OCRv4 table text extraction based on IoT indicators and the table content understanding method based on BERT-BiLSTM-Attention-CRF.

[0037] The three steps are described in detail below.

[0038] Step 1: Table area recognition and structural analysis based on improved Mask R-CNN and Unet

[0039] In table area recognition and structured analysis, the present invention adopts a method combining an improved Mask R-CNN and Unet model to achieve efficient table area detection and structural analysis. First of all, as a basic model, Mask R-CNN has powerful target detection and instance segmentation capabilities, and can accurately detect the area of ​​the table. The principle is to add a branch on the basis of Faster R-CNN to generate pixel-level segmentation masks. However, Mask R-CNN is prone to missed detection or false detection when dealing with complex backgrounds or fine table lines, especially when the table structure is complex and the background noise is high. In order to solve this problem, the present invention has made three improvements to the Mask R-CNN model. The technical framework diagram is shown below. Figure 2 shown.

[0040] (1) ResNeXt-101 is used as the new backbone network, and group convolution is introduced to improve the model's ability to capture details of the table area;

[0041] (2) Introducing channel attention and spatial attention mechanisms into the FPN network enables the model to focus more on table edges and structural features, thereby improving detection accuracy;

[0042] (3) The edge-aware loss function is introduced to enable the model to give higher weights to table edge pixels during the learning process, further improving the detection accuracy of the table area.

[0043] The specific method of introducing the edge-aware loss function to improve the Mask R-CNN model is as follows: First, use the image gradient or other edge detection algorithms to generate an edge map to indicate whether each pixel in the image is a table edge pixel:

[0044]

[0045] For each pixel, the segmentation error is associated with its edge label and the edge-aware loss is defined as:

[0046]

[0047] Where: p(x,y) is the actual table pixel label (0 or 1); is the pixel segmentation probability predicted by the model; ω(x, y) is the weight function, which is defined according to the edge graph E(x, y):

[0048] ω(x,y)=1+αE(x,y) (3)

[0049] Where α is a constant and satisfies α>1, so as to give higher weights to edge pixels. The final loss function can be expressed as:

[0050] L total =L mask +λL edge (4)

[0051] Among them, L mask is the loss function of the original Mask R-CNN model, and λ is the importance coefficient for controlling edge loss.

[0052] Then, the improved Mask R-CNN and Unet models are combined, and the table area detection results of the former are output to provide accurate initial areas for the structured analysis of the latter. The specific implementation process is: first, the improved Mask R-CNN model is used to detect and locate the table area, and a pixel-level mask map of the table is generated; then, the pixel-level mask map is input into the Unet model for detailed segmentation of columns and cells to obtain a structured representation of the table. This method can effectively combine the regional detection capability of Mask R-CNN and the pixel-level segmentation advantage of Unet to achieve high-precision analysis of complex table structures.

[0053] For some special uses that occur when scanning paper documents, such as broken lines, after obtaining the structured representation of the table, morphological operations (such as dilation and erosion) are needed to process the table lines to ensure the integrity of the lines; then the Hough transform is used to detect straight lines and calculate the intersection positions of horizontal and vertical lines; by setting the threshold, the noise lines are removed and only the lines that conform to the table frame structure are retained to achieve the positioning of the cell area in the table.

[0054] Step 2: Based on IoT indicators, use the PP-OCRv4 model to extract table text

[0055] Firstly, the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model is used to detect the text area in the table of the structured representation obtained in step one using DBNet (Detection Model). By binarizing the text area in the table, the text area can be quickly and accurately located in a complex background, and the binarization threshold is adjusted through back propagation to improve the flexibility and accuracy of detection. The lightweight convolutional recurrent neural network structure adopted by the text recognition module is used to extract features in the text area, and sequence modeling is performed through recurrent neural networks to capture the temporal dependencies of text.

[0056] Then, in order to solve the problem that the text in some real data exceeds the table area, the IoT (Intersection_Area over Text_Area) indicator is introduced to measure the ratio of the overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely inside the table cell. The formula is:

[0057]

[0058] Set an IoT threshold to determine whether the text is large enough to fit inside the table cell to adjust the table cell border. When the IoT value is too low, the table cell size can be increased by automatically adjusting the table cell border so that more text is included in the table cell. If the text exceeds the table cell, the text can be split into multiple lines or displayed in pages to fit the space of the table cell. For text segments that span pages, the text can be split into upper and lower parts based on the calculation results of IoT, and the span content can be marked in the recognition results of the span text area to ensure that the text order remains unchanged.

[0059] Step 3: Based on BERT-BiLSTM-Attention-CRF (i.e., a model consisting of BERT network, BiLSTM network, Attention mechanism, and CRF), we can understand the text content in the table and establish a knowledge base

[0060] In the table content understanding and information extraction tasks, for the text content of the table text area detection that has completed step 2, the present invention first inputs the text extracted from the table into the BERT network, obtains the context feature representation of each word, and outputs the embedding vector of each word. The BERT network, as a pre-trained language model, can capture deep bidirectional context information.

[0061] Then, by inputting the embedding vector output by the BERT network into the BiLSTM network, the sequence features of the embedding vector can be further encoded, the forward and backward dependencies in the sequence can be captured, and the modeling ability of text sequence information can be enhanced. The output of the BiLSTM network is a feature vector with sequence features added. At the same time, in order to highlight important information, the Attention mechanism is added after the BiLSTM network. The Attention mechanism generates a weight distribution by calculating the correlation between the hidden state and the global context, giving more important words a higher weight. This enables the BiLSTM network to focus on key semantic information and improve the accuracy of information extraction.

[0062] Finally, the conditional random field (CRF) is used to decode the sequence annotations and output the final predicted text content in the cell. The conditional random field can take into account the dependencies between labels and ensure that the predicted label sequence is globally optimal.

[0063] The recognition of table data from unstructured to structured has been completed. Next, the content extracted from various tables is added to the knowledge base to realize the construction of the knowledge base. Since there is a large amount of unstructured table data in real data, how to automatically realize its structured storage and form systematic knowledge data is the key to building a knowledge base. In order to solve this problem and lay a good data foundation for the subsequent knowledge base construction, the present invention explores an adaptive table format parsing and reconstruction technology.

[0064] First, according to the corresponding knowledge system and existing table materials, the table materials are classified according to content and form. The tables in real data can be roughly divided into the following two categories:

[0065] (1) Data tables: used to display large amounts of structured data, such as project reports, financial statistics, and research data displays. Data tables usually have clear row-column relationships and detailed header information.

[0066] (2) Input forms: forms used for data input and information collection, such as personal applications, questionnaires, meeting opinions, etc. This type of form focuses more on filling in and collecting information. The header is mostly a title and field name, and the content is filled in in a variety of ways.

[0067] Then, the table storage framework is dynamically constructed according to different tables. Here, a dictionary between table types and main fields is constructed according to the characteristics of actual data, providing multiple checks for the main field confirmation of each type of table.

[0068] (1) For data tables, the table header fields are parsed based on the table area recognition technology in step 1, and then the table data is stored in a "row-column" structure using a relational database. The main field of the data can be set according to the specific identification field of the table header field (such as the project number, etc.) to facilitate subsequent query and statistical analysis;

[0069] (2) For input forms, semantic analysis is performed on the title and fields of the input form to identify the main fields of the form (such as "applicant's name", "project name", etc.). Combined with the semantic analysis of the title content and the form fields, the main fields of the input form are determined and stored in the graph database according to the storage structure of [primary field, secondary field, secondary field content], forming an association relationship between the fields in the input form, which is convenient for the generation of the knowledge base and subsequent semantic queries.

[0070] Secondly, handle the unusual situations of multi-level headers and merged cells that may exist in various tables.

[0071] For multi-level titles, it is necessary to establish the hierarchical relationship between each title and the data content to form a structured data tree. The specific steps are as follows:

[0072] (1) Preliminary identification of the hierarchical relationship of titles through position feature analysis:

[0073] The vertical and horizontal coordinates of the title area are used to determine the relative position relationship between different titles. For example, if a title is located below another title and is horizontally centered, it may be a subordinate title. The width and height of the text box are used to determine the visual hierarchy of the title in the header. For example, the title with a larger width and located at the top is usually at a higher level. The detected header areas are then grouped according to their relative positions to preliminarily form a hierarchical distribution of the titles.

[0074] (2) Determine the title hierarchy through semantic relationship analysis:

[0075] Semantic analysis is performed on the text content of each title to extract its main meaning and contextual relationship. Through the contextual embedding vector, the similarity between titles is analyzed to determine which titles belong to the same topic or category, and thus their hierarchical relationship is further determined.

[0076] If there are merged cells, the corresponding content cells are split based on the hierarchical field relationship, the content is recognized and semantic analysis is performed through OCR, and the text content is bound to the corresponding cell coordinates.

[0077] Finally, in the standardization process, the content information of the cell identification that has been split and processed is mapped to the reconstructed structured table framework according to the table classification, and the data is stored in a standardized manner according to the table type-primary field mapping dictionary.

[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A heterogeneous table extraction method based on hybrid visual strategy, characterized in that: The following steps are involved: Step 1: Table area recognition and structural analysis based on improved Mask R-CNN and Unet In table area recognition and structured analysis, a method combining the improved Mask R-CNN and Unet models is used to achieve table area detection and structure analysis. The Mask R-CNN model is improved as follows: an edge-aware loss function is introduced to enable the model to give higher weights to table edge pixels during the learning process; The specific method of introducing the edge-aware loss function to improve the Mask R-CNN model is as follows: First, use the image gradient to generate the edge map E(x, y), which indicates whether each pixel in the image is a table edge pixel: For each pixel, the segmentation error is associated with its edge label and the edge-aware loss is defined as: Where: p(x,y) is the actual table pixel label; is the pixel segmentation probability predicted by the model; ω(x, y) is the weight function, which is defined according to the edge graph E(x, y): ω(x,y)=1+αE(x,y) (3) Where α is a constant and satisfies α>1 to give higher weights to edge pixels; the final loss function is expressed as: THE total =L mask +λL edge (4) Among them, L mask is the loss function of the original Mask R-CNN model, and λ is the importance coefficient for controlling edge loss; Then, the improved Mask R-CNN and Unet models are combined, and the table area detection results of the former are used to provide accurate initial areas for the structured analysis of the latter. The specific implementation process is as follows: first, the improved Mask R-CNN model is used to detect and locate the table area, and a pixel-level mask map of the table is generated; then, the pixel-level mask map is input into the Unet model for detailed segmentation of columns and cells to obtain a structured representation of the table; Step 2: Based on IoT indicators, use the PP-OCRv4 model to extract table text First, the differentiable binarization algorithm of the text detection module of the PP-OCRv4 model is used to detect the text area in the table of the structured representation obtained in step 1 using DBNet. The text area in the table is binarized to locate the text area in the complex background, and the binarization threshold is adjusted through back propagation. The lightweight convolutional recurrent neural network structure adopted by the text recognition module is used to extract features in the text area, and sequence modeling is performed through the recurrent neural network to capture the temporal dependency of the text. Then, the IoT indicator is introduced to measure the ratio of the overlap between the text area and the table area. By evaluating the overlap ratio between the text area and the table cell, it is determined whether the text is completely inside the table cell. The formula is: Set an IoT threshold to determine whether the text is large enough to fit inside the table cell to adjust the table cell border. When the IoT value is too low, the table cell size is increased by automatically adjusting the table cell border so that more text is included in the table cell. If the text is too large to fit inside the table cell, the text is split into multiple lines or displayed in pages to fit the space of the table cell. Step 3: Based on BERT-BiLSTM-Attention-CRF, a model consisting of BERT network, BiLSTM network, Attention mechanism, and CRF is used to understand the text content in the table and establish a knowledge base.

2. The method according to claim 1, characterized in that In step 1, the Mask R-CNN model is also improved as follows: (1) ResNeXt-101 is used as the backbone network, and group convolution is introduced to improve the model's ability to capture details of the table area; (2) Introducing channel attention and spatial attention mechanisms into the FPN network enables the model to focus more on table edges and structural features.

3. The method according to claim 1, characterized in that For some special uses that occur when scanning paper documents, after obtaining the structured representation of the table in step 1, morphological operations are used to process the table lines to ensure the integrity of the lines; then Hough transform is used to detect straight lines and calculate the intersection positions of horizontal and vertical lines; By setting a threshold, noise lines are removed and only lines that conform to the table frame structure are retained to achieve the positioning of the cell area in the table.

4. The method according to claim 1, characterized in that In step 2, for the text segment that spans across pages, the text is split into two parts, upper and lower parts, according to the calculation results of IoT, and the content of the span is marked in the recognition results of the text area that spans across pages to ensure that the text order remains unchanged.

5. The method according to claim 1, characterized in that Step 3: In the table content understanding and information extraction tasks, for the text content of the table text area detection that has completed step 2, first input the text extracted from the table into the BERT network to obtain the context feature representation of each word and output the embedding vector of each word. The BERT network, as a pre-trained language model, can capture deep bidirectional context information. Then, by inputting the embedding vector output by the BERT network into the BiLSTM network, the sequence features of the embedding vector can be further encoded, the forward and backward dependencies in the sequence can be captured, and the modeling ability of text sequence information can be enhanced. The output of the BiLSTM network is a feature vector with sequence features added. At the same time, the Attention mechanism is added after the BiLSTM network. The Attention mechanism generates a weight distribution by calculating the correlation between the hidden state and the global context, and gives more important words higher weights. Finally, the conditional random field CRF is used to realize the decoding of the sequence annotation and output the content of the text in the final predicted cell.

6. The method according to claim 1, characterized in that Conditional random fields can take into account the dependencies between labels and ensure that the predicted label sequence is globally optimal.

7. The method according to claim 1, characterized in that After completing the recognition of table data from unstructured to structured in step three, the next step is to add the content extracted from various tables to the knowledge base to realize the construction of the knowledge base.

8. The method according to claim 7, characterized in that In the process of building the knowledge base, an adaptive table format parsing and reconstruction method is used to achieve structured storage and form systematic knowledge data.

9. The method according to claim 8, characterized in that The following adaptive table format parsing and reconstruction methods are used to achieve structured storage and form systematic knowledge data: First, the table materials are divided into the following two categories according to content and form: (1) Data table: used to display structured data. The data table has a clear row-column relationship. (2) Input forms: forms used for data entry and information collection; Then, we dynamically build a table storage framework based on different tables, build a dictionary between table types and primary fields based on data characteristics, and provide multiple checks for primary field confirmation of each type of table; (1) For a data table, the table header fields are parsed based on the table area recognition technology in step 1, and then the table data is stored in a "row-column" structure using a relational database. The main field of the data is set according to the specific identification field of the table header field; (2) For the input table, semantic analysis is performed on the title and fields of the input table to identify the main field of the table. The main field of the input table is determined by combining the semantic analysis of the title content and the table field. The main field is stored in the graph database according to the storage structure of [main field, secondary field, secondary field content] to form an association relationship between the fields in the input table; Secondly, handle the unconventional situations of multi-level headers and merged cells in various tables; For multi-level titles, establish the hierarchical relationship between each title and the data content to form a structured data tree. The specific steps are as follows: (1) Preliminary identification of the hierarchical relationship of titles through position feature analysis: The relative position relationship between different titles is determined by using the vertical and horizontal coordinates of the title area. The visual hierarchy of the title in the header is determined by the width and height of the text box. Then the detected header areas are grouped according to their relative positions to preliminarily form a hierarchical distribution of the titles. (2) Determine the title hierarchy through semantic relationship analysis: Semantically analyze the text content of each title to extract its meaning and contextual relationship; analyze the similarity between titles through context embedding vectors to determine which titles belong to the same topic or category, and further determine their hierarchical relationship; If there are merged cells, the corresponding content cells are split according to the hierarchical field relationship, the content is recognized and semantically analyzed through OCR, and the text content is bound to the corresponding cell coordinates; Finally, in the standardization process, the content information of the split cell identification is mapped to the reconstructed structured table framework according to the table classification, and the data is stored in a standardized manner according to the table type-primary field mapping dictionary.

10. A system for implementing the method according to any one of claims 1 to 9.

Images