Bulletin PDF table detection and identification method and system based on hybrid strategy

Abstract

The invention relates to the technical field of document information processing, and provides an announcement PDF table detection and recognition method and system based on a hybrid strategy, and the method comprises the steps: extracting table data through a rule driving tool and a deep learning model in parallel, and obtaining two sets of results containing coordinates and texts; performing data alignment and consistency verification on the basis of coordinates of the deep learning model, when texts are inconsistent, calculating a geometric difference between the two sets of coordinates, and executing cell merging or boundary re-division operation according to whether the difference exceeds a threshold value or not to generate corrected data; using the corrected table data to train a document understanding prediction model fusing text, image, layout and cell characteristics; finally, the PDF is directly analyzed through the model, and the accurate content and the row and column position relation of each cell are output. Through the hybrid strategy and the intelligent correction mechanism, the structure detection robustness and the content recognition accuracy of the complex announcement PDF table are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of document information processing technology, specifically to a method and system for detecting and recognizing announcement PDF tables based on a hybrid strategy. Background Technology

[0002] The table extraction process for listed company announcement PDFs includes three sub-tasks: table region detection, table structure detection, and cell content recognition. For specific types of announcement PDFs, since the tables of a certain type to be extracted generally appear in specific chapters and paragraphs, the existing technologies are relatively mature. High accuracy has been achieved by combining deep learning technologies such as YOLOv3, Faster R-CNN, and BERT with rule-driven methods. However, considering that although the position of the table is relatively fixed, the table structure often changes and the content description is often varied, there is still considerable room for improvement in the table structure detection and cell content recognition sub-tasks. Summary of the Invention

[0003] To help solve the above-mentioned technical problems, this application provides a method and system for detecting and recognizing announcement PDF forms based on a hybrid strategy, which adopts the following technical solution.

[0004] A method for detecting the table structure and recognizing content in announcement PDFs based on a hybrid strategy includes the following steps: S1: Input the PDF document of the announcement to be processed into the rule-driven table recognition tool and the deep learning-based cell region prediction model respectively to generate the first table data and the second table data. The first table data contains the first text content and the first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. S2: Using the second coordinate information as a reference, align and match the first table data with the second table data; for each target cell position, perform consistency verification between the corresponding first text content and the second text content extracted from the area corresponding to the second coordinate information in the announcement PDF document. If the verification fails, calculate the geometric difference between the first coordinate information and the second coordinate information. If the geometric difference exceeds the preset threshold, perform the merging operation of adjacent row cells to generate corrected table data. S3: Construct a document understanding pre-trained model whose input feature vectors fuse at least text features, image features, layout features, and cell features. Use corrected tabular data to train or update the parameters of the document understanding pre-trained model. S4: Input the PDF document of the announcement to be recognized into the document understanding pre-trained model optimized in step S3. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

[0005] The announcement PDF document is a financial announcement PDF document released by a listed company that contains share reduction plans and financial data information. The training of the model in S3 includes a mask cell position prediction task to learn the correspondence between cells and document image regions. The information output by the model in S4 further includes the semantic understanding results of the table structure. The semantic understanding results include at least one or more of the following: table title, data type, and the relationship between data items.

[0006] S2 includes: The geometric difference is specifically the difference between the height of the area defined by the first coordinate information and the height of the area defined by the second coordinate information of the same target cell.

[0007] S2 includes: If the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is greater than the height of the area defined by the second coordinate information, then it is determined that the first table data identifies the cell area too large, and a merging operation is performed on adjacent row cells to integrate the content that should belong to the same cell.

[0008] S2 includes: If the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is less than the height of the area defined by the second coordinate information, it is determined that the first table data has too small an identification of the cell area, and a re-division operation is performed on the current cell boundary, using the second coordinate information as the cell boundary.

[0009] The input feature vector of the deep learning-based cell region prediction model is represented as follows: Where T represents text features, V represents image features, C represents cell features, S represents segment features, P represents paragraph length features, and M represents modal features.

[0010] The formula for constructing text feature T is as follows: Among them, Emb t (S) represents the feature obtained by embedding and encoding the input sequence S, and L represents the layout feature obtained by embedding and encoding the coordinate information of words or characters. The formula for constructing the layout feature L is as follows: (x0, y0) and (x1, y1) represent the top-left and bottom-right coordinates of the word or character bounding box, respectively, and w and h represent the width and height of the bounding box, respectively. The formula for constructing the cell feature C is as follows: Among them, Emb c (C) indicates that cell features are embedded and encoded, and L represents layout features. E i P represents the center code of the cell. i Indicates the coordinate code of the cell, O i The edge code represents the cell's edge code, which is used to characterize the connection relationship between cells. The sequence of the paragraph feature S is represented as follows: [CLS] and [SEP] are special markers, c j i The cell content label in the i-th row and j-th column is represented by the image feature V, which is a visual feature extracted by a feature pyramid network connected to a multi-layer convolutional neural network. The modal feature M is used to identify whether the input feature belongs to the text, image or cell modality.

[0011] In step S2, each cell information in the first table data and the second table data is represented in the form of a triplet, that is... Where T′ represents the text content of the cell, L′ represents the cell's position information, and M... ij Represents the row and column number of the cell.

[0012] The consistency check is specifically as follows: For each target cell, the cell text content T1′ from the first table data is compared with the text content T2′ extracted from the corresponding area of ​​the second coordinate information. When T1′=T2′, it is determined that the parsing is correct; when T1′ and T2′ cannot match, the corresponding position information L1′ and L2′ are compared, and the difference between height1 and height2 is calculated, where height1 is the height of the area defined by the first coordinate information, and height2 is the height of the area defined by the second coordinate information.

[0013] A hybrid strategy-based system for detecting the table structure and recognizing content in announcement PDFs, wherein the system comprises: The data extraction module is used to input the announcement PDF document to be processed into a rule-driven table recognition tool and a deep learning-based cell region prediction model, respectively, to generate first table data and second table data. The first table data contains the first text content and first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. The data correction module is used to align and match the first table data and the second table data based on the second coordinate information; for each target cell position, the corresponding first text content is checked for consistency with the text content extracted from the area corresponding to the second coordinate information in the announcement PDF document; when the check fails, the geometric difference between the first coordinate information and the second coordinate information is calculated; if the geometric difference exceeds a preset threshold, the adjacent row cells are merged or the current cell boundary is re-divided to generate corrected table data. The model building and optimization module is used to build a document understanding pre-trained model, whose input feature vector integrates text features, image features, layout features, and cell structure features; and is configured to use the corrected table data to train or update the parameters of the document understanding pre-trained model. The information recognition module is used to input the announcement PDF document to be recognized into the document understanding pre-trained model optimized by the model building and optimization module. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

[0014] In summary, the method of this application targets the detection of table structure and content recognition in PDFs of listed company announcements. It introduces deep learning pre-training technology and integrates it with traditional rule engines to improve extraction accuracy, achieving more efficient, accurate and flexible extraction of semi-structured table information. Compared with traditional single or rule-based PDF table extraction tools, the hybrid strategy tool has a significant improvement in accuracy. Attached Figure Description

[0015] Figure 1 A table illustration from a PDF of a listed company's announcement of a share reduction plan. Figure 2 for Figure 1 A schematic diagram of the results after Camelot recognition; Figure 3 This is a flowchart of the method in this application. Detailed Implementation

[0016] The present application will be further described below with reference to the accompanying drawings. The structure and principle of the present application are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0017] An empirical study of table extraction results under the NER task revealed that traditional rule-based methods or open-source deep learning methods often have insufficient accuracy in identifying table cell regions (cells containing field attributes and field value pairs), resulting in a low final semantic information extraction accuracy.

[0018] Taking the table in the PDF of the share reduction plan announcement disclosed by the listed company as an example ( Figure 1 The results after Camelot recognition are as follows: Figure 2 Errors such as text splicing and line breaks in cells are found in sections like "Shareholder Name," "Planned Reduction Quantity (Shares)," and "Period of Competitive Bidding for Reduction," as shown in the red boxes. Furthermore, announcements from listed companies also exhibit issues such as varied table formats, nested tables, and tables spanning multiple pages. Introducing a semantic recognition algorithm based on cell structure can correct problems in complex table extraction scenarios involving cell merging and splitting.

[0019] Step 1: Use tools such as Camelot to identify tables in specific types of listed company announcements. The output is a table information file containing cell text and cell coordinates. Further analysis revealed that the cell range in the Camelot results was inaccurate, specifically manifested as missed cells, and cells with recognition ranges that were too large or too small.

[0020] Step 2: Predict cell region coordinates using the StrucTexT-Cell algorithm. StrucTexT is an open-source intelligent document understanding deep learning model developed and released by Baidu, covering both text and images. Based on the StrucTexT algorithm, cell features are introduced to enhance the detection and recognition of table cells.

[0021] Step 3: The cell information obtained using Camelot and the cell information predicted using StrucTexT-Cell can both be written as a triplet.

[0022] Step 4: Input the results into the LayoutLMv3-Cell model for training. This model adds cell features to the LayoutLMv3 model, and considering that our cells are not all the same size, it adds text, location, and association information to the cell information. By masking the positions of some cells, the model predicts which region in the entire image these cells belong to.

[0023] The input features of LayoutLMv3-Cell include text features, 1D layout features, 2D layout features, image features, and our newly added cell features. Ultimately, the output contextual representation can be used for the following specific task layers.

[0024] Step 5: Perform downstream SER tasks such as table extraction, and obtain information such as table title, type, data, and relation by category.

[0025] Specifically, this application provides a method for detecting the table structure and recognizing content in announcement PDFs based on a hybrid strategy, including the following steps: S1: Input the PDF document of the announcement to be processed into the rule-driven table recognition tool and the deep learning-based cell region prediction model respectively to generate the first table data and the second table data. The first table data contains the first text content and the first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. S2: Using the second coordinate information as a reference, align and match the first table data with the second table data; for each target cell position, perform consistency verification between the corresponding first text content and the second text content extracted from the area corresponding to the second coordinate information in the announcement PDF document. If the verification fails, calculate the geometric difference between the first coordinate information and the second coordinate information. If the geometric difference exceeds the preset threshold, perform the merging operation of adjacent row cells to generate corrected table data. S3: Construct a document understanding pre-trained model whose input feature vectors fuse at least text features, image features, layout features, and cell features. Use corrected tabular data to train or update the parameters of the document understanding pre-trained model. S4: Input the PDF document of the announcement to be recognized into the document understanding pre-trained model optimized in step S3. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

[0026] In this embodiment of the application, the announcement PDF document is a financial announcement PDF document issued by a listed company that includes a share reduction plan and financial data information. The training of the model in S3 includes a mask cell position prediction task to learn the correspondence between cells and document image regions. The information output by the model in S4 further includes the semantic understanding result of the table structure. The semantic understanding result includes at least one or more of the following: table title, data type, and the relationship between data items.

[0027] S2 includes: The geometric difference is specifically the difference between the height of the area defined by the first coordinate information and the height of the area defined by the second coordinate information of the same target cell.

[0028] If the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is greater than the height of the area defined by the second coordinate information, then it is determined that the first table data identifies the cell area too large, and a merging operation is performed on adjacent row cells to integrate the content that should belong to the same cell.

[0029] Meanwhile, in an optional embodiment, the following technical means can also be used: if the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is less than the height of the area defined by the second coordinate information, it is determined that the first table data has too small an identification of the cell area, and a re-division operation of the current cell boundary is performed, using the second coordinate information as the cell boundary.

[0030] The input feature vector of the deep learning-based cell region prediction model is represented as follows: Where T represents text features, V represents image features, C represents cell features, S represents segment features, P represents paragraph length features, and M represents modal features. This is equivalent to adding cell features to the StrucTexT framework.

[0031] The formula for constructing text feature T is as follows: Among them, Emb t (S) represents the feature obtained by embedding and encoding the input sequence S, and L represents the layout feature obtained by embedding and encoding the coordinate information of words or characters. The formula for constructing the layout feature L is as follows: (x0, y0) and (x1, y1) represent the top-left and bottom-right coordinates of the word or character bounding box, respectively, and w and h represent the width and height of the bounding box, respectively. The formula for constructing the cell feature C is as follows: Among them, Emb c (C) indicates that cell features are embedded and encoded, and L represents layout features. E i P represents the center code of the cell. i Indicates the coordinate code of the cell, O i The edge code represents the cell's edge code, which is used to characterize the connection relationship between cells. The sequence of the paragraph feature S is represented as follows: [CLS] and [SEP] are special markers, c j i The cell content label in the i-th row and j-th column is represented by the image feature V, which is a visual feature extracted by a feature pyramid network connected to a multi-layer convolutional neural network. The modal feature M is used to identify whether the input feature belongs to the text, image or cell modality.

[0032] In step S2, each cell information in the first table data and the second table data is represented in the form of a triplet, that is... Where T′ represents the text content of the cell, L′ represents the cell's position information, and M... ij Represents the row and column number of the cell.

[0033] The consistency check is specifically as follows: For each target cell, the cell text content T1′ from the first table data is compared with the text content T2′ extracted from the corresponding area of ​​the second coordinate information. When T1′=T2′, it is determined that the parsing is correct; when T1′ and T2′ cannot match, the corresponding position information L1′ and L2′ are compared, and the difference between height1 and height2 is calculated, where height1 is the height of the area defined by the first coordinate information, and height2 is the height of the area defined by the second coordinate information.

[0034] For example, the correction process starts from the first row of the table. , These represent the text of the first line in the Camelot and StrucTexT-Cell results, respectively. If the first line is correctly parsed, its content is stored in the result; otherwise... and If a match cannot be found, it may be because the cell range is too large or too small, requiring further comparison. and ,check If the value is less than a threshold, then adjacent rows (the second row) are merged. This ultimately yields more accurate cell and content information.

[0035] This application also provides a hybrid strategy-based system for detecting and recognizing the structure and content of announcement PDF tables. Employing the aforementioned hybrid strategy-based method for detecting and recognizing the structure and content of announcement PDF tables, the system includes: The data extraction module is used to input the announcement PDF document to be processed into a rule-driven table recognition tool and a deep learning-based cell region prediction model, respectively, to generate first table data and second table data. The first table data contains the first text content and first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. The data correction module is used to align and match the first table data and the second table data based on the second coordinate information; for each target cell position, the corresponding first text content is checked for consistency with the text content extracted from the area corresponding to the second coordinate information in the announcement PDF document; when the check fails, the geometric difference between the first coordinate information and the second coordinate information is calculated; if the geometric difference exceeds a preset threshold, the adjacent row cells are merged or the current cell boundary is re-divided to generate corrected table data. The model building and optimization module is used to build a document understanding pre-trained model, whose input feature vector integrates text features, image features, layout features, and cell structure features; and is configured to use the corrected table data to train or update the parameters of the document understanding pre-trained model. The information recognition module is used to input the announcement PDF document to be recognized into the document understanding pre-trained model optimized by the model building and optimization module. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

Claims

1. A method for detecting the table structure and recognizing content in announcement PDFs based on a hybrid strategy, characterized in that, Includes the following steps: S1: Input the PDF document of the announcement to be processed into the rule-driven table recognition tool and the deep learning-based cell region prediction model respectively to generate the first table data and the second table data. The first table data contains the first text content and the first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. S2: Using the second coordinate information as a reference, align and match the first table data with the second table data; for each target cell position, perform consistency verification between the corresponding first text content and the second text content extracted from the area corresponding to the second coordinate information in the announcement PDF document. If the verification fails, calculate the geometric difference between the first coordinate information and the second coordinate information. If the geometric difference exceeds the preset threshold, perform the merging operation of adjacent row cells to generate corrected table data. S3: Construct a document understanding pre-trained model whose input feature vectors fuse at least text features, image features, layout features, and cell features. Use corrected tabular data to train or update the parameters of the document understanding pre-trained model. S4: Input the PDF document of the announcement to be recognized into the document understanding pre-trained model optimized in step S3. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

2. The method according to claim 1, characterized in that, The announcement PDF document is a financial announcement PDF document released by a listed company that contains share reduction plans and financial data information. The training of the model in S3 includes a mask cell position prediction task to learn the correspondence between cells and document image regions. The information output by the model in S4 further includes the semantic understanding results of the table structure. The semantic understanding results include at least one or more of the following: table title, data type, and the relationship between data items.

3. The method according to claim 1, characterized in that, S2 include: The geometric difference is specifically the difference between the height of the area defined by the first coordinate information and the height of the area defined by the second coordinate information of the same target cell.

4. The method according to claim 3, characterized in that, S2 include: If the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is greater than the height of the area defined by the second coordinate information, then it is determined that the first table data identifies the cell area too large, and a merging operation is performed on adjacent row cells to integrate the content that should belong to the same cell.

5. The method according to claim 3, characterized in that, S2 include: If the difference is greater than the preset threshold, and the height of the area defined by the first coordinate information is less than the height of the area defined by the second coordinate information, it is determined that the first table data has too small an identification of the cell area, and a re-division operation is performed on the current cell boundary, using the second coordinate information as the cell boundary.

6. The method according to claim 1, characterized in that, The input feature vector of the deep learning-based cell region prediction model is represented as follows: Where T represents text features, V represents image features, C represents cell features, S represents segment features, P represents paragraph length features, and M represents modal features.

7. The method according to claim 6, characterized in that, The formula for constructing text feature T is as follows: Among them, Emb t (S) represents the feature obtained by embedding and encoding the input sequence S, and L represents the layout feature obtained by embedding and encoding the coordinate information of words or characters. The formula for constructing the layout feature L is as follows: (x0, y0) and (x1, y1) represent the top-left and bottom-right coordinates of the word or character bounding box, respectively, and w and h represent the width and height of the bounding box, respectively. The formula for constructing the cell feature C is as follows: Among them, Emb c (C) indicates that cell features are embedded and encoded, and L represents layout features. E i P represents the center code of the cell. i Indicates the coordinate code of the cell, O i The edge code represents the cell's edge code, which is used to characterize the connection relationship between cells. The sequence of the paragraph feature S is represented as follows: [CLS] and [SEP] are special markers, c j i The cell content label in the i-th row and j-th column is represented by the image feature V, which is a visual feature extracted by a feature pyramid network connected to a multi-layer convolutional neural network. The modal feature M is used to identify whether the input feature belongs to the text, image or cell modality.

8. The method according to claim 1, characterized in that, In step S2, each cell information in the first table data and the second table data is represented in the form of a triplet, that is... Where T′ represents the text content of the cell, L′ represents the cell's position information, and M... ij Represents the row and column number of the cell.

9. The method according to claim 8, characterized in that, The consistency check is specifically as follows: For each target cell, the cell text content T1′ from the first table data is compared with the text content T2′ extracted from the corresponding area of ​​the second coordinate information. When T1′=T2′, it is determined that the parsing is correct; when T1′ and T2′ cannot match, the corresponding position information L1′ and L2′ are compared, and the difference between height1 and height2 is calculated, where height1 is the height of the area defined by the first coordinate information, and height2 is the height of the area defined by the second coordinate information.

10. A system for detecting the table structure and recognizing content in announcement PDFs based on a hybrid strategy, characterized in that, The system employs the hybrid strategy-based announcement PDF table structure detection and content recognition method as described in any one of claims 1 to 9, wherein the system comprises: The data extraction module is used to input the announcement PDF document to be processed into a rule-driven table recognition tool and a deep learning-based cell region prediction model, respectively, to generate first table data and second table data. The first table data contains the first text content and first coordinate information of each cell, and the second table data contains the second coordinate information of each cell. The data correction module is used to align and match the first table data and the second table data based on the second coordinate information; for each target cell position, the corresponding first text content is checked for consistency with the text content extracted from the area corresponding to the second coordinate information in the announcement PDF document; when the check fails, the geometric difference between the first coordinate information and the second coordinate information is calculated; if the geometric difference exceeds a preset threshold, the adjacent row cells are merged or the current cell boundary is re-divided to generate corrected table data. The model building and optimization module is used to build a document understanding pre-trained model, whose input feature vector integrates text features, image features, layout features, and cell structure features; and is configured to use the corrected table data to train or update the parameters of the document understanding pre-trained model. The information recognition module is used to input the announcement PDF document to be recognized into the document understanding pre-trained model optimized by the model building and optimization module. The model directly parses and outputs the final text content of each cell in the table and its row and column position relationship.

Images