A deep learning-based PDF document intelligent recognition and content extraction method

Abstract

The application discloses a kind of based on deep learning's PDF document intelligent identification and content extraction method, it is related to artificial intelligence, deep learning, computer vision and document image processing technical field, including: obtaining the positioning table area of each table in PDF whole page image;Obtain basic grid structure;Obtain cell with cross row or cross column structure;With the consistency detection and repair of cell with cross row or cross column structure using structure checking network, obtain the table structure after repair;Text recognition is carried out to each logical cell in the table structure after repair, and the row and column position information corresponding to each logical cell is bound, to obtain the table content that can be output as pre-set structured format.The application can effectively process scanned copy, picture and other various forms of PDF table, adapt to different table style, font and background, reduce the requirement to input image quality, ensure high-precision table recognition and content extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of artificial intelligence, deep learning, computer vision and document image processing, in particular to a PDF document intelligent recognition and content extraction method based on deep learning. BACKGROUND

[0002] PDF documents are widely used in information storage and transmission due to their cross-platform and fixed format advantages. However, the tables in PDF documents, especially complex tables in the form of scanned documents or images, are often difficult to extract structured data directly. Traditional methods rely on heuristic rules or image edge detection, and the processing process has strong dependence on the integrity of table lines, the consistency of styles, the rules of cell merging and splitting, and the direction of text.

[0003] When facing actual situations such as missing table lines, complex structures, and various formats, the recognition accuracy is often low and the robustness is poor. This not only limits the degree of automation, but also forces a large amount of table information to rely on manual input, which is not only inefficient but also prone to errors, seriously hindering the efficient circulation and application of structured data in business processes.

[0004] Currently, there is no effective solution to the problems in the related art. SUMMARY

[0005] To overcome the above technical problems existing in the prior art, the present application proposes a PDF document intelligent recognition and content extraction method based on deep learning.

[0006] To this end, the specific technical solutions adopted by the present application are as follows:

[0007] A PDF document intelligent recognition and content extraction method based on deep learning, comprising:

[0008] using a table detection network to recognize the table area of the input PDF whole page image, and obtaining the positioning table area of each table in the PDF whole page image;

[0009] performing row and column structure analysis on the positioning table area, and using a cell merging network to predict the segmentation line of the row and column structure to obtain a basic grid structure;

[0010] based on the basic grid structure, using a cell merging network to predict the merging of adjacent cells to obtain cells with cross-row or cross-column structure;

[0011] using a structure verification network to detect and repair the consistency of the cells with cross-row or cross-column structure to obtain the repaired table structure;

[0012] Text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound to obtain table content that can be output in a preset structured format.

[0013] Further, the table detection network is used to perform table region identification on the input PDF whole-page image to obtain the positioning table region of each table in the PDF whole-page image, including:

[0014] The table detection network is used to process the input PDF whole-page image, and multi-level visual features of the PDF whole-page image are extracted and fused through a backbone network, a neck network and a head network to obtain a feature map in the PDF whole-page image;

[0015] Based on the feature map, table bounding box prediction is performed on the PDF whole-page image to obtain a bounding box of a candidate table region as a preliminary estimation of the table position;

[0016] Based on the smooth adaptive intersection over union loss function, the bounding box of the candidate table region is optimized to obtain a target table positioning box;

[0017] Based on the target table positioning box, the PDF whole-page image is subjected to region cutting processing to obtain the positioning table region of each table in the PDF whole-page image.

[0018] Further, based on the feature map, table bounding box prediction is performed on the PDF whole-page image to obtain a bounding box of a candidate table region as a preliminary estimation of the table position, including:

[0019] A scale confidence prediction branch is introduced in the table detection network output, and the scale confidence is jointly trained using the smooth adaptive intersection over union loss function to obtain a scale confidence score;

[0020] Based on the scale confidence score, a dynamic threshold function is constructed, and detection result screening is performed on different types of tables;

[0021] The dynamic threshold function is used to adaptively judge the predicted bounding box to obtain the bounding box of the candidate table region as the preliminary estimation of the table position.

[0022] Further, based on the smooth adaptive intersection over union loss function, the bounding box of the candidate table region is optimized to obtain a target table positioning box, including:

[0023] The smooth adaptive intersection over union loss function is used to punish the distance between the predicted box and the center point of the real box to obtain an initial optimization direction;

[0024] Based on the initial optimization direction, the distance between the predicted box and the corner point of the real box is adjusted to obtain an optimized prediction box that conforms to the actual boundary of the table.

[0025] The optimized prediction frame is comprehensively measured by using a smooth adaptive intersection over union loss function, and a target table positioning frame is obtained.

[0026] Further, the row and column structure inside the positioned table region is analyzed, and a cell merging network is used to predict the segmentation line of the row and column structure to obtain a basic grid structure including:

[0027] A convolutional neural network is used to extract visual features of the table region image to obtain a two-dimensional feature map containing table row and column layout information.

[0028] The two-dimensional feature map is subjected to horizontal and vertical pooling operations to obtain a one-dimensional feature sequence representing row and column information.

[0029] A plurality of preset structure encoders are used to analyze the global context of the one-dimensional feature sequence to obtain a structure representation sequence containing global information.

[0030] Based on the structure representation sequence, a binary classification prediction is made for each position to obtain a logical row and column segmentation line position, and a preliminary basic grid structure is constructed.

[0031] Further, based on the structure representation sequence, a binary classification prediction is made for each position to obtain a logical row and column segmentation line position, and a preliminary basic grid structure is constructed including:

[0032] Based on the obtained structure representation sequence, a binary classification prediction is made for each position to obtain a probability sequence representing the segmentation line confidence.

[0033] A cost graph is constructed using the probability sequence of the segmentation line confidence, with the segmentation points as nodes and the probabilities as the basis, to obtain a global cost structure.

[0034] A path transition cost function is defined using preset structure prior knowledge to obtain a complete path scoring mechanism.

[0035] Based on the constructed cost graph and path scoring mechanism, a dynamic programming algorithm is used to solve all candidate paths to obtain a minimum total cost path.

[0036] The preliminary segmentation line is globally optimized using the minimum total cost path to obtain a logically continuous and reasonably distributed logical row and column segmentation line position, and a preliminary basic grid structure is constructed.

[0037] Further, based on the basic grid structure, a cell merging network is used to merge adjacent cells to obtain a cell with a cross-row or cross-column structure including:

[0038] The visual features of each cell are extracted using a convolutional neural network, and the normalized coordinates in the convolutional neural network are combined with the text embedding to obtain a multi-dimensional feature vector;

[0039] Based on the multi-dimensional feature vector, all cells are arranged in row and column order into a sequence to obtain a cell feature sequence input into the cell merging network;

[0040] The self-attention mechanism of the cell merging network encoder is used to process the cell feature sequence, calculate the correlation between any two cells, and learn non-local merging logic;

[0041] Based on the feature representation output by the cell merging network encoder, the classification head is used to predict the merging of each cell with its right or lower cell, and the cell with a cross-row or cross-column structure is obtained.

[0042] Further, the structure verification network is used to detect and repair the consistency of the cell with the cross-row or cross-column structure, and the repaired table structure includes:

[0043] The structure verification network is used to check the merged cell structure, and the cell combination with potential logical conflicts is obtained;

[0044] Based on the logical conflict area output by the structure verification network, editing actions are performed on the logical conflict area to obtain a target structure rationality reward;

[0045] The target structure rationality reward is used to optimize the strategy of all editing actions, and the repaired table structure with logical continuity and accurate cross-row and cross-column information is obtained.

[0046] Further, text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound to obtain a table content that can be output in a preset structured format, including:

[0047] Each logical cell image slice in the repaired table structure is classified to obtain a class label;

[0048] The class label is used to call the corresponding optical character recognition strategy for the logical cell image to obtain the target extracted logical cell text content information;

[0049] Based on the logical cell text content information and the repaired table structure, and binding the row and column positions and merging information of each logical cell, a table content that can be output in a preset structured format is obtained.

[0050] Further, the class label is used to call the corresponding optical character recognition strategy for the logical cell image to obtain the target extracted logical cell text content information, including:

[0051] Based on the final table structure, optical character recognition is performed on each logical cell to obtain preliminary logical cell text content;

[0052] Using the preliminary logical cell text content, the logical cells across columns in the final table structure are extracted, triggering a consistency checking mechanism;

[0053] Based on the semantic weighted edit distance algorithm, the header logical cell and its underlying sub-logical cell content are compared for consistency to obtain a semantic consistency score result;

[0054] Using the semantic consistency score result, the recognized text is judged and corrected to obtain the target extracted logical cell text content information.

[0055] The beneficial effects of the present application are:

[0056] 1. The present application combines the feature learning ability of the deep learning cell merging network, significantly improves the accuracy of table region detection, structure analysis and content extraction, especially in complex tables with incomplete table lines and merged / split cells, showing stronger robustness. Thus, it can effectively process various forms of PDF tables such as scanned documents and images, adapt to different table styles, fonts and backgrounds, reduce the requirements for input image quality, and ensure high-precision table recognition and content extraction.

[0057] 2. The present application realizes the full automation process from table recognition to content extraction, greatly reduces manual intervention and improves work efficiency. Thus, it is suitable for financial statements, contracts, invoices, questionnaires and other PDF documents containing complex tables. Its wide applicability and automation characteristics not only provide great convenience for data analysis and information management, but also greatly improve the intelligent level of industry workflow, reduce manual operation errors and time cost.

[0058] 3. The present application converts the cell merging problem into a classification task based on global context, uses the self-attention mechanism of Transformer for intelligent reasoning, and gets rid of the dependence on physical lines. Thus, the cell merging network can consider global information such as table layout, column division line, header content, etc., to accurately determine the position of merged cells. Through this strategy of first structure and then classification, fine content recognition can be achieved, greatly improving the efficiency and accuracy of table data extraction. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and other accompanying drawings can also be obtained by those skilled in the art without any creative effort on the basis of these accompanying drawings.

[0060] Figure 1 is a flowchart of a deep learning-based PDF document intelligent recognition and content extraction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to further illustrate the embodiments, the present application provides accompanying drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. Those skilled in the art should understand other possible embodiments and advantages of the present application by referring to these contents.

[0062] According to an embodiment of the present application, a deep learning-based PDF document intelligent recognition and content extraction method is provided.

[0063] The present application will be further described in conjunction with the accompanying drawings and specific embodiments. As shown in the drawings, the deep learning-based PDF document intelligent recognition and content extraction method according to an embodiment of the present application comprises: Figure 1

[0064] S1, using a table detection network (YOLOv8 plus SAIoU Loss) to recognize the table area of the input PDF whole page image, and obtaining the positioning table area of each table in the PDF whole page image;

[0065] Specifically, high-precision table positioning: through a specially optimized target detection network (i.e. table detection network), all the positions of the tables in the input whole page image are accurately recognized and framed.

[0066] Specifically, the table detection network (YOLOv8 plus SAIoU Loss): YOLOv8 is selected as the basic framework, but its innovation lies in not using the conventional regression loss function, but using the SAIoU Loss loss function for training. This enables the network to achieve intelligent optimization of fast movement first and fine tuning later when positioning the table boundary, with much higher accuracy than conventional methods.

[0067] Specifically, YOLOv8 is used as the basic target detector. YOLOv8 is a typical one-stage detector, and its structure contains three parts:

[0068] ​1) Backbone (backbone network): responsible for extracting visual features at different levels from input images.

[0069] 2) Neck (neck network): for example, FPN (feature pyramid network), responsible for fusing multi-scale features output by the backbone network to better detect targets of different sizes.

[0070] 3) Head (head network): final classification and bounding box regression prediction on the fused feature map.

[0071] It should be explained that the conventional target detection cell merging network (such as standard YOLO) usually uses general loss functions such as SmoothL1, GIoU, DIoU, etc. for bounding box regression during training. Although these loss functions are effective, they have the problems of slow convergence and insensitivity to boundary accuracy for targets with variable shapes and large aspect ratios such as tables. For example, GIoU can only provide effective movement gradient when the predicted box and the real box have no overlap, which is very common at the beginning of training, resulting in low training efficiency. The present application does not use the above general loss function, but introduces a smooth adaptive intersection over union loss function (SAIoU Loss) to guide network training.

[0072] SAIoU Loss is used in the training process of the table detection network, and its working principle is to dynamically and stage-by-stage optimize the positioning of the bounding box, achieving intelligent training of fast positioning first and fine adjustment later. The specific steps are as follows:

[0073] Early training (when the predicted box is far away from the real box):

[0074] At this stage, SAIoU Loss focuses on calculating and punishing the center point distance between the predicted box and the real box.

[0075] This metric provides a clear and smooth gradient for the cell merging network, guiding the predicted box to quickly converge to the real table area, solving the problem of gradient disappearance when the two boxes do not overlap in traditional IoU loss.

[0076] Late training (when the predicted box has good overlap with the real box):

[0077] At this stage, the focus of SAIoU Loss shifts to minimizing the distance between the corresponding corner points.

[0078] Through fine adjustment of the corner points, the size and shape of the predicted box are fine-tuned to better fit the actual boundary of the table, which is crucial for preventing rows and columns from being incorrectly cut.

[0079] Throughout the training process:

[0080] As the most basic metric, Intersection over Union (IoU) is always part of the loss function, focusing on the overlapping area between the predicted box and the true table region.

[0081] By adaptively combining and adjusting the focus of these three geometric metrics during training, SAIoULoss guides the cell merging network to efficiently and accurately complete the table localization task.

[0082] The working principle of SAIoULoss: This loss function cleverly combines three different geometric metrics and can adaptively adjust the focus during training.

[0083] Intersection over Union (IoU): Focuses on the overlapping area between the predicted box and the true table region, which is the most basic metric.

[0084] Center point distance: In the early stage of training, when the predicted box is far from the true box, this term penalizes the distance between the centers of the two, providing a clear and smooth gradient for the cell merging network to guide the predicted box to quickly converge to the correct position, solving the problem of gradient disappearance when traditional IoU loss is not overlapping.

[0085] Corner point distance: In the later stage of training, when the predicted box has good overlap with the true box, this term minimizes the distance between the corresponding corner points of the two, fine-tuning the size and shape of the predicted box to better fit the actual boundaries of the table. This is crucial to ensure that the rows and columns of the table are not incorrectly cut.

[0086] S2, analyze the row and column structure inside the located table region, and use the cell merging network to predict the segmentation lines of the row and column structure to obtain the basic grid structure;

[0087] Specifically, row and column segmentation: For each located table region, use a Transformer-based cell merging network to intelligently predict the positions of all logical row and column segmentation lines, thereby dividing the table into a basic grid.

[0088] S3, based on the basic grid structure, use the cell merging network to predict the merging of adjacent cells to obtain cells with cross-row or cross-column structure;

[0089] Specifically, cell merging: Again, use a Transformer cell merging network (i.e., a Transformer-based cell merging network) to globally analyze all cells in the basic grid and predict which adjacent cells should be logically merged into a cross-row or cross-column cell.

[0090] S4, utilize the structure verification network to detect and repair the cells with cross-row or cross-column structure for consistency, and obtain the repaired table structure;

[0091] Specifically, structure verification: in order to ensure the absolute logical correctness of the final structure, an intelligent verification agent based on reinforcement learning will review and fine-tune the merged structure, and automatically repair potential logical conflicts.

[0092] S5, text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound, and the table content in a preset structured format (such as HTML structured format) is obtained.

[0093] Specifically, content extraction and structured output: on the completely determined table structure, fine-grained content recognition is performed on each logical cell, and the recognized text and its corresponding row and column position and merging information are bound, and finally output in HTML structured format. Output HTML or JSON file containing complete structure and accurate content, input a PDF document.

[0094] In this optional embodiment, a table detection network is used to identify the table area in the input PDF whole-page image, and the positioning table area of each table in the PDF whole-page image is obtained, which includes:

[0095] The table detection network is used to process the input PDF whole-page image, and the multi-level visual features of the PDF whole-page image are extracted and fused through the backbone network, neck network and head network to obtain the feature map in the PDF whole-page image;

[0096] Based on the feature map, the table boundary box of the PDF whole-page image is predicted to obtain the boundary box of the candidate table area as a preliminary estimate of the table position;

[0097] Based on the smooth adaptive intersection over union loss function (SAIoU Loss loss function), the boundary box of the candidate table area is optimized to obtain the target table positioning box;

[0098] Based on the obtained target table positioning box, the region cutting processing is performed on the PDF whole-page image to obtain the positioning table area of each table in the PDF whole-page image.

[0099] Specifically, the document picture is received as input, and the YOLOv8 network processes the whole-page image by extracting and fusing features through its backbone network (Backbone), neck network (Neck) and head network (Head). In the training process, the SAIoULoss loss function will guide the network to perform intelligent optimization of fast movement first and fine-tuning later, and finally output the accurate boundary box of the position of all tables in the head network.

[0100] In this optional embodiment, based on the feature map, the table boundary box prediction is performed on the PDF whole-page image to obtain the boundary box of the candidate table region as the preliminary estimation of the table position, including:

[0101] A scale confidence prediction branch is introduced in the table detection network output, and a smooth adaptive intersection over union loss function is used to jointly train the scale confidence to obtain a scale confidence score;

[0102] Based on the obtained scale confidence and classification confidence, a dynamic threshold function is constructed, and the detection results of different types of tables are screened;

[0103] The adaptive threshold function is used to adaptively judge the predicted boundary box to obtain the boundary box of the candidate table region as the preliminary estimation of the table position.

[0104] Specifically, the present application introduces a scale-confidence adaptive thresholding (SCAT) mechanism coupled with network training to enhance the stability and accuracy of table detection of different scales and different clarity. The specific implementation is as follows:

[0105] 1) Network output enhancement: modify the head of the table detection network (YOLOv8) to output an additional scale confidence (Scale Confidence Score) while predicting the bounding box (Bounding Box) and classification confidence (Classification Confidence). This score represents the network's confidence in the scale and clarity of the target (i.e., table) within the current prediction box. The learning of this value is guided by SAIoU Loss.

[0106] Adaptive threshold decision: in the inference stage, instead of using a fixed threshold (e.g., 0.5) to filter the detection results as in traditional methods. The present application uses a dynamic threshold function:

[0107] .

[0108] where, S scale represents the scale confidence of the network output; C classrepresents the classification confidence; the core idea of this dynamic threshold function is: when the network predicts a high scale confidence (i.e. the network thinks this is a large size, clear feature table), the final judgment threshold will be automatically raised. This can effectively filter out non-table noise in the background, prevent false detection in large table areas, and achieve the best of the best. When the network predicts a low scale confidence (i.e. the network thinks this is a small size or fuzzy table), the judgment threshold will be automatically lowered. This can save real tables with low classification confidence due to unclear features, and greatly improve the recall rate of small targets and low-quality images.

[0109] In this optional embodiment, based on the smooth adaptive intersection over union loss function, the bounding box of the candidate table region is optimized to obtain the target table positioning box, including:

[0110] The smooth adaptive intersection over union loss function is used to punish the distance between the predicted box and the real box center point, and an initial optimization direction is obtained;

[0111] Based on the obtained initial optimization direction, the distance between the predicted box and the real box corner point is adjusted to obtain an optimized prediction box that conforms to the actual boundary of the table;

[0112] The smooth adaptive intersection over union loss function is used to comprehensively measure the optimized prediction box to obtain the target table positioning box.

[0113] In this optional embodiment, the row and column structure analysis is performed on the positioned table region, and the cell merging network is used to predict the segmentation line of the row and column structure to obtain the basic grid structure, including:

[0114] The convolutional neural network is used to extract visual features of the table region image to obtain a two-dimensional feature map containing table row and column layout information;

[0115] The two-dimensional feature map is subjected to horizontal and vertical direction pooling operations to obtain a one-dimensional feature sequence representing row and column information;

[0116] A plurality of preset structure encoders (i.e. Transformer encoders) are used to perform global context analysis on the row and column one-dimensional feature sequence to obtain a structure representation sequence containing global information;

[0117] Based on the structure representation sequence, a binary classification prediction is performed on each position to obtain the logical row and column segmentation line position, and the preliminary basic grid structure is constructed.

[0118] Specifically, each table region image is input into a row-column segmentation network (CNN plus Transformer). The row-column segmentation network first extracts a visual feature map through CNN (i.e., ResNet), then converts the feature map into two one-dimensional feature sequences (one representing rows and one representing columns), and finally inputs the two sequences into independent Transformer encoders for global context analysis, and finally predicts the positions of all logical row and column segmentation lines through a fully connected layer.

[0119] Specifically, the row-column segmentation network (CNN plus Transformer): The core of this network is to convert two-dimensional table image features into one-dimensional feature sequences through compression (i.e., Pooling), and then input them into a Transformer encoder. Unlike traditional CNNs that can only see local pixels, the global attention mechanism of the Transformer allows the cell merging network to see the entire table layout when determining whether a certain place is a segmentation line, thus making more accurate judgments on complex cases such as wireless tables and broken tables.

[0120] It should be explained that a standard convolutional neural network (i.e., ResNet) is used as the backbone network, which inputs the detected table image and outputs a two-dimensional visual feature map (i.e., Feature Map), cleverly converting the two-dimensional table recognition task into two independent one-dimensional sequence labeling problems.

[0121] Specifically, the implementation of two-dimensional to one-dimensional: a CNN (ResNet) is used to extract a two-dimensional visual feature map (i.e., Feature Map) from the table image. Then, the two-dimensional visual feature map is converted into two independent one-dimensional sequences through pooling operations.

[0122] For column segmentation: the two-dimensional feature map is globally averaged pooled in the horizontal direction to flatten it into a one-dimensional vertical feature sequence. Each element of this sequence condenses all the visual information of a vertical slice in the table image.

[0123] For row segmentation: similarly, the two-dimensional feature map is pooled in the vertical direction to obtain a one-dimensional horizontal feature sequence, each element of which represents the information of a horizontal slice.

[0124] Through the above implementation, the complex two-dimensional structure recognition problem is cleverly decomposed into two relatively simple one-dimensional sequence labeling problems.

[0125] Specifically, the pooling method: when processing column segmentation, global average pooling is used. When row segmentation is used, the same pooling method is used. Row segmentation uses the same pooling mechanism as column segmentation, that is, global average pooling is performed in the vertical direction.

[0126] Specifically, the role of the Transformer encoder: after inputting two one-dimensional feature sequences into the Transformer encoder, the encoder processes them using its core multi-head self-attention mechanism. Its specific role is to enable each element in the sequence (representing an image slice) to simultaneously pay attention to all other elements in the sequence when calculating its new representation, and assign different attention weights based on the correlation. The purpose of this process is to give the cell merging network a global receptive field, enabling the cell merging network to understand the context of each slice based on the layout of the entire table (such as table headers, borders, alignment, and other global information), rather than relying solely on local pixel information.

[0127] Specifically, the prediction method: after the Transformer encoder finishes processing, its output is connected to a simple fully connected layer (i.e., the classification head). The classification head performs binary classification prediction for each position in the sequence. The basis for determining whether a position is a segmentation point is the feature vector output by the Transformer encoder, which contains global context information. The document explicitly states that because the cell merging network has global reasoning capabilities, it considers global information such as alignment on the left side of the table, the border on the right side, and the layout of the table header when determining whether a certain position is a segmentation line. This global context-based judgment enables the invention to be independent of physical lines and exhibits high robustness to wireless tables and broken-line tables.

[0128] In this optional embodiment, based on the structure representation sequence, binary classification prediction is performed for each position to obtain the logical row and column segmentation line position, and a preliminary basic grid structure is constructed, including:

[0129] Based on the obtained structure representation sequence, binary classification prediction is performed for each position to obtain a probability sequence representing the segmentation line confidence;

[0130] A cost graph is constructed using the probability sequence of segmentation line confidence, with segmentation points as nodes and probabilities as the basis, to obtain a global cost structure;

[0131] A path transition cost function is defined using the pre-set structure prior knowledge to obtain a complete path scoring mechanism;

[0132] Based on the constructed cost graph and path scoring mechanism, a dynamic programming algorithm is used to solve all candidate paths to obtain the path with the minimum total cost;

[0133] The total cost minimum path is used to globally optimize the preliminary segmentation line to obtain a logically continuous and reasonably distributed logical row and column segmentation line position, and to realize the construction of a preliminary basic grid structure.

[0134] It should be explained that after the row and column segmentation based on the Transformer, the application introduces a global context guided dynamic programming (Global-Context Guided Dynamic Programming, GCG-DP) algorithm to optimize the segmentation line sequence output by the Transformer to ensure the absolute logical continuity of the segmentation line in an extremely irregular table, and the specific implementation is as follows:

[0135] 1) Obtain a global context probability sequence: first, according to the original scheme, process the table image by using the row and column segmentation network (CNN plus Transformer) to output a probability sequence representing whether each row and column position is a segmentation line. This sequence contains the global understanding of the table layout by the cell merging network.

[0136] 2) Build a segmentation path cost graph: convert the above probability sequence into a cost graph. Each node in the graph represents a potential segmentation point, and the cost is inversely proportional to the probability output by the Transformer (the higher the probability, the lower the cost).

[0137] 3) Dynamic programming to solve the optimal path: apply a dynamic programming algorithm to find a path with the lowest total cost in the cost graph. The core of this process is to design a transition cost function (TransitionCost Function) that includes structural prior knowledge.

[0138] 4) Continuity reward: give low transition cost or reward to continuous and smooth segmentation line paths to encourage line continuity.

[0139] 5) Jump penalty: impose a high transition cost on paths that appear to be logically discontinuous, jumping or sharp, thereby suppressing discontinuous segmentation results in decision-making.

[0140] 6) Distance constraint: according to the average row and column width of the table, paths with too small or too large distances between two segmentation lines are penalized to ensure the reasonableness of the segmentation.

[0141] In this optional embodiment, based on the basic grid structure, the cell merging network is used to merge adjacent cells to obtain a cell with a cross-row or cross-column structure, including:

[0142] The visual features of each cell are extracted using a convolutional neural network, and combined with normalized coordinates and text embeddings in the convolutional neural network to obtain a multi-dimensional feature vector.

[0143] Based on the multi-dimensional feature vector, all cells are arranged in row and column order into a sequence to obtain a cell feature sequence input to the cell merging network.

[0144] The self-attention mechanism of the cell merging network (Transformer) encoder is used to process the cell feature sequence, calculate the correlation between any two cells, and learn non-local merging logic.

[0145] Based on the feature representation output by the cell merging network encoder, a classification head is used to predict the merging of each cell with its right or lower cell, obtaining a cell with a cross-row or cross-column structure.

[0146] Specifically, based on the segmented basic grid, visual, position and preliminary text features are extracted for each cell, and the feature vectors of all cells are input into the Transformer encoder as a sequence. The cell merging network calculates the correlation between any two cells through the self-attention mechanism, and finally predicts which adjacent cells need to be logically merged by the classification head.

[0147] Specifically, the cell merging network (Transformer): This network inputs the features (visual, position, text) of all basic grid cells as a sequence into the Transformer encoder. The self-attention mechanism is used to calculate the correlation between any two cells, thereby learning complex merging rules, for example, the cell merging network can understand that two cells with similar content, consistent background color and in the same column should be merged.

[0148] It needs to be explained that the multi-dimensional features (visual, position, and preliminary recognized semantics) of all basic cells are input into the cell merging network as a sequence, and the self-attention mechanism of the Transformer calculates the correlation between any two cells, thereby learning complex merging rules, for example, the cell merging network can understand that two cells with similar content, consistent background color and in the same column should be merged, and a special classification head will predict whether each cell needs to be merged with its right or lower cell. After completing the row and column segmentation, a basic grid layout is obtained.

[0149] Specifically, for each segmented basic grid cell, a multi-dimensional feature vector is extracted, which is composed of three parts:

[0150] 1) Visual Feature: From the ResNet backbone network, extract the region feature according to the coordinates of the cell.

[0151] 2) Position Feature: The normalized row and column coordinates of the cell in the grid.

[0152] 3) Semantic Feature: Through a lightweight OCR engine, the content of the cell is preliminarily identified, and the text is converted into a word embedding vector.

[0153] Specifically, the merging relationship prediction: use a Transformer encoder cell merging network to predict whether adjacent grid cells should be merged. Arrange the feature vectors of all basic grid cells in the table in a sequence from top to bottom and left to right as the input of the Transformer. Similar to row and column segmentation, the self-attention mechanism of the cell merging network will calculate the correlation score between any two grid cells. At the output end of the Transformer, a classification head is designed. For each grid cell, the classification head will predict whether it needs to be merged with the cell to its right and whether it needs to be merged with the cell below it.

[0154] In this optional embodiment, a structure verification network (i.e. Reinforcement Learning Agent) is used to detect and repair the consistency of cells with cross-row or cross-column structures, and the repaired table structure includes:

[0155] Using the structure verification network to check the merged cell structure, the cell combination situation with potential logical conflicts is obtained;

[0156] Based on the logical conflict area output by the structure verification network, perform editing actions on the logical conflict area to obtain the target structure rationality reward;

[0157] Using the target structure rationality reward to optimize all editing actions, the repaired table structure with logical continuity and accurate cross-row and cross-column information is obtained.

[0158] Specifically, the structure verification network: this is an intelligent agent based on reinforcement learning. Its core is a small policy network (i.e. Policy Network, usually MLP), which learns how to correct the table structure through a series of editing actions (such as merging and splitting) to obtain the highest structure rationality reward. This is a dynamic, self-learning correction process, rather than an immutable rule.

[0159] In the optional embodiment, text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound to obtain table content that can be output in a preset structured format, including:

[0160] Each logical cell image slice in the repaired table structure is classified to obtain a category label.

[0161] The logical cell image is called using the category label to perform recognition using a corresponding optical character recognition strategy (i.e., an OCR strategy) to obtain target extracted logical cell text content information.

[0162] Based on the logical cell text content information and the repaired table structure, and binding the row and column positions and merging information of each logical cell, table content that can be output in a preset structured format is obtained.

[0163] Specifically, on the final table structure that has been verified, first, a small CNN is used to classify the content of each logical cell image slice (for example, pure text, pure numbers, blank, etc.). Then, according to the classification result, the corresponding OCR strategy is called to perform fine content recognition. The recognized text is bound to its corresponding row and column positions and merging information (i.e., rowspan, colspan), and is output as a structured file in HTML or JSON format.

[0164] Specifically, on the completely determined table structure, a small convolutional neural network (i.e., MobileNet) is first used to quickly classify each final logical cell image slice into predefined categories such as pure text, pure numbers, mixed text and graphics, or blank. According to the classification result, a differentiated recognition strategy is adopted. For example, an OCR cell merging network optimized for number recognition is called for regions classified as pure numbers. This way, computing resources are concentrated in the areas that need them the most, achieving adaptive and fine-grained recognition, avoiding the misidentification of image noise as characters, and improving efficiency. After fine-grained recognition of each cell, the final OCR result is bound to its logical position (row number, column number) and merging information after RL verification, and a standard, machine-readable HTML file structured data is generated.

[0165] In the optional embodiment, the logical cell image is called using the category label to perform recognition using a corresponding optical character recognition strategy to obtain target extracted logical cell text content information, including:

[0166] Based on the final table structure, optical character recognition is performed on each logical cell to obtain preliminary logical cell text content.

[0167] The logical cell text content is extracted from the final table structure, and a consistency checking mechanism is triggered;

[0168] Based on the semantic weighted edit distance algorithm (SWED algorithm), the consistency of the table header logical cell and the content of the sub-logical cell below is compared, and the semantic consistency score result is obtained.

[0169] The semantic consistency score result is used to judge and correct the recognized text, and the logical cell text content information of the target extraction is obtained.

[0170] It should be explained that the present application introduces a cross-cell content checking mechanism based on semantic weighted edit distance (SWED), which is used for logical verification of the recognition result of the merged cell, so as to improve the content extraction accuracy under the complex table header, and the specific implementation mode is as follows:

[0171] 1) Preliminary content extraction: complete table structure analysis and content OCR of each logical cell.

[0172] 2) Trigger checking mechanism: automatically identify the merged cell in the structure (especially the cross table as the table header).

[0173] 3) Cross-cell content checking: for a table header cell spanning multiple columns, the contents of multiple sub-column cells below will be logically consistent with the content of the table header cell. This checking is completed by the SWED algorithm.

[0174] 4) Semantic weighting: the traditional edit distance algorithm treats all characters equally. The SWED algorithm classifies characters by semantics (for example: Chinese characters, English letters, numbers, punctuation) before calculating the distance. When comparing, the replacement between different categories of characters will result in a very high cost. For example, when matching the Chinese character gold in the table header with the number 8 in the sub-column content, the cost is much higher than matching the amount with the amount.

[0175] 5) Consistency score: by calculating the SWED score between the table header text and the first row text of each sub-column, the system can judge whether the OCR recognized table header content is consistent with the data below in terms of semantic category. For example, the name table header should be text, and the quantity table header should be number.

[0176] In order to facilitate the understanding of the above technical solutions of the present application, the following will explain the PDF document intelligent recognition and content extraction based on deep learning in the actual process of the present application.

[0177] I. Input file: the scanned PDF standard file is used as the input image.

[0178] II. Table detection: First, all table regions in the table are accurately identified by the deep learning cell merging network, and are marked with bounding boxes.

[0179] III. Structure analysis: For each identified table region, its internal structure is further analyzed. For example, there may be a total cell that merges data from multiple months, or a detailed cell that splits multiple sub-items. The cell merging network of the present application can accurately identify these merged or split cells and restore the true logical structure of the table. Even if the table lines are incomplete or there are interfering lines, the cell merging network can accurately determine the boundaries of rows and columns through learned features.

[0180] IV. Content extraction: For each structured cell, an OCR engine is called to recognize the text and numerical content therein.

[0181] V. Data output: Finally, the recognized structured data is output in HTML format, which includes the row and column information of the table, the coordinates, content, and logical row and column of each cell, and even the cross-row and cross-column information of the merged cell.

[0182] In summary, with the above technical solutions of the present application, the present application realizes a fully automated process from table recognition to content extraction, greatly reducing manual intervention and improving work efficiency. It is suitable for financial statements, contracts, invoices, questionnaires, and other PDF documents containing complex tables. Its wide applicability and automation characteristics not only provide great convenience for data analysis and information management, but also greatly improve the intelligent level of industry workflow, reduce human error and time cost. The present application converts the cell merging problem into a classification task based on global context, uses the self-attention mechanism of Transformer for intelligent reasoning, and gets rid of the dependence on physical lines. Thus, the cell merging network can consider global information such as table layout, column division lines, and table header content to accurately determine the position of the merged cell. Through this strategy of first structure and then classification, fine content recognition can be achieved, greatly improving the efficiency and accuracy of table data extraction.

[0183] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based PDF document intelligent recognition and content extraction method, characterized in that, The method comprises the following steps: A table area in a PDF whole-page image is identified by using a table detection network to obtain a positioning table area of each table in the PDF whole-page image; Row and column structure analysis is performed on the positioning table area, and a cell merging network is used to predict the segmentation lines of the row and column structure to obtain a basic grid structure; Based on the basic grid structure, adjacent cells are merged by using the cell merging network to obtain cells with cross-row or cross-column structure; A consistency detection and repair is performed on the cells with cross-row or cross-column structure by using a structure verification network to obtain a repaired table structure; Text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound to obtain table content that can be output in a preset structured format; The row and column structure analysis on the positioning table area and the prediction of the segmentation lines of the row and column structure by using the cell merging network to obtain the basic grid structure comprise the following steps: Visual feature extraction is performed on the table area image by using a convolutional neural network to obtain a two-dimensional feature map containing table row and column layout information; A one-dimensional feature sequence representing row and column information is obtained by performing horizontal and vertical pooling operations on the two-dimensional feature map; Global context analysis is performed on the row and column one-dimensional feature sequence by using a plurality of preset structure encoders to obtain a structure representation sequence containing global information; Based on the structure representation sequence, a binary classification prediction is performed on each position to obtain logical row and column segmentation line positions, thereby realizing the construction of a preliminary basic grid structure, which comprises the following steps: Based on the obtained structure representation sequence, a binary classification prediction is performed on each position to obtain a probability sequence representing the confidence of the segmentation line; A cost graph is constructed by using the probability sequence of the segmentation line confidence, taking the segmentation points as nodes and the probabilities as the basis, to obtain a global cost structure; A path transition cost function is defined by using preset structure prior knowledge to obtain a complete path scoring mechanism; Based on the constructed cost graph and path scoring mechanism, a dynamic programming algorithm is used to solve all candidate paths to obtain a minimum total cost path; The preliminary segmentation lines are globally optimized by using the minimum total cost path to obtain logical row and column segmentation line positions with continuous structure and reasonable distribution, thereby realizing the construction of a preliminary basic grid structure.

2. The PDF document intelligent recognition and content extraction method based on deep learning according to claim 1, characterized in that, The table area in the input PDF whole-page image is identified by using the table detection network to obtain the positioning table area of each table in the PDF whole-page image, which comprises the following steps: The input PDF whole-page image is processed by using the table detection network, and multi-level visual features of the PDF whole-page image are extracted and fused by a backbone network, a neck network and a head network to obtain a feature map of the PDF whole-page image; Based on the feature map, a table bounding box prediction is performed on the PDF whole-page image to obtain a bounding box of a candidate table area as a preliminary estimation of the table position; Based on a smooth adaptive intersection over union loss function, the bounding box of the candidate table area is optimized to obtain a target table positioning box; Based on the target table positioning box, a region cutting process is performed on the PDF whole-page image to obtain the positioning table area of each table in the PDF whole-page image. 3.The PDF document intelligent recognition and content extraction method based on deep learning according to claim 2, characterized in that, The feature map-based table boundary box prediction on the PDF whole-page image obtains the boundary box of the candidate table region as the preliminary estimation of the table position, including: A scale confidence prediction branch is introduced into the table detection network output, and a smooth adaptive intersection over union loss function is used to jointly train the scale confidence to obtain a scale confidence score; Based on the scale confidence score, a dynamic threshold function is constructed, and the detection results of different types of tables are screened; The dynamic threshold function is used to adaptively judge the predicted boundary box to obtain the boundary box of the candidate table region as the preliminary estimation of the table position.

4. The PDF document intelligent recognition and content extraction method based on deep learning according to claim 2, characterized in that, The smooth adaptive intersection over union loss function is used to optimize the boundary box of the candidate table region to obtain the target table positioning box, including: The smooth adaptive intersection over union loss function is used to punish the distance between the predicted box and the real box center point to obtain an initial optimization direction; Based on the initial optimization direction, the distance between the predicted box and the real box corner point is adjusted to obtain an optimized prediction box that conforms to the actual boundary of the table; The smooth adaptive intersection over union loss function is used to comprehensively measure the optimized prediction box to obtain the target table positioning box.

5. The PDF document intelligent recognition and content extraction method based on deep learning according to claim 1, characterized in that, Based on the basic grid structure, the adjacent cells are merged by using a cell merging network to obtain cells with cross-row or cross-column structure, including: The visual features of each cell are extracted by using a convolutional neural network, and the normalized coordinates and text embedding in the convolutional neural network are combined to obtain a multi-dimensional feature vector; Based on the multi-dimensional feature vector, all cells are arranged in row and column order to obtain a cell feature sequence input to the cell merging network; The self-attention mechanism of the cell merging network encoder is used to process the cell feature sequence to calculate the correlation between any two cells and learn the non-local merging logic; Based on the feature representation output by the cell merging network encoder, each cell and its right or lower cell are merged by a classification head to obtain cells with cross-row or cross-column structure.

6. The PDF document intelligent recognition and content extraction method based on deep learning according to claim 1, characterized in that, The structure verification network is used to detect and repair the consistency of the cells with cross-row or cross-column structure to obtain the repaired table structure, including: The merged cell structure is checked by using the structure verification network to obtain the cell combination situation with potential logical conflicts; Based on the logical conflict area output by the structure verification network, editing actions are performed on the logical conflict area to obtain a target structure rationality reward; The target structure rationality reward is used to optimize the strategy of all editing actions to obtain a repaired table structure that is logically continuous and accurate in cross-row and cross-column information.

7. The PDF document intelligent recognition and content extraction method based on deep learning according to claim 1, characterized in that, Text recognition is performed on each logical cell in the repaired table structure, and the row and column position information corresponding to each logical cell is bound to obtain a table content that can be output in a preset structured format, including: Each logical cell image slice in the repaired table structure is classified to obtain a class label; The logical cell image is called using the class label to perform recognition using the corresponding optical character recognition strategy to obtain the target extracted logical cell text content information. Based on the logical cell text content information and the repaired table structure, and binding the row and column positions and the merging information of each logical cell, the table content in a preset structured format is obtained. 8.The method of claim 7, wherein, The logical cell image is called by using the category label to perform recognition by using a corresponding optical character recognition strategy, and logical cell text content information of target extraction is obtained. Based on the final table structure, optical character recognition is performed on each logical cell to obtain preliminary logical cell text content; By using the preliminary logical cell text content, logical cells crossing columns in the final table structure are extracted, and a consistency checking mechanism is triggered; Based on a semantic weighted edit distance algorithm, consistency comparison is performed on the table header logical cell and the content of the sub-logical cell below the table header logical cell, and a semantic consistency score result is obtained; By using the semantic consistency score result, the recognized text is judged and corrected, and logical cell text content information of target extraction is obtained.

Images