Document table extraction method and device, equipment and medium

Abstract

The invention discloses a document table extraction method and device, equipment and a medium, and relates to the technical field of computer information processing. The extraction method comprises the following steps: performing OCR (Optical Character Recognition) on a to-be-processed document table image to obtain a text block; performing visual feature coding on the document table image to obtain deep visual features; performing semantic feature coding on the text sequence of the text block to obtain a semantic feature vector; performing spatial feature coding on the bounding box of the text block to obtain a spatial feature vector; performing feature fusion processing on the deep visual features, the semantic feature vectors and the spatial feature vectors to obtain multi-modal guide features; and performing structured decoding processing on the multi-modal guide features to obtain structured representation of the table. According to the method, the text and the position information pre-recognized by the OCR are fused with the visual features of the document table, so that the visual features are guided to be expressed again and are actively aligned to the logic structure defined by the prior information, and the extraction accuracy of the table logic structure is improved.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and in particular to a method, apparatus, device, and medium for extracting tables from documents. Background Technology

[0002] Document table extraction is a key technology that detects and structurally analyzes table regions in document images, extracting cell content (OCR recognition). It has wide applications in financial statement analysis, bill processing, and document digitization. Existing technologies typically follow a sequential process of "image preprocessing -> table structure recognition -> cell content recognition" when handling document table recognition tasks. However, in the real physical world, document images acquired through scanning or photography generally suffer from geometric distortions (such as perspective distortion, wrinkles, and curvature), uneven lighting, low resolution, and noise. These defects directly cause the rows and columns of the table to become visually misaligned, severely interfering with subsequent table structure recognition.

[0003] While existing image correction algorithms can alleviate global geometric distortion to some extent, they often fall short when dealing with complex local deformations. When structural recognition is flawed—for example, incorrectly merging two rows into one or splitting a column into two—subsequent OCR recognition steps, even if they accurately identify individual text blocks, will suffer from serious errors in the association between cell content and table headers due to incorrect structural attribution; these errors result in "serialization" or "misalignment" problems. These errors are structural and cannot be solved simply by improving the accuracy of the OCR engine.

[0004] Therefore, how to overcome the visual defects of document images and achieve accurate recognition of the logical structure of tables is a technical problem that urgently needs to be solved in the field of document table extraction. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for extracting tables from documents to solve the aforementioned technical problems.

[0006] This invention is achieved through the following technical solution: A first aspect of the present invention provides a method for extracting tables from a document, comprising: Perform OCR recognition on the table image of the document to be processed to obtain text blocks; Visual feature encoding is performed on the document table image to obtain deep visual features; The semantic feature vector is obtained by semantically encoding the text sequence of the text block. Spatial feature encoding is performed on the bounding box of the text block to obtain a spatial feature vector; The deep visual features, the semantic feature vector, and the spatial feature vector are fused to obtain multimodal guidance features; The multimodal guidance features are subjected to structured decoding to obtain a structured representation of the table.

[0007] The method of this invention addresses the problem that existing processing methods separate visual features from textual semantic information, failing to fully utilize prior textual knowledge to assist and calibrate table structure recognition. It innovatively uses pre-recognized text content and its spatial location information via OCR as semantic and spatial prior guiding features, deeply fusing them with the visual features of the document table to form a fused feature representation with multimodal prior information. This guides the re-expression of visual features, enabling the model to no longer rely on traditional, limited-effect pixel-level image correction. Instead, within a high-dimensional feature space, it forces visual features to actively align to the logical structure defined by prior information, allowing the decoding model to handle complex local deformations such as wrinkles and bends that are difficult to handle with traditional methods, thus improving decoding robustness.

[0008] Furthermore, before performing OCR recognition on the document table image to be processed, the method further includes: Obtain the document image to be processed, identify and extract the table region in the document image to obtain the document table image to be processed; The document images are converted from documents in real-world scenarios, and the conversion methods include taking photos and scanning.

[0009] Furthermore, OCR recognition is performed on the document table image to be processed to obtain text blocks, including: At least two heterogeneous OCR recognition engines are invoked to perform OCR recognition on the document table image to be processed, and the recognition results output by each OCR recognition engine are obtained; the recognition results include text blocks and confidence scores of the text blocks; The confidence level of each text region is obtained based on the confidence score of each text block in each text region; Based on the credibility level, each of the text regions is divided into a first text region and a second text region; Select the text block with the highest confidence score from all text blocks in the first text region as the final text block corresponding to the first text region; The text sequence of the second text region is corrected to obtain the corrected text sequence; The text blocks of the second text region are corrected according to the corrected text sequence to obtain corrected text blocks, and the corrected text blocks are used as the final text blocks corresponding to the second text region.

[0010] Further, the text sequence of the second text region is corrected to obtain a corrected text sequence, including: Obtain a local image of the second text region and extract the visual features of the local image; Obtain a preliminary text sequence of the second text region, and extract the text features of the preliminary text sequence; the preliminary text sequence is obtained from the existing text sequence in the text block of the second text region; The visual features and the text features are input into the cross-attention mechanism to obtain the cross-attention fusion feature vector; Based on the cross-attention fusion feature vector, the initial text sequence is corrected character by character to obtain the corrected text sequence.

[0011] Further, the text blocks of the second text region are corrected according to the corrected text sequence to obtain corrected text blocks, including: Obtain the confidence score of the corrected text sequence; If the confidence score of the corrected text sequence is not lower than the confidence threshold, then the existing text sequence in the text block of the second text region is replaced with the corrected text sequence to obtain the corrected text block; If the confidence score of the corrected text sequence is lower than the confidence threshold, the existing text sequence in the text block of the second text region is replaced with a predefined semantic placeholder to obtain the corrected text block.

[0012] Furthermore, the method also includes replacing the text sequence of the text block whose text sequence is empty with a predefined semantic placeholder.

[0013] Furthermore, the deep visual features, the semantic feature vector, and the spatial feature vector are fused to obtain multimodal guidance features, including: The deep visual features, the semantic feature vector, and the spatial feature vector are projected onto the same feature dimension, and then the deep visual features, the semantic feature vector, and the spatial feature vector are concatenated along the feature dimension to obtain multimodal guidance features.

[0014] A second aspect of the present invention provides a document table extraction apparatus, comprising: The OCR recognition module is used to perform OCR recognition on the document table image to be processed to obtain text blocks; A visual feature encoding module is used to encode the visual features of the document table image to obtain deep visual features; A semantic prior extraction model is used to encode the semantic features of the text sequence of the text block to obtain a semantic feature vector; The spatial prior extraction module is used to encode the spatial features of the bounding box of the text block to obtain a spatial feature vector; The multimodal fusion module is used to perform feature fusion processing on the deep visual features, the semantic feature vector, and the spatial feature vector to obtain multimodal guidance features; The structured decoding module is used to perform structured decoding processing on the multimodal guiding features to obtain a structured representation of the table.

[0015] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the document table extraction method according to any one of the first aspects of the present invention.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the document table extraction method according to any one of the first aspects of the present invention.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: By using prior information from text and space to "guide" and "calibrate" the understanding of visual features, the model can correctly infer the logical structure of a table based on the logical connections of the text content and the spatial arrangement rules, even when visual lines are blurred or broken. This fundamentally solves the problem of structural misjudgment caused by unreliable visual information. Through multi-source OCR collaborative decision-making and adaptive correction mechanism, this invention can generate high-quality and highly reliable text prior information, effectively combating noise and distortion in real scene images. This improves the quality of input information from the source and lays a solid foundation for subsequent recognition. By introducing semantic placeholders, the complete topological structure of the table is preserved. Through an end-to-end structured decoder, the structure and content of the entire table can be generated globally and in one go. This approach avoids the accumulation of errors in intermediate steps and can effectively prevent serious structural errors such as "serialized" or "misaligned" cell content, thus ensuring the logical correctness of the output results. The framework design of this invention is modular, allowing for flexible replacement or upgrading of the OCR engine, visual backbone network, and language model to adapt to technological developments and the needs of different application scenarios, thus exhibiting good versatility and scalability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of a document table extraction method according to an embodiment of the present invention; Figure 2 This is a flowchart of a prior information extraction method based on collaborative decision-making and credibility grading according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a table extraction process for a financial statement image according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a table extraction result according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0020] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to other steps or units inherent in the device.

[0021] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in commonly used dictionaries) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0022] Due to non-ideal factors such as geometric distortion and visual noise in document images, the accuracy of table structure recognition relying on pure visual models is low. Furthermore, existing table extraction methods based on optical character recognition (OCR) usually treat OCR as an isolated post-processing step, which prevents the valuable textual semantic information and inherent spatial layout information in the table structure from being effectively utilized in the structure recognition stage, resulting in a disconnect between visual and semantic information.

[0023] Based on this, this invention proposes a document table recognition method guided by multimodal prior information. The core idea of ​​this method is to construct a powerful semantic and spatial prior information from the pre-recognized text content and its spatial location information by OCR. This prior information is used to actively guide and modulate the expression of visual features, forming a deeply fused multimodal feature representation. Based on this feature representation, the decoding model can more deeply understand the logical layout of the table, ultimately achieving end-to-end, high-precision joint recognition of the table's logical structure and cell content.

[0024] See Figure 1 The flowchart below illustrates a method for extracting a document table according to an embodiment of the present invention, which includes the following steps.

[0025] S1 performs OCR recognition and visual feature encoding on the document table image to be processed, respectively, to obtain text blocks and deep visual features.

[0026] First, an image of the document table to be processed is acquired, which includes the table objects to be extracted. The original form of the document table image is a document file, which can be obtained from various document file formats through format conversion, such as converting Word / PDF documents to common image formats, or directly obtaining it from a document in a real scene by taking a picture or scanning. Document images obtained by scanning or photographing are more prone to visual defects such as geometric distortion, uneven lighting, low resolution, and noise. It is understood that before step S1 of this invention, basic preprocessing, such as document structure recognition, table region recognition and extraction, has already been performed on the acquired document image.

[0027] OCR processing of document table images yields individual text blocks. This step can be implemented using an existing OCR engine. The OCR engine outputs the text sequence, bounding boxes, and confidence scores recognized from the document table image. The text sequence is the recognized text content, such as "business revenue" or "¥1,500,000.00". The bounding boxes are stored as position vectors, such as (x_min, y_min, x_max, y_max), where x_min and x_max correspond to the minimum and maximum values ​​of the bounding box in the x-direction, and y_min and y_max correspond to the minimum and maximum values ​​of the bounding box in the y-direction. The confidence score represents the computer's level of confidence in the recognition result, expressed as a number between 0 and 1. For example, 0.98 represents a 98% certainty that the recognition result is "business revenue", while 0.6 represents only a 60% certainty, reflecting the reliability of the text sequence.

[0028] Visual feature encoding of document table images yields deep visual features. This step can be implemented using a visual backbone network, such as the Swin Transformer. The raw, uncorrected complete document table image is input into the visual backbone network, which analyzes the image region by region, extracting deep visual feature maps that characterize the global and local texture, edges, and layout information of the image.

[0029] S2, semantic feature encoding is performed on the text sequence of the text block to obtain a semantic feature vector, and spatial feature encoding is performed on the bounding box of the text block to obtain a spatial feature vector.

[0030] This step performs two-dimensional positional encoding on the bounding box position vectors of each text block obtained in S1, projecting them into a high-dimensional space to generate spatial feature vectors that accurately describe the relative position and size of each text block in the global image. For example, by normalizing each bounding box and then projecting it into a high-dimensional space using a positional encoding function, a spatial feature vector is obtained. The normalization of the bounding boxes can be achieved by dividing the width and height of the bounding box by the width and height of the image, respectively, to obtain the relative height and relative width, thus transforming the absolute coordinates into uniform relative coordinates. The positional encoding function, such as sinusoidal positional encoding or its variants, transforms the four normalized coordinates into a high-dimensional vector (e.g., 768-dimensional), with each coordinate corresponding to multiple numbers, ultimately forming a vector representation of 768 numbers. Each text block corresponds to a high-dimensional spatial vector.

[0031] Semantic feature encoding obtains semantic feature vectors for each text sequence through a semantic model. Semantic features and spatial features of the same text block have a correspondence. Based on this correspondence, semantic features and spatial features can be fused to obtain text block-level semantic and spatial prior features. Concatenating the semantic and spatial prior features of all text blocks yields document-level spatial prior features.

[0032] S3 performs feature fusion processing on deep visual features, semantic feature vectors, and spatial feature vectors to obtain multimodal guided features.

[0033] First, project each feature onto the same feature dimension, then concatenate them along that feature dimension to obtain multimodal guided features. Alternatively, concatenate semantic and spatial feature vectors to obtain prior features, then concatenate them with deep visual features along the same feature dimension. For example, using two independent, trainable projection networks to linearly project the aforementioned visual and prior features onto a unified feature dimension, and then concatenating these two projected feature vectors, generates the final multimodal guided features.

[0034] This step integrates prior information containing semantics and location into visual features, ensuring that each element in the fused modal guidance features simultaneously contains three dimensions of information: "what it looks like" (visual), "where it is located" (spatial), and "what it says" (semantic). The essence of the guided fusion mechanism is to use the prior information generated by S2 as calibration anchors to guide the re-expression of visual features within the feature space, enabling it to transcend pixel-level geometric distortion and directly align to the inherent logical structure of the table.

[0035] S4. Perform structured decoding on the multimodal guided features to obtain a structured representation of the table.

[0036] By leveraging the multimodal guided features generated by S3, the final structured parsing of the table is completed, accurately reconstructing the "position-text" table logic. This step can be achieved through a Transformer-based structured decoder, which is specifically designed to understand the row and column relationships of the table. By inputting the multimodal guided features into the Transformer-based structured decoder, the model, under the effect of the self-attention mechanism, can determine which text blocks belong to the same row or column based on the spatial and semantic prior information in the fused features, even if they are visually misaligned in the original image, thus significantly improving the robustness of the table structure parsing.

[0037] The structure decoder generates a Markdown structured data file describing the entire logical structure and content of the table in an autoregressive manner from end to end. This output includes row and column relationships, cross-row and cross-column information, and the final text within the cells.

[0038] This invention differs from traditional serial processing paradigms by innovatively employing a multimodal guided fusion mechanism. It utilizes prior textual and spatial information to "guide" and "calibrate" the model's understanding of visual features. This deep fusion enables the model to see beyond visual phenomena to the essence of structure. Even when visual lines are blurred or broken, it can correctly infer the logical structure of the table based on the logical connections within the textual content (e.g., similar data should be in the same column) and spatial arrangement rules, fundamentally solving the problem of structural misjudgment caused by unreliable visual perception.

[0039] In a further improved embodiment, when performing OCR recognition on the document table image to be processed, at least two heterogeneous OCR recognition engines are invoked to perform OCR recognition on the document table image to be processed, and the recognition results output by each recognition engine are obtained. The recognition result output by each recognition engine includes the text block corresponding to each text region of the document table image and the confidence score of the text block.

[0040] Heterogeneity is reflected in differences in model architecture, training datasets, or recognition paradigms, ensuring that the capabilities of each engine are complementary and reducing the risk of systematic bias in a single model. This embodiment aims to robustly and accurately extract text content and its bounding box positions from potentially distorted document table images, and based on collaborative decision-making and confidence grading, provide more accurate prior information for subsequent steps.

[0041] Prior information is extracted and corrected based on collaborative decision-making and credibility grading to ensure its credibility. Specific operations are as follows: Figure 2 As shown.

[0042] S2-1: Based on the confidence scores of each text block in each text region, the confidence level of each text region is obtained.

[0043] S2-2, divide each text region into a first text region and a second text region according to the credibility level. The division rule can be: the text region with a credibility level higher than a preset level is designated as the first text region, and the rest is designated as the second text region.

[0044] S2-3, differentiate between the first and second text regions: S2-31, For the first text region: Select the text block with the highest confidence score among all text blocks in the first text region as the final text block corresponding to the first text region; S2-32, for the second text region: (1) perform correction processing on the text sequence of the second text region to obtain the corrected text sequence; (2) perform correction processing on the text block of the second text region according to the corrected text sequence to obtain the corrected text block; (3) take the corrected text block as the final text block corresponding to the second text region.

[0045] For example, confidence levels can be divided into high, medium, and low. For text regions at the medium and high levels, the text block with the highest confidence score is selected for prior information construction. Text blocks with high confidence scores have higher recognition reliability, while text blocks with low confidence scores have a higher probability of error. When the confidence scores of all text blocks corresponding to a certain text region are low (e.g., all below the preset score threshold of 0.6), the overall reliability of that region is low, and the recognition results of each engine are unreliable. Correction processing is required before incorporating them into the prior information construction to ensure the accuracy of the prior information.

[0046] Using actual data as an example, if the confidence scores of all engines for recognizing a certain region are higher than the first preset threshold (e.g., 0.95), and the text blocks of each recognition result are highly consistent within the preset string similarity threshold (e.g., normalized edit distance less than 0.1) and bounding box intersection-union (IoU) threshold (e.g., greater than 0.9), then the region is classified as a high-confidence region, the result with the highest confidence score is adopted, and it is marked as a high-confidence prior; or the consistent result (e.g., the voted text and the fused bounding box) is adopted. If the recognition results of each engine are inconsistent, but the confidence score of one engine is significantly higher than that of other engines and higher than the second preset threshold (e.g., 0.8), then the region is classified as a medium-confidence region, the result with the highest confidence score is adopted, and it is marked as a medium-confidence prior. If there are significant conflicts in the results of each engine, and the confidence scores of all engines are lower than the second preset threshold, then the region is marked as a low-confidence region, awaiting further correction processing.

[0047] This embodiment utilizes a multi-source OCR collaborative decision-making and adaptive correction mechanism to generate high-quality, highly reliable textual prior information, effectively combating noise and distortion in real-world images. This improves the quality of input information from the source, laying a solid foundation for subsequent recognition.

[0048] The credibility grading mechanism enables subsequent fusion models to learn to "differentiate" and "dynamically weigh" prior information from different labels. The model can intelligently switch the focus of dependence between reliable text priors and noisy visual features, avoiding interference from unreliable information on the overall decision-making process, and enabling the model to exhibit higher intelligence and reliability in real and complex scenarios.

[0049] In a further improved embodiment, text sequences in a second text region with a confidence level not higher than a preset level are corrected as follows.

[0050] S3-1. Obtain a local image of the second text region and extract visual features of the local image.

[0051] S3-2. Obtain a preliminary text sequence of the second text region and extract text features of the preliminary text sequence.

[0052] S3-3. Input the visual features and text features into a cross-attention mechanism to obtain a cross-attention fusion feature vector.

[0053] S3-4. Based on the cross-attention fusion feature vector, perform character-by-character correction on the preliminary text sequence to obtain a corrected text sequence.

[0054] In this embodiment, for a text region with a substandard confidence level, by instantiating a lightweight vision-language correction module, the local image slices of the text region and the preliminary text sequence are input into the vision-language correction module for adaptive correction. Among them, the preliminary text sequence is obtained from the existing text sequence in the text block of the second text region, that is, the text sequence obtained by preliminarily recognizing the text region through at least two recognition engines. Through the cross-attention mechanism, the model can focus on the local image texture features, re-evaluate and correct the preliminary text sequence character by character or word by word, and finally output the corrected text sequence and a new confidence score.

[0055] For example, the three OCR results of a certain second text region are "administrative expenses", "custody expenses", and "administrative expenses". The character (or word) that appears most frequently is taken as the initial text sequence. For example, if the character "guan" is the same in all three results, it is retained. For "ya, jia, li", they are inconsistent, and all possibilities are temporarily retained (as candidates). For "expenses", they are the same, so it is retained. Finally, the initial text sequence formed is "guan [ya / jia / li] expenses". Or using the confidence of the characters, select the character with the highest confidence among the candidate characters to form the initial text sequence. For example, among "ya, jia, li", the confidence of "li" is the highest, and finally the initial text sequence of "guanli expenses" is formed.

[0056] Input "guanli expenses" and the visual features of the local image into the cross-attention mechanism model. Based on the verification results of the cross-attention, correct the initial text sequence one by one, change "li" to "li", obtain "administrative expenses", and update the confidence score of "administrative expenses" to 0.92.

[0057] Further, in S2-32, the text block of the second text region is corrected according to the corrected text sequence to obtain the corrected text block. Specifically, if the confidence score of the corrected text sequence is not lower than the confidence threshold (e.g., 0.92>0.9), the existing text sequence in the text block of the second text region is replaced with the corrected text sequence to obtain the corrected text block; if the confidence score of the corrected text sequence is lower than the confidence threshold, the existing text sequence in the text block of the second text region is replaced with a predefined semantic placeholder to obtain the corrected text block.

[0058] For example, for regions where the confidence level remains below a preset threshold (e.g., 0.5) after correction, unreliable text is not retained at those locations. Instead, a predefined placeholder with specific semantics is generated at the original bounding box location, such as...<UNCERTAIN_TEXT> The innovation of this approach lies in the fact that while discarding unreliable content information, it fully preserves the fact that the cell exists and its precise spatial location within the entire table, thereby ensuring the integrity of the table's topological structure and providing crucial layout clues for subsequent structural identification.

[0059] As for updating the bounding box of the correction text block, one of the three can be selected, or the coordinates of the three bounding boxes can be averaged to obtain a new bounding box as the bounding box of the correction text block.

[0060] Furthermore, the text sequence of text blocks with empty text sequences is replaced with predefined semantic placeholders.

[0061] Similar to the placeholder handling for semantic uncertainty, semantic placeholders transform "recognition failure" into a key clue of "structural success," converting content recognition failure (a negative event) into a strong structural signal (a positive clue) that retains precise spatial location, effectively blocking error propagation. This semantic placeholder ensures that even when some cell content cannot be recognized, the overall topological skeleton of the table remains intact, fundamentally solving the persistent problem of global, catastrophic structural errors such as "serialization" and "row misalignment" caused by local errors in existing serial methods, thus ensuring the logical correctness of the output results.

[0062] The following example illustrates the technical effects of the improved solution of this invention. The example involves processing a financial statement image taken with a mobile phone, exhibiting slight perspective distortion and slightly dim lighting on the right side. The statement includes columns for item names and amounts. The key process for extracting this data is described below. Figure 3 As shown.

[0063] I. Acquisition and correction of prior information.

[0064] 1.1 OCR recognition of text blocks.

[0065] The system concurrently invokes three OCR engines (Engine A: Tesseract-based; Engine B: PaddleOCR-based; Engine C: proprietary CNN+CTC model) to process images.

[0066] 1.2. Extract credible prior information according to the collaborative decision-making mechanism.

[0067] For the "Operating Income" cell in the report, all three engines clearly recognize it as "Operating Income", with the IoU of the bounding box > 0.95 and the confidence levels of all > 0.98. This result is determined to be a high-confidence prior; For an amount cell "¥1,500,000.00", Engine A recognizes it as "¥1,500,00(J.00" (confidence level 0.75) due to lighting issues, Engine B recognizes it as "¥1,500,000.00" (confidence level 0.96), and Engine C recognizes it as "¥1,500,000.OO" (confidence level 0.88). According to the collaborative decision-making mechanism, the result of Engine B is adopted due to the highest confidence level and is marked as a medium-confidence prior.

[0068] For a cell blurred due to paper wrinkles, the three engines give conflicting results with low confidence levels (all < 0.6) such as "Administrative Expenses", "Detention Expenses", "Administrative Expenses", etc., and this area is marked as a low-confidence text area.

[0069] 1.3. Prior information correction.

[0070] The system sends the image slice of the "Administrative Expenses" area and the preliminarily recognized text "Administrative Expenses" to the vision-language correction module. Through the cross-attention mechanism, this module discovers that the visual features of the "理" radical in the image are more inconsistent with "押" or "赍", and finally corrects the text to "Administrative Expenses" and gives an updated confidence level of 0.92.

[0071] 1.4. Introduce semantic placeholders.

[0072] For a completely blank cell, the system generates the **<EMPTY_CELL>** placeholder at its bounding box position.

[0073] II. Multimodal feature encoding and fusion.

[0074] 2.1. Visual feature encoding.

[0075] Input the entire financial report image into the Swin Transformer-Base model to obtain a deep visual feature.

[0076] 2.2. Spatial feature encoding.

[0077] Obtain the normalized bounding box coordinates of each text / placeholder, such as (0.1, 0.2, 0.3, 0.25) for "Operating Revenue", and generate its spatial feature vector.

[0078] 2.3 Multimodal prior information fusion.

[0079] Two MLPs (Multilayer Perceptrons) are used as projection networks to project the two feature vectors mentioned above onto a 768-dimensional network. The identified text is then converted into vectors and concatenated along the feature dimensions to form a multimodal guided feature sequence containing all text / placeholder information.

[0080] III. Structured Decoding Based on Prior Guidance.

[0081] The multimodal guided feature sequence is concatenated with task prompts (such as "generate Markdown table") and then input into a Transformer-based table recognition model (similar to the decoder part of Donut or Pix2Struct).

[0082] The decoder eventually generates autoregressively, such as Figure 4 The Markdown output shown accurately reproduces the logical structure and content of the table. As shown in the figure, because "Operating Revenue" and "¥1,500,000.00" are spatially close in the vertical direction in the fused features, and semantically one is an item name and the other is an amount, the model can correctly infer that they belong to different columns but the same row. Even in the original image, due to perspective distortion, the vertical center of "¥1,500,000.00" is slightly lower than that of "Operating Revenue", and the model can still be calibrated using prior information.

[0083] During model training, a phased fine-tuning strategy is adopted to efficiently train the model used in the above-mentioned case of the present invention.

[0084] (1) First stage: Freeze the main parameters of the visual backbone network and the language model, and train only the two projection networks in step 2.3. This stage aims to teach the model how to align and initially fuse information from different modalities.

[0085] (2) Second stage: Based on the first stage, unfreeze all model parameters (or set a lower learning rate) and fine-tune the entire model end-to-end. This stage enables the modules to optimize collaboratively to maximize the final accuracy of table recognition. This training strategy leverages the power of pre-trained large models while ensuring efficiency in adapting to specific tasks.

[0086] The synergistic effect of the above-mentioned technical points of the present invention enables efficient, intelligent, and precise matching of supply and demand for science and technology services, significantly improving the platform's service capabilities and user experience, and has broad practical application value.

[0087] Embodiments of the present invention also provide a document table extraction device, comprising: The OCR recognition module is used to perform OCR recognition on the document table image to be processed to obtain text blocks; The visual feature encoding module is used to encode the visual features of document table images to obtain deep visual features; A semantic prior extraction model is used to encode semantic features of text sequences of text blocks to obtain semantic feature vectors; The spatial prior extraction module is used to encode the spatial features of the bounding boxes of text blocks to obtain spatial feature vectors; The multimodal fusion module is used to fuse deep visual features, semantic feature vectors, and spatial feature vectors to obtain multimodal guidance features; The structured decoding module is used to perform structured decoding processing on multimodal guided features to obtain a structured representation of the table.

[0088] Furthermore, the OCR recognition module includes at least two heterogeneous optical character recognition engines, each of which performs OCR recognition on the document table image to be processed, and obtains the recognition results output by each recognition engine.

[0089] The document table extraction device mentioned above also includes a prior information acquisition module and a prior correction module. The prior information acquisition module extracts high-confidence prior information from the recognition results of each recognition engine based on a collaborative decision-making mechanism; the prior correction module corrects the text region if the confidence level does not meet the requirements, and obtains the corrected prior information.

[0090] Furthermore, the prior correction module also includes a semantic placeholder configuration module, which replaces the text sequence of text blocks that are still unreliable after correction processing with predefined semantic placeholders.

[0091] The semantic feature extraction module performs semantic feature extraction on each text sequence, including semantic feature extraction on the replaced semantic placeholders. That is, the replaced semantic placeholders are used as the updated text sequence. The extraction results indicate that the text sequence at the corresponding bounding box position is unreliable or is a blank cell.

[0092] Furthermore, the visual encoding module employs the Swin Transformer visual backbone network; the semantic feature extraction module uses a language model; the spatial feature extraction module uses a normalization layer to normalize the bounding boxes of each text block, and then projects them into a high-dimensional space through a positional encoding function to generate spatial feature vectors; the multimodal guidance module uses two independent, trainable projection networks to linearly project visual features and prior features onto a unified feature dimension, and then concatenates the two projected feature vectors to generate the final multimodal guidance features; the structured decoding module uses a structured decoder based on the Transformer architecture.

[0093] Furthermore, the document table extraction device also includes a fine-tuning training module for performing phased fine-tuning training on the aforementioned model structure.

[0094] Embodiments of the present invention also provide an electronic device including a processor and a memory, wherein the number of processors may be one or more. The memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory, thereby implementing the document table extraction method of any of the above embodiments of the present invention.

[0095] The memory may primarily comprise a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0096] Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document table extraction method of any embodiment of the present invention.

[0097] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0099] Embodiments of the present invention also provide a computer program product that, when run on a computer, causes the computer to execute the document table extraction method of any of the above embodiments of the present invention.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting tables from a document, characterized in that, include: The document table image to be processed is subjected to OCR recognition to obtain text blocks; Visual feature encoding is performed on the document table image to obtain deep visual features; The semantic feature vector is obtained by semantically encoding the text sequence of the text block. Spatial feature encoding is performed on the bounding box of the text block to obtain a spatial feature vector; The deep visual features, the semantic feature vector, and the spatial feature vector are fused to obtain multimodal guidance features; The multimodal guidance features are subjected to structured decoding to obtain a structured representation of the table.

2. The method for extracting document tables according to claim 1, characterized in that, Before performing OCR recognition on the document table image to be processed, the method further includes: Obtain the document image to be processed, identify and extract the table region in the document image to obtain the document table image to be processed; The document images are converted from documents in real-world scenarios, and the conversion methods include taking photos and scanning.

3. The method for extracting document tables according to claim 1, characterized in that, The document table image to be processed is subjected to OCR recognition to obtain text blocks, including: At least two heterogeneous OCR recognition engines are invoked to perform OCR recognition on the document table image to be processed, and the recognition results output by each OCR recognition engine are obtained; the recognition results include text blocks and confidence scores of the text blocks; The confidence level of each text region is obtained based on the confidence score of each text block in each text region; Based on the credibility level, each of the text regions is divided into a first text region and a second text region; Select the text block with the highest confidence score from all text blocks in the first text region as the final text block corresponding to the first text region; The text sequence of the second text region is corrected to obtain the corrected text sequence; The text blocks of the second text region are corrected according to the corrected text sequence to obtain corrected text blocks, and the corrected text blocks are used as the final text blocks corresponding to the second text region.

4. The method for extracting document tables according to claim 3, characterized in that, The text sequence of the second text region is corrected to obtain a corrected text sequence, including: Obtain a local image of the second text region and extract the visual features of the local image; Obtain a preliminary text sequence of the second text region, and extract the text features of the preliminary text sequence; the preliminary text sequence is obtained from the existing text sequence in the text block of the second text region; The visual features and the text features are input into the cross-attention mechanism to obtain the cross-attention fusion feature vector; Based on the cross-attention fusion feature vector, the initial text sequence is corrected character by character to obtain the corrected text sequence.

5. The method for extracting document tables according to claim 3 or 4, characterized in that, The text blocks of the second text region are corrected according to the corrected text sequence to obtain corrected text blocks, including: Obtain the confidence score of the corrected text sequence; If the confidence score of the corrected text sequence is not lower than the confidence threshold, then the existing text sequence in the text block of the second text region is replaced with the corrected text sequence to obtain the corrected text block; If the confidence score of the corrected text sequence is lower than the confidence threshold, the existing text sequence in the text block of the second text region is replaced with a predefined semantic placeholder to obtain the corrected text block.

6. The method for extracting document tables according to claim 5, characterized in that, The method further includes replacing the text sequence of the text block where the text sequence is empty with a predefined semantic placeholder.

7. The method for extracting document tables according to claim 1, characterized in that, The deep visual features, the semantic feature vector, and the spatial feature vector are fused to obtain multimodal guidance features, including: The deep visual features, the semantic feature vector, and the spatial feature vector are projected onto the same feature dimension, and then the deep visual features, the semantic feature vector, and the spatial feature vector are concatenated along the feature dimension to obtain multimodal guidance features.

8. A document table extraction device, characterized in that, include: The OCR recognition module is used to perform OCR recognition on the table image of the document to be processed to obtain text blocks; A visual feature encoding module is used to encode the visual features of the document table image to obtain deep visual features; A semantic prior extraction model is used to encode the semantic features of the text sequence of the text block to obtain a semantic feature vector; The spatial prior extraction module is used to encode the spatial features of the bounding box of the text block to obtain a spatial feature vector; The multimodal fusion module is used to perform feature fusion processing on the deep visual features, the semantic feature vector, and the spatial feature vector to obtain multimodal guidance features; The structured decoding module is used to perform structured decoding processing on the multimodal guiding features to obtain a structured representation of the table.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the document table extraction method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the document table extraction method according to any one of claims 1-7.

Images