Document table intelligent identification method and system, and electronic device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-07
- Publication Date
- 2026-08-11
AI Technical Summary
[0005]尽管表格识别技术已实现从多阶段到端到端的范式演进，但现有技术方案仍存在诸多缺陷，难以满足金融、科研等严谨场景对表格识别精度与鲁棒性的高要求，具体缺陷及原因如下：
[0033]First, a dual-stream structure prior injection architecture is used to extract the geometric skeleton features of the table using an independent side encoder. The structure prior is then injected layer by layer into the VLM decoder. Combined with a hierarchical learnable scalar, dynamic weighted fusion of semantic and geometric features is achieved. This not only retains the efficiency of the end-to-end framework but also allows the model generation process to refer to the physical row and column lines in real time, significantly improving the robustness of recognition in complex table scenarios. At the same time, a frozen SPN structure predictor is used, combined with explicit visual cue rendering and error calibration mechanism. This not only ensures the stability of the structure prior but also calibrates the structure prediction error through feature complementarity, avoiding the impact of a single structure prediction bias on the overall recognition effect.
Smart Images
Figure CN122551376A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of document intelligent processing technology, and in particular to a document table intelligent recognition method, system and electronic device, as well as a computer-readable storage medium. Background Technology
[0002] Table recognition (TR) is a core task in the field of intelligent document parsing, and its accuracy and efficiency directly determine the practicality and reliability of intelligent document parsing systems. With the rapid development of computer vision, natural language processing, and multimodal large model technologies, the field of table recognition has undergone a paradigm shift from a "multi-stage pipeline framework" to an "end-to-end generative framework." Both technical paths have their own characteristics, but both face insurmountable technical bottlenecks.
[0003] (1) Multi-stage Pipelines: The core idea of this type of scheme is to break down the table recognition task into multiple independent sub-tasks, realize the complete recognition of the table through step-by-step processing, train and reason independently for each sub-task, and finally fill the text content obtained by OCR recognition into the table grid obtained by structure recognition through coordinate matching algorithm to complete the structured parsing of the table.
[0004] (2) End-to-end Visual Language Models (VLMs): In recent years, with the rise of large-scale multimodal models (such as Qwen2.5-VL, LLaVA, etc.), the field of table recognition has ushered in a paradigm shift, beginning to move towards an end-to-end generation mode based on the Encoder-Decoder architecture. This type of method abandons the idea of multi-stage step-by-step processing, directly taking the table image as input, and mapping the image information into structured sequences (such as HTML, Markdown, etc.) through multimodal models. By utilizing the pre-trained semantic understanding capabilities of multimodal models, the overall process of the table recognition system is greatly simplified.
[0005] Although table recognition technology has undergone a multi-stage, end-to-end paradigm evolution, existing technical solutions still have many shortcomings and cannot meet the high requirements for accuracy and robustness in rigorous scenarios such as finance and scientific research. The specific shortcomings and their causes are as follows:
[0006] (1) The dilemma of balancing multi-stage error accumulation and end-to-end black boxing: Although the traditional multi-stage pipeline framework can ensure the accuracy of local recognition to a certain extent through the physical coordinate constraints of each sub-task, due to the excessive task splitting, there is strong coupling between each stage. The recognition error of the previous stage will continue to be transmitted and accumulated in the subsequent stages, eventually leading to a decrease in overall recognition accuracy. While the existing end-to-end VLM (Vision-Language Model) solution solves the problem of multi-stage error accumulation, it adopts a "black box" generation mode and lacks explicit geometric structure guidance. When dealing with complex table scenarios such as tilt, blur, and no bounding box, the robustness of the model is poor, and problems such as misalignment of structure recognition and content matching errors are likely to occur.
[0007] (2) Logical collapse caused by “empty column hallucination”: The existing end-to-end VLM is essentially a sequence generation model driven by visual semantics. Its understanding of table structure is highly dependent on the visual semantic signals in the image. When processing frameless tables and sparse tables, due to the lack of explicit physical boundaries (such as border lines and separators), the model tends to skip those “visual vacuum” areas where visual information is missing, and thus ignores cells that exist logically but are visually empty, resulting in the phenomenon of “empty column hallucination”. This will directly lead to a complete disorder in the correspondence between table data items and column headers, making the table recognition results lose practical application value;
[0008] (3) Cascaded misalignment error caused by lack of physical spatial perception: Since the existing end-to-end VLM lacks explicit spatial inductive bias, its understanding of table structure relies entirely on semantic association in the autoregressive generation process, and it cannot effectively perceive the physical spatial layout of the table. Once the model misses a cell tag (such as in HTML tags) during the sequence generation process due to weak visual signals or semantic ambiguity, the generation of all subsequent cells will be "left-shifted" in the logical topology, forming a cascaded misalignment. This error has a strong cumulative effect. A small lack of spatial perception can cause the parsing results of the entire row or even the entire table to fail, seriously affecting the reliability of table recognition.
[0009] (4) Mismatch between semantic drive and physical structure: Existing end-to-end VLMs rely too much on textual semantic information to infer table structure, while ignoring the core physical attributes of a table as a geometric layout. When there are many complex scenarios such as cross-row merging, cross-column merging, or irregular layout in the table, it is difficult to accurately restore the complex topological relationship of the table based solely on textual semantic association. At the same time, in the absence of strong spatial alignment signal constraints, the structured sequence (such as HTML tags) generated by the model often cannot accurately correspond to the physical area of the table in the original image, resulting in the recognition result accuracy failing to meet the application requirements of rigorous scenarios, thus limiting the industrial application of end-to-end table recognition technology.
[0010] In summary, although existing table recognition technologies have completed a paradigm evolution from multi-stage to end-to-end, they have many shortcomings and lack technical solutions that balance efficiency, accuracy, and robustness, making it difficult to meet the application requirements of rigorous scenarios. Summary of the Invention
[0011] To address the aforementioned technical problems in the prior art, namely how to achieve intelligent table recognition that balances efficiency, accuracy, and robustness, embodiments of this application provide a document table intelligent recognition method, system, and electronic device.
[0012] In a first aspect of this application, a document table intelligent recognition method is provided, comprising:
[0013] Using a pre-trained, frozen structure proposal network, row and column grids are generated from the original table image;
[0014] The row and column grids are superimposed on the original table image as line segments to generate a structure enhancement diagram. The line segments are of a preset color;
[0015] The original table image is processed using a visual encoder based on a large visual language model to extract features, obtaining main visual features that include text region information and visual semantic information. ;
[0016] The geometric topological features of the structure enhancement map are extracted using a side encoder. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position;
[0017] In each Transformer decoding layer of the decoder in the large visual language model, a cross-attention submodule is set after the self-attention submodule. The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , N is an integer, where N is the total number of Transformer decoding layers, and the decoder is an autoregressive Transformer decoder.
[0018] Calculate the first Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer. , Presentation layer normalization operation, The initial value of the hidden state of the Transformer decoding layer is a hierarchically learnable scalar. , Represents word embedding function, Represents the historical true target sequence, Indicates the current time;
[0019] With the first The output of the Transformer decoding layer is the input of the multi-task position-aware output head of the decoder. The multi-task position-aware output head outputs the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell.
[0020] In a second aspect of this application, a document table intelligent recognition system is provided, comprising:
[0021] The SPN module is used to generate a row and column grid from the original table image using a pre-trained, frozen structure proposal network.
[0022] The rendering module is used to overlay the row and column grids onto the original table image as line segments to generate a structure-enhanced diagram. The line segments are of a preset color;
[0023] A visual encoder is used to extract features from the original table image to obtain main visual features that include text region information and visual semantic information. The visual encoder belongs to the large visual language model;
[0024] A side encoder is used to extract the geometric topological features of the structure enhancement map. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position;
[0025] The decoder is an autoregressive Transformer decoder. Each Transformer decoding layer of the decoder is followed by a cross-attention submodule. The decoder is used for... The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , Let N be an integer, where N is the total number of Transformer decoding layers, and the number of layers is used to calculate the [number of layers]. Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer. , Presentation layer normalization operation, The initial value of the hidden state of the Transformer decoding layer is a hierarchically learnable scalar. , Represents word embedding function, Represents the historical true target sequence, This indicates that at the current moment, the decoder belongs to the large visual language model;
[0026] Multi-task position-aware output head, used for... The output of the Transformer decoding layer is the input, and the output is the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell. The multi-task position-aware output head belongs to the visual language large model.
[0027] In a third aspect of this application, an electronic device is provided, the electronic device comprising:
[0028] At least one processor; and,
[0029] A memory communicatively connected to at least one of the processors; wherein,
[0030] The memory stores instructions that can be executed by the processor to implement the above-described document table intelligent recognition method.
[0031] In a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being executed by the computer to implement the above-described document table intelligent recognition method.
[0032] The document table intelligent recognition method and system provided in this application, compared with existing table recognition technologies, effectively overcomes the defects such as multi-stage error accumulation, end-to-end black boxing, cascade misalignment, and semantic-physical structure mismatch. Through the synergy of multiple technologies, it achieves simultaneous improvement in table recognition efficiency, accuracy, and robustness. The specific beneficial effects are as follows:
[0033] First, a dual-stream structure prior injection architecture is used to extract the geometric skeleton features of the table using an independent side encoder. The structure prior is then injected layer by layer into the VLM decoder. Combined with a hierarchical learnable scalar, dynamic weighted fusion of semantic and geometric features is achieved. This not only retains the efficiency of the end-to-end framework but also allows the model generation process to refer to the physical row and column lines in real time, significantly improving the robustness of recognition in complex table scenarios. At the same time, a frozen SPN structure predictor is used, combined with explicit visual cue rendering and error calibration mechanism. This not only ensures the stability of the structure prior but also calibrates the structure prediction error through feature complementarity, avoiding the impact of a single structure prediction bias on the overall recognition effect.
[0034] Second, by using a spatial-semantic joint modeling generation strategy, the content token and the cell normalized coordinates are output synchronously. Coordinate regression is used as a strong auxiliary task, which forces the decoder to focus on the physical location. This effectively solves the problem of cascading misalignment caused by the lack of spatial awareness in existing models, ensuring that the generated structured sequence is accurately aligned with the physical area of the original table, and improving the reliability of the recognition results.
[0035] Third, by using the explicit empty cell modeling scheme, and through the training strategy with empty cells combined with geometric clues, a dedicated placeholder token is introduced to decouple the existence of table structure from the existence of content, effectively eliminating the "empty column illusion". Even in tables without borders and with sparse content, the logical topology structure can be completely preserved, avoiding the confusion between data items and column headers, and adapting to the needs of rigorous scenarios such as finance and government affairs.
[0036] Fourth, the side encoder adopts a lightweight design, balancing parameter efficiency and geometric topology feature extraction capabilities. Combined with the frozen SPN design, it significantly reduces model training costs and overfitting risks, while improving model convergence speed, making it easier for industrial applications. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 A flowchart illustrating the document table intelligent recognition method provided in this application embodiment;
[0039] Figure 2 This is a structural block diagram of the document table intelligent recognition system provided in the embodiments of this application. Detailed Implementation
[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0041] The following will describe in detail, with reference to the accompanying drawings, a document table intelligent recognition method according to an embodiment of this application. Figure 1 This is a flowchart illustrating the document table intelligent recognition method provided in an embodiment of this application. Figure 1 As shown, the document table intelligent recognition method of the first embodiment of this application includes:
[0042] Step S101: Using the pre-trained frozen structure proposal network, generate a row and column grid based on the original table image;
[0043] Step S102: Overlay the row and column grids onto the original table image in the form of line segments to generate a structure enhancement diagram. The line segments are of a preset color;
[0044] Step S103: Use the visual encoder of the visual language large model to extract features from the original table image to obtain the main visual features containing text region information and visual semantic information. ;
[0045] Step S104: Use a side encoder to extract the geometric topological features of the structure enhancement map. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position;
[0046] Step S105, in each Transformer decoding layer of the decoder of the large visual language model, a cross-attention submodule is set after the self-attention submodule. The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , N is an integer, where N is the total number of Transformer decoding layers, and the decoder is an autoregressive Transformer decoder.
[0047] Step S105, calculate the first... Output of the Transformer decoding layer and with the first Output of the Transformer decoding layer For the first The hidden state of the Transformer decoding layer. Presentation layer normalization operation, As a hierarchical learnable scalar, the initial values of the hidden states of the Transformer decoding layer are based on the main visual features. get;
[0048] Step S106, with the first The output of the Transformer decoding layer is the input of the multi-task position-aware output head of the decoder. The multi-task position-aware output head outputs the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell.
[0049] Specifically, in step S101, the Structure Proposal Network (SPN) can use a pre-trained SLANeXt (Baidu Table Structure Recognition Model), taking the original table image as input and outputting a row and column grid. Specifically, the SPN first predicts the logical grid of the table based on the original table image and generates an auxiliary image containing only grid lines in the background, thus obtaining the row and column grid. In step S102, through physical rendering, the row and column grid is overlaid on the original table image with line segments of a preset color (e.g., red line segments) to generate a structure enhancement map. . , This represents the original image of the table. This represents the mapping function that performs row and column boundary prediction and line segment rendering.
[0050] Specifically, in step S103, the visual encoder (e.g., a standard VLM visual encoder) of a large visual language model (e.g., Qwen-VL, Qwen2.5-VL) is used to extract features from the original table image to obtain the main visual features. (Also known as global semantic features), representing the text features in the table. It should be noted that the visual language big data model is pre-trained to obtain the optimal parameters for table recognition.
[0051] Specifically, in step S104, a side encoder with dynamic resolution adaptation is introduced to extract the structure enhancement map. Geometric structure and topological features Specifically, the extraction expression is: ,in, Embedded for patch, This represents the linear projection function, a standard function module of VLM, used to project features (i.e., ... The input (here, geometric topological features) is mapped to a feature space consistent with the feature dimensions of the decoder in the large visual language model. This refers to a pre-trained structure encoding network (e.g., LayoutLM, MobileViT) used to extract the geometric topological features of a table from a structure enhancement graph. This is the interpolation position encoding. By using the position embedding matrix Generate 2D bilinear interpolation. This refers to the position embedding matrix learned during the training phase of the visual language large model. The position embedding matrix is a parameter matrix corresponding to the number of patches, enabling the visual language large model to recognize the spatial order of each patch. The position corresponding to the position embedding matrix represents the coordinate position of each patch within the two-dimensional patch grid after the fixed-resolution input image is divided into two-dimensional patch grids of the same size as the fixed-resolution input image during the training phase. This is achieved through... Perform 2D bilinear interpolation. When the resolution of the original input table image changes (e.g., from 224×224 to 448×448), the original position embedding matrix is used. When the size no longer matches the number of new patches, Resample to a size that matches the new number of patches, and generate ,make Aligned with the current input resolution, which is the spatial resolution of the two-dimensional patch grid of the same size as the original table image obtained by dividing the original table image, the position embedding matrix is adjusted by bilinear interpolation to obtain geometric structural features that are completely aligned with the spatial structure of the current original table image, adapting to tables with different aspect ratios. The side encoder can freeze the first m layers and fine-tune the last n layers, where m and n can be determined empirically to ensure extraction efficiency while capturing key geometric topologies. By introducing the side encoder, structural prior injection can be achieved. These structural priors are fused and integrated into the decoder of the large visual-language model, achieving a balance between semantics and geometry.
[0052] In the visual language big data model used in the document table intelligent recognition method provided in this application embodiment, the decoder adopts an autoregressive Transformer decoder, including multiple stacked Transformer decoding layers. The Nth Transformer decoding layer is connected to a multi-task position-aware output head, where N is the total number of Transformer decoding layers. Each Transformer decoding layer includes a self-attention submodule, a cross-attention submodule, and a feedforward network submodule connected in sequence. Specifically, in step S105, the Nth Transformer decoding layer... The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , For integers; calculate the first... Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer, i.e. The initial values of the hidden states of the Transformer decoding layer are based on the main visual features. Specifically, we obtain the initial values of the hidden states of the Transformer decoding layer. , The word embedding function is a standard foundational function module for the autoregressive Transformer decoder. Represents the historical true target sequence, Indicates the current moment, that is This represents the historical true target sequence prior to the current moment. The true target sequence is obtained through manual annotation. The representation layer normalization operation is used to normalize the input features along the feature dimension to improve the stability and convergence of the model. As a hierarchical learnable scalar, its initial value can be considered as 0.2. By introducing a hierarchical learnable variable for learnable gating control, the injection intensity of structural information is dynamically controlled, preventing geometric features from masking language semantics. Through the above-mentioned gating cross-attention injection mechanism, dynamic weighted fusion of semantic and geometric features is achieved.
[0053] Specifically, in step S106, the output of the Nth Transformer decoding layer is used as the input to the multi-task position-aware output head of the decoder. The multi-task position-aware output head ultimately outputs the structured representation result of the original table image. The structured representation result includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell. For example, the structured representation result may contain HTML tags and cell content. With cell bounding box Intertwined sequences.
[0054] In one possible implementation, in step S106, when the main visual feature There are regions without text, and their geometric structure and topological features are present. A grid structure exists in the textless region, and the decoder is based on a conditional probability distribution. A placeholder token is generated at the location corresponding to the textless area. Finally, the multi-task location-aware output head outputs the placeholder in the textless area. Let be the real token at position t in the real target sequence (obtained by manual annotation), where t represents the time step index of the real target sequence generation, and its value range is 1≤t≤|Y|. This represents the generated real target sequence up to step t-1. ,by The contextual information currently predicted by the large visual language model. This refers to all learnable parameters of the visual language large-scale model. Specifically, placeholder tokens are predefined labels in the vocabulary. During the training phase of the visual language large-scale model, placeholders are marked on textless regions (empty cells or valid blank areas) in the real target sequence. This enables the model to learn to generate placeholder tokens at corresponding positions, which are defined in the vocabulary. During the operation of the visual language large-scale model, the decoder progressively selects output tokens from the vocabulary. When a corresponding position lacks valid content, a placeholder token is generated, i.e., an output placeholder token is selected from the vocabulary. Through the placeholder mechanism, the existence of structure and the existence of content are effectively decoupled. Even in blank columns with no text, the integrity of the table's logical topology can be maintained. For example, in the final output (such as standard HTML code), blank areas are correctly filled with placeholders (e.g., [EMPTY_CELL]) instead of data misalignment, thereby eliminating cascading structural misalignment errors.
[0055] In one possible implementation, the document table intelligent recognition method provided in this application further includes: pre-training the large visual language model based on minimizing the negative log-likelihood loss to obtain the optimal parameters of the large visual language model. The optimization objective of training the large visual language model is to minimize the negative log-likelihood loss, and the negative log-likelihood loss function is... , Used to measure how well the sequence generated by the model fits the real target sequence. The true target sequence consists of the table HTML tag sequence and the cell content sequence, where |Y| represents the total length of the true target sequence. Let represent the conditional probability distribution of the visual language large-scale model, and let represent the predicted probability of the visual language large-scale model for outputting a token at the current position. Here, t represents the real token at position t in the real target sequence, where t also represents the time step index of the real target sequence generation. The current position is the position in the current model-generated sequence. For example, if the current real target sequence is at position t, then the current position is the position in the model-generated sequence corresponding to the t-th position of the real target sequence. ,by The contextual information predicted by the current large visual language model. These are all the learnable parameters of the visual language model. After obtaining the optimal parameters of the visual language model, the optimized visual language model is used to perform the above-mentioned intelligent document table recognition.
[0056] The document table intelligent recognition method provided in this application was applied to an ablation experiment, and the experimental results are shown in the table below:
[0057]
[0058] In the table above, TSR (Table Structure Recognition) is table structure recognition, Direct TSRInference is direct table structure recognition inference, SA-Table (Predicted) is the document table intelligent recognition method provided by the embodiments of this application, SA-Table (Oracle) is the baseline model using the true ideal benchmark (a dedicated baseline for ablation experimental control), TEDS (Table Edit Distance Similarity) is table edit distance similarity, TEDS-S (TEDS-Structure) is pure structure table edit distance similarity, TEDS and TEDS-S are test scores on regular document tables, M-TEDS (Modified TEDS) is modified table edit distance similarity, M-TEDS-S (Modified TEDS-Structure) is modified pure structure table edit distance similarity, and M-TEDS and M-TEDS-S are test scores on the mask-enhanced test set. The results show that even when the prior skeleton predicted by the Structure Proposal Network (SPN) is noisy (imperfect prediction), the TEDS score (90.45%) of the document table intelligent recognition method SA-Table (Structure-Aware Table Recognition Model) provided in this application embodiment is still much higher than the result of directly using TSR prediction (70.85%). Other scores are also better, indicating that the document table intelligent recognition method provided in this application embodiment can dynamically balance visual features and structural priors, and has the ability to self-correct and refine the skeleton.
[0059] The document table intelligent recognition method provided in this application effectively overcomes the defects of multi-stage error accumulation and end-to-end black boxing. Through multi-technology collaboration, it simultaneously improves the efficiency, accuracy, and robustness of table recognition: Using a dual-stream prior injection architecture, the side encoder extracts the geometric skeleton features of the table and injects them layer by layer into the VLM decoder. Combined with a hierarchical learnable scalar, it achieves dynamic fusion of semantic and geometric features, balancing end-to-end efficiency and physical row and column reference, thus improving the robustness of complex table recognition. Freezing the SPN, combined with visual rendering and error calibration, ensures structural prior stability while calibrating structural prediction errors. Through spatial... - Semantic joint modeling outputs content tokens and cell normalized coordinates synchronously, using coordinate regression as an aid to force the decoder to focus on physical location, solving the cascading misalignment problem and ensuring accurate alignment between structured sequences and table physical areas; Explicit empty cell modeling combines training with empty cells and geometric cues, introducing dedicated placeholder tokens to decouple the existence of structure and content, eliminating the "empty column illusion," and adapting to rigorous scenarios such as finance and government affairs; The lightweight design of the side encoder, combined with frozen SPN, balances parameter efficiency and geometric feature extraction, reduces training costs and overfitting risks, improves convergence speed, and facilitates industrial deployment.
[0060] A second aspect of this application provides a document table intelligent recognition system. Figure 2 The diagram shows a structural block diagram of the document table intelligent recognition system provided in an embodiment of this application, such as... Figure 2 As shown, the document table intelligent recognition system of the second embodiment of this application includes:
[0061] SPN module 201 is used to generate a row and column grid from the original table image using a pre-trained frozen structure proposal network;
[0062] Rendering module 202 is used to overlay the row and column grids onto the original table image in the form of line segments to generate a structure enhancement diagram. The line segments are of a preset color;
[0063] Visual encoder 203 is used to extract features from the original table image to obtain main visual features containing text region information and visual semantic information. ;
[0064] Side encoder 204 is used to extract the geometric topological features of the structure enhancement map. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position;
[0065] Decoder 205, wherein the decoder is an autoregressive Transformer decoder, and each Transformer decoding layer of the decoder is followed by a cross-attention submodule, the decoder being used for the first... The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , Let N be an integer, where N is the total number of Transformer decoding layers, and the number of layers is used to calculate the [number of layers]. Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer. Presentation layer normalization operation, The initial value of the hidden state of the Transformer decoding layer is a hierarchically learnable scalar. , Represents word embedding function, Represents the historical true target sequence, Indicates the current time;
[0066] Multi-task position-aware output head 206, used for the first The output of the Transformer decoding layer is taken as input, and the output is the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates of the cell.
[0067] The visual encoder 203, decoder 205, and multi-task position-aware output head 206 belong to a large visual language model. Training the large visual language model simultaneously trains the visual encoder 203, decoder 205, and multi-task position-aware output head 206. The decoder 205 includes multiple stacked Transformer decoding layers. The Nth Transformer decoding layer is connected to the multi-task position-aware output head 206, where N is the total number of Transformer decoding layers. Each Transformer decoding layer includes a self-attention submodule, a cross-attention submodule, and a feedforward network submodule connected sequentially. The multi-task position-aware output head 206 can also belong to the decoder 205, connected to the Nth Transformer decoding layer. After the Transformer decoding layer, the side encoder 204 is connected to the decoder 205.
[0068] In the calculation of the side encoder 204, By embedding the location Perform 2D bilinear interpolation generation. Align with the current input resolution. The position embedding matrix is learned during the training phase of the visual language large model. The position corresponding to the position embedding matrix is the coordinate position of each patch in the two-dimensional patch grid after the fixed resolution input image is divided into two-dimensional patch grids of the same size as the fixed resolution input image during the training phase of the visual language large model. The current input resolution is the spatial resolution of the two-dimensional patch grids of the same size as the original table image obtained by dividing the original table image.
[0069] In one possible implementation, the decoder 205 is further configured to: when the main visual feature There are regions without text, and their geometric structure and topological features are present. A grid structure exists in the textless region, based on a conditional probability distribution. A placeholder token is generated at the position corresponding to the textless area. The placeholder token is a marker predefined in the vocabulary.
[0070] In one possible implementation, the document table intelligent recognition system of this application further includes a training module. This training module is used to pre-train the large visual language model based on minimizing the negative log-likelihood loss to obtain the optimal parameters of the large visual language model. The optimization objective of the training is to minimize the negative log-likelihood loss, and the negative log-likelihood loss function is... , Used to measure how well the sequence generated by the model fits the real target sequence. The true target sequence is defined as a sequence of table HTML tags and a sequence of cell content, where |Y| represents the total length of the true target sequence. Let represent the conditional probability distribution of the visual language big data model, and let represent the predicted probability of the visual language big data model outputting a token at the current position. This refers to the real token at position t in the real target sequence, where t is the time step index of the real target sequence generation, and its value ranges from 1 to t to |Y|. This represents the generated real target sequence up to step t-1. ,by For the context information of the current prediction, These are all the learnable parameters of the large visual language model.
[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the device described above can be referred to the corresponding process in the foregoing method embodiments, and therefore will not be repeated here.
[0072] The document table intelligent recognition system provided in this application incorporates a dual-stream structure prior into its architecture. It employs a side encoder to extract geometric features and injects them layer by layer into the VLM decoder, dynamically fusing semantic and geometric features to improve the robustness of complex table recognition. A frozen SPN combined with visual rendering and error calibration ensures structural prior stability while calibrating errors. Spatial-semantic joint modeling enables the synchronous output of content tokens and cell normalized coordinates, using coordinate regression as an aid to solve the cascading misalignment problem in table recognition and ensure alignment between the structure and physical regions. Explicit empty cell modeling, combined with training and geometric cues, introduces dedicated placeholders to decouple structure and content, effectively eliminating the "empty column illusion" and adapting to rigorous scenarios. The lightweight side encoder, combined with frozen SPN, balances parameter efficiency and geometric extraction, reducing training costs and overfitting risks. The document table intelligent recognition system provided in this application, through multi-technology collaboration, effectively improves the efficiency, accuracy, and robustness of table recognition.
[0073] It should be noted that the document and table intelligent recognition system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of this application can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of this application are only for distinguishing the various modules or steps and are not considered as an improper limitation of this application.
[0074] A third aspect of the embodiments of this application also provides an electronic device, the electronic device comprising:
[0075] At least one processor; and,
[0076] A memory communicatively connected to at least one of the processors; wherein,
[0077] The memory stores instructions that can be executed by the processor to implement the above-described document table intelligent recognition method.
[0078] A fourth aspect of this application also provides a computer-readable storage medium storing computer instructions that are executed by the computer to implement the above-described document table intelligent recognition method.
[0079] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0081] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.
[0082] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.
[0083] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.
[0084] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for intelligent recognition of document tables, characterized in that, include: Using a pre-trained, frozen structure proposal network, row and column grids are generated from the original table image; The row and column grids are superimposed on the original table image as line segments to generate a structure enhancement diagram. The line segments are of a preset color; The original table image is processed using a visual encoder based on a large visual language model to extract features, obtaining main visual features that include text region information and visual semantic information. ; The geometric topological features of the structure enhancement map are extracted using a side encoder. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position; In each Transformer decoding layer of the decoder in the large visual language model, a cross-attention submodule is set after the self-attention submodule. The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , N is an integer, where N is the total number of Transformer decoding layers, and the decoder is an autoregressive Transformer decoder. Calculate the first Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer. , Presentation layer normalization operation, The initial value of the hidden state of the Transformer decoding layer is a hierarchically learnable scalar. , Represents word embedding function, Represents the historical true target sequence, Indicates the current time; With the first The output of the Transformer decoding layer is the input of the multi-task position-aware output head of the decoder. The multi-task position-aware output head outputs the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell.
2. The document table intelligent recognition method according to claim 1, characterized in that, By using the position embedding matrix Perform 2D bilinear interpolation generation. Align with the current input resolution. The position embedding matrix is learned during the training phase of the visual language large model. The position corresponding to the position embedding matrix is the coordinate position of each patch in the two-dimensional patch grid after the fixed resolution input image is divided into two-dimensional patch grids of the same size as the fixed resolution input image during the training phase of the visual language large model. The current input resolution is the spatial resolution of the two-dimensional patch grids of the same size as the original table image obtained by dividing the original table image.
3. The document table intelligent recognition method according to claim 1, characterized in that, Also includes: The visual language model is pre-trained based on minimizing the negative log-likelihood loss to obtain its optimal parameters. The optimization objective of training the visual language model is to minimize the negative log-likelihood loss, and the negative log-likelihood loss function is: , Used to measure how well the sequence generated by the model fits the real target sequence. The true target sequence is defined as a sequence of table HTML tags and a sequence of cell content, where |Y| represents the total length of the true target sequence. Let represent the conditional probability distribution of the visual language big data model, and let represent the predicted probability of the visual language big data model outputting a token at the current position. This refers to the real token at position t in the real target sequence, where t represents the time step index of the real target sequence generation, and its value ranges from 1 to t to |Y|. This represents the generated real target sequence up to step t-1. ,by For the context information of the current prediction, These are all the learnable parameters of the large visual language model.
4. The document table intelligent recognition method according to claim 3, characterized in that, The document table intelligent recognition method also includes: When the main visual features There are regions without text, and their geometric structure and topological features are present. A grid structure exists in the textless region, and the decoder is based on a conditional probability distribution. A placeholder token is generated at the position corresponding to the textless area. The placeholder token is a marker predefined in the vocabulary.
5. A document table intelligent recognition system, characterized in that, include: The SPN module is used to generate a row and column grid from the original table image using a pre-trained, frozen structure proposal network. The rendering module is used to overlay the row and column grids onto the original table image as line segments to generate a structure-enhanced diagram. The line segments are of a preset color; A visual encoder is used to extract features from the original table image to obtain main visual features that include text region information and visual semantic information. The visual encoder belongs to the large visual language model; A side encoder is used to extract the geometric topological features of the structure enhancement map. ,in, Embedded for patch, This represents a linear projection function used to map features to a feature space consistent with the feature dimensions of the decoder in a large visual language model. This represents a pre-trained structure encoding network used to extract the geometric and topological features of the table from the structure enhancement map. Encode the interpolation position; The decoder is an autoregressive Transformer decoder. Each Transformer decoding layer of the decoder is followed by a cross-attention submodule. The decoder is used for... The first cross-attention submodule is the first Hidden state of the Transformer decoding layer As the query vector, based on the geometric topological features Given a key vector and a value vector, calculate the first... The output of each attention submodule ,in, , and It is a learnable linear transformation matrix. Let be the dimension of the key vector. This refers to the number of layers in the Transformer decoding layer. , Let N be an integer, where N is the total number of Transformer decoding layers, and the number of layers is used to calculate the [number of layers]. Output of the Transformer decoding layer and with For the first The hidden state of the Transformer decoding layer. , Presentation layer normalization operation, The initial value of the hidden state of the Transformer decoding layer is a hierarchically learnable scalar. , Represents word embedding function, Represents the historical true target sequence, This indicates that at the current moment, the decoder belongs to the large visual language model; Multi-task position-aware output head, used for... The output of the Transformer decoding layer is the input, and the output is the structured representation of the original table image. The structured representation includes table structure information, cell text content, and cell spatial location information. The table structure information is represented in the form of structure tags. The cell text content includes normal text and empty cell identifiers. The spatial location information is the bounding box coordinates corresponding to the cell. The multi-task position-aware output head belongs to the visual language large model.
6. The document table intelligent recognition system according to claim 5, characterized in that, By embedding the location Perform 2D bilinear interpolation generation. Align with the current input resolution. The position embedding matrix is learned during the training phase of the visual language large model. The position corresponding to the position embedding matrix is the coordinate position of each patch in the two-dimensional patch grid after the fixed resolution input image is divided into two-dimensional patch grids of the same size as the fixed resolution input image during the training phase of the visual language large model. The current input resolution is the spatial resolution of the two-dimensional patch grids of the same size as the original table image obtained by dividing the original table image.
7. The document table intelligent recognition system according to claim 5, characterized in that, It also includes a training module, which is used to pre-train the visual language large model based on minimizing the negative log-likelihood loss to obtain the optimal parameters of the visual language large model. The optimization objective of training the visual language large model is to minimize the negative log-likelihood loss, and the negative log-likelihood loss function is: , Used to measure how well the sequence generated by the model fits the real target sequence. The true target sequence is defined as a sequence of table HTML tags and a sequence of cell content, where |Y| represents the total length of the true target sequence. Let represent the conditional probability distribution of the visual language big data model, and let represent the predicted probability of the visual language big data model outputting a token at the current position. This refers to the real token at position t in the real target sequence, where t is the time step index of the real target sequence generation, and its value ranges from 1 to t to |Y|. This represents the generated real target sequence up to step t-1. ,by For the context information of the current prediction, These are all the learnable parameters of the large visual language model.
8. The document table intelligent recognition system according to any one of claims 5-7, characterized in that, The decoder is also used for: When the main visual features There are regions without text, and their geometric structure and topological features are present. A grid structure exists in the textless region, based on a conditional probability distribution. A placeholder token is generated at the position corresponding to the textless area. The placeholder token is a marker predefined in the vocabulary.
9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor to implement the document table intelligent recognition method as described in any one of claims 1-4.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by the computer to implement the document table intelligent recognition method as described in any one of claims 1-4.