Unified method for end-to-end text reading and information extraction
By integrating text reading and information extraction tasks through the unified end-to-end neural network architecture of TRIE, the problem of collaborative optimization of text reading and information extraction tasks is solved, achieving efficient and accurate document understanding, applicable to diverse document types, and reducing system complexity and cost.
Patent Information
- Application Number
- CN202512018540.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-21
AI Technical Summary
In existing technologies, text reading and information extraction tasks are isolated from each other and lack collaborative optimization, resulting in information loss and performance bottlenecks. Moreover, most models are designed for a single task, which increases system complexity and maintenance costs, and makes it difficult to adapt to diverse document types and layouts.
The TRIE architecture, a unified end-to-end neural network, integrates text reading and information extraction tasks. By jointly training and leveraging the complementarity of the two tasks, it combines deep convolutional neural network Mask R-CNN, recurrent neural network CRNN, graph neural network GCN, and attention mechanism to achieve text detection, recognition, and extraction of key entity relationships.
It significantly improves the accuracy and efficiency of understanding rich visual documents, simplifies the modeling process, reduces development and maintenance costs, has strong generalization capabilities, is suitable for diverse document types, and has a fast processing speed to meet the needs of large-scale document processing.
Smart Images
Figure CN121904785A_ABST
Abstract
Description
[Technical Field]
[0001] This invention relates to the field of computer vision and natural language processing, specifically to a unified method for end-to-end text reading and information extraction. [Background Technology]
[0002] With the acceleration of digital transformation, rich visual documents are increasingly used in business, administration, and academia. These documents not only contain a large amount of textual information but also integrate complex visual elements (such as tables, headings, and signatures). Their layouts and formats are diverse, ranging from fixed layouts (such as invoices) to variable layouts (such as resumes), and from structured to semi-structured text, carrying rich semantic information. Automatically understanding and extracting this information is crucial for improving information processing efficiency.
[0003] Traditional document understanding methods typically decompose the problem into two independent stages: text reading and information extraction. The text reading stage uses Optical Character Recognition (OCR) technology for text detection and recognition, with common methods including the CRNN framework based on Convolutional Neural Networks (CNNs) and improved models with attention mechanisms. The information extraction stage extracts key entities (such as dates and amounts) and relationships from the recognized plain text, relying on Natural Language Processing (NLP) techniques such as Named Entity Recognition (NER) and relation extraction. However, existing methods have significant shortcomings: First, the text reading and information extraction tasks are isolated from each other, lacking collaborative optimization, leading to information loss and performance bottlenecks. Second, most models are designed for a single task, requiring a multi-stage pipeline strategy, increasing system complexity and maintenance costs. Furthermore, the diversity of document types and layouts (such as different industry invoice formats and personalized resume designs) places higher demands on the model's generalization ability, making it difficult for existing single-task models to adapt to large-scale, diverse real-world application scenarios.
[0004] Therefore, the industry urgently needs a unified framework capable of simultaneously handling text reading and information extraction within a single model to achieve collaborative optimization between tasks, simplify the modeling process, improve processing efficiency, and enhance adaptability to diverse documents. This invention addresses these technical challenges by proposing an innovative solution. [Summary of the Invention]
[0005] This invention overcomes the shortcomings of existing technologies and provides a unified method for end-to-end text reading and information extraction, which is named TRIE (Text Reading and Information Extraction) in this case. TRIE integrates text reading and information extraction tasks by designing a unified end-to-end neural network architecture. It aims to fully utilize the complementarity of the two tasks through joint training, thereby significantly improving the accuracy and efficiency of understanding visually rich documents.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A unified method for end-to-end text reading and information extraction, characterized by: including...
[0008] S1. Input processing: Input a rich visual document image, which contains various text information and visual elements.
[0009] S2, Text Reading: The text reading module is based on deep convolutional neural network Mask R-CNN, recurrent neural network CRNN and attention mechanism to detect and recognize text regions in visually rich document images and generate visual features and text features.
[0010] S3. Information Extraction: The information extraction module is based on Graph Neural Network (GCN) and TextualContext Block (TextualContext Block) to locate and extract key entities and relationships, and output structured information.
[0011] S4. Joint optimization: The network is optimized by combining the detection and recognition loss of text reading with the classification and regression loss of information extraction.
[0012] S5. Output Results: Based on user instructions, output the coordinates, content, extracted entities, and relationships of the text area.
[0013] The unified method for end-to-end text reading and information extraction described above is characterized in that: the size of the rich visual document image in S1 is 2666×1600 pixels; the rich visual document image includes invoices, resumes, tickets or flyers; and the visual elements include tables, titles or signatures.
[0014] The unified method for end-to-end text reading and information extraction described above is characterized in that: S2 includes...
[0015] S21. Text detection: The text reading module uses a deep convolutional neural network Mask R-CNN to locate text regions from visually rich document images and generate visual features.
[0016] S22. Text recognition: The text reading module combines a recurrent neural network (CRNN) and an attention mechanism to identify the text features that distinguish the detected text.
[0017] The unified method for end-to-end text reading and information extraction described above is characterized in that: in S21, the text reading module crops a 32×512 pixel image block from the located text region.
[0018] The unified method for end-to-end text reading and information extraction described above is characterized in that: the text reading module in S2 uses multiple rich visual document images covering different document types and whose data labels include text location and content for pre-training and fine-tuning optimization.
[0019] The unified method for end-to-end text reading and information extraction described above is characterized in that: S3 includes...
[0020] S31. Feature Fusion: The information extraction module fuses visual and textual features and models the spatial and semantic relationships between texts using a graph neural network (GCN).
[0021] S32. Multi-task processing: The information extraction module locates, identifies, and extracts key entities, and extracts key entity relationships based on the Textual Context Block, outputting structured information.
[0022] The unified method for end-to-end text reading and information extraction described above is characterized in that: the visual features in S21 and S31 include text bounding boxes and layout information; and the text features in S22 and S31 include character sequences.
[0023] The unified method for end-to-end text reading and information extraction described above is characterized in that: the key entities in S32 include date, amount or name; the key entity relationships include the relationship between date and amount.
[0024] The unified method for end-to-end text reading and information extraction described above is characterized in that: the structured information in S32 is entity-relation pairs in JSON format.
[0025] The unified method for end-to-end text reading and information extraction described above is characterized in that: the information extraction module in S3 adopts a full parameter fine-tuning strategy and is trained by weight adjustment based on a dataset of multiple labeled, different document types and domain knowledge-rich visual document images.
[0026] The beneficial effects of this invention are:
[0027] This invention offers several advantages: Firstly, it improves performance. Through end-to-end training, TRIE fully leverages the synergistic effect of text reading and information extraction, significantly enhancing document understanding accuracy. Experimental results show that its F1 score on various datasets outperforms traditional methods. Secondly, it simplifies modeling. The unified framework eliminates the complexity of multi-stage pipeline modeling, reducing development and maintenance costs. Thirdly, it exhibits strong generalization capabilities, adapting to diverse document types and layouts, making it suitable for various scenarios such as invoice processing, resume parsing, and contract analysis. Fourthly, it boasts high processing efficiency. TRIE achieves an inference speed of 1.76 frames per second on a single device, surpassing traditional pipeline methods and meeting the demands of large-scale document processing. Finally, this invention has broad application prospects in areas such as office automation, financial management, document digitization, and intelligent information extraction, providing a novel technological path for the automated understanding of visually rich documents. [Image Description]
[0028] Figure 1 This is a flowchart of the end-to-end training and testing process of the present invention;
[0029] Figure 2 This is a diagram of the TRIE model architecture of the present invention;
[0030] Figure 3 This is a flowchart of the information extraction module of the present invention. [Detailed Implementation]
[0031] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.
[0032] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of the components in a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. Furthermore, descriptions involving "preferred," "second-best," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "preferred" or "second-best" may explicitly or implicitly include at least one of those features.
[0033] like Figure 1-3 As shown, a unified method for end-to-end text reading and information extraction includes:
[0034] S1. Input processing: Input a rich visual document image. The rich visual document image contains various text information and visual elements. The size of the rich visual document image is 2666×1600 pixels. The rich visual document image includes invoices, resumes, tickets or flyers, etc. The visual elements include tables, titles or signatures, etc.
[0035] S2, Text reading, such as Figure 2 As shown, the text reading module, based on a deep convolutional neural network Mask R-CNN, a recurrent neural network CRNN, and an attention mechanism, detects and identifies text regions in visually rich document images, generating visual and text features; specifically, S2 includes...
[0036] S21. Text detection: Utilizing the improved deep convolutional neural network Mask R-CNN framework, text regions are located from high-resolution images (2666×1600 pixels), generating accurate text bounding boxes, supporting documents with complex layouts.
[0037] S22. Text recognition: Character sequence recognition is performed on the detected text region (image block cropped to 32×512 pixels). By combining the recurrent neural network (CRNN) framework and attention mechanism, the problem of recognizing long text and irregular layout is solved.
[0038] The text reading module is trained using a diverse dataset containing approximately 500,000 images, covering document types such as invoices, receipts, resumes, and flyers. Data labels include text location and content, ensuring the model's adaptability to different scenarios. Through pre-training and fine-tuning, this module significantly improves the robustness and accuracy of text detection and recognition, providing high-quality input for subsequent information extraction tasks.
[0039] S3, Information Extraction, such as Figure 3 As shown, the information extraction module, based on Graph Neural Network (GCN) and Textual Context Block (TLC), locates and extracts key entities and relationships, outputting structured information; specifically, S3 includes...
[0040] S31. Feature fusion: The visual features (layout information of text regions) and text features (character sequences) provided by the text reading module are fused together, and the spatial and semantic relationships between texts are modeled through GCN.
[0041] S32, multi-task processing, supports named entity recognition (NER, such as extracting date, amount, and name) and relation extraction (such as the "date-amount" relationship in an invoice), enhances context modeling capabilities through a multi-head attention mechanism, and outputs structured information (such as entity-relation pairs in JSON format).
[0042] The information extraction module employs a full-parameter fine-tuning strategy during training, adjusting model weights based on a labeled, rich visual document dataset to adapt to different document types and domain knowledge. Through joint training with the text reading module, this module leverages the visual and semantic information of the documents to improve the accuracy of information extraction.
[0043] S4. Joint optimization: The network is optimized by combining the detection and recognition loss of text reading with the classification and regression loss of information extraction.
[0044] S5. Output Results: Based on user instructions, output the coordinates, content, extracted entities, and relationships of the text area.
[0045] The technical solution of this invention includes a unified end-to-end network architecture: a multimodal fusion framework is designed to combine the visual features (based on image layout and text region information) and text features (character sequence information) of the text reading module with the semantic analysis capabilities of the information extraction module. Through a feature sharing mechanism, the model completes text detection, recognition, and information extraction in a single forward propagation, reducing computational redundancy. Task collaborative optimization: through a joint loss function, the semantic information of the information extraction module provides back-supervision for the optimization of the text reading module, improving the accuracy of text detection and recognition; simultaneously, the visual and text features of the text reading module provide richer contextual information for information extraction, enhancing the extraction effect of entities and relationships. Multi-task training strategy: a full-parameter fine-tuning method is adopted, based on a labeled dataset containing various document types (such as invoices, receipts, and resumes). The training data covers fixed and variable layouts, structured and semi-structured text, ensuring the model's generalization ability. Efficient inference mechanism: by optimizing the network structure and computational process, TRIE achieves fast inference while maintaining high accuracy, meeting the needs of real-time document processing.
[0046] The TRIE proposed in this invention provides an efficient and accurate unified solution for text reading and information extraction from visually rich documents through an end-to-end framework. Its core advantage lies in achieving collaborative optimization of text reading and information extraction through joint training, significantly improving overall performance and solving the task fragmentation problem of traditional pipeline methods. The unified model eliminates the complexity of multi-stage modeling, reducing development, deployment, and maintenance costs. Simultaneously, it achieves fast inference through feature sharing and computational optimization. Experiments show that it achieves 1.76 frames per second on a single Intel i7 device, outperforming the 1.13-1.69 frames per second of traditional methods. TRIE demonstrates excellent generalization ability on fixed and variable layouts, as well as structured and semi-structured documents. Its F1 score reaches 93.26% on the taxi invoice dataset and improves performance by approximately 1.5% in resume parsing, making it suitable for various scenarios such as invoice processing, resume parsing, and contract analysis. The model supports flexible hyperparameter tuning and dataset expansion, and is easy to adapt to new document types. Prototype system testing has verified its robustness and practicality in automated document understanding and intelligent information extraction. It provides an innovative technical route for office automation, financial management and archive digitization, and has broad commercial and academic application prospects.
[0047] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct or indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. A unified method for end-to-end text reading and information extraction, characterized in that: Including S1. Input processing: Input a rich visual document image, which contains various text information and visual elements. S2, Text Reading: The text reading module is based on deep convolutional neural network Mask R-CNN, recurrent neural network CRNN and attention mechanism to detect and recognize text regions in visually rich document images and generate visual features and text features. S3. Information Extraction: The information extraction module is based on Graph Neural Network (GCN) and Textual Context Block (Textual ContextBlock) to locate and extract key entities and relationships, and output structured information. S4. Joint optimization: The network is optimized by combining the detection and recognition loss of text reading with the classification and regression loss of information extraction. S5. Output Results: Based on user instructions, output the coordinates, content, extracted entities, and relationships of the text area.
2. The unified method for end-to-end text reading and information extraction according to claim 1, characterized in that: The rich visual document images in S1 are 2666×1600 pixels in size; rich visual document images include invoices, resumes, tickets or flyers; visual elements include tables, titles or signatures.
3. The unified method for end-to-end text reading and information extraction according to claim 1, characterized in that: S2 includes S21. Text detection: The text reading module uses a deep convolutional neural network Mask R-CNN to locate text regions from visually rich document images and generate visual features. S22. Text recognition: The text reading module combines a recurrent neural network (CRNN) and an attention mechanism to identify the text features that distinguish the detected text.
4. The unified method for end-to-end text reading and information extraction according to claim 3, characterized in that: The S21 text reading module will crop a 32×512 pixel image block from the located text area.
5. A unified method for end-to-end text reading and information extraction according to claim 1, characterized in that: The text reading module in S2 is pre-trained and fine-tuned using multiple rich visual document images covering different document types, with data labels including text location and content.
6. A unified method for end-to-end text reading and information extraction according to claim 3, characterized in that: S3 includes S31. Feature Fusion: The information extraction module fuses visual and textual features and models the spatial and semantic relationships between texts using a graph neural network (GCN). S32. Multi-task processing: The information extraction module locates, identifies, and extracts key entities, and extracts key entity relationships based on the Textual Context Block, outputting structured information.
7. A unified method for end-to-end text reading and information extraction according to claim 6, characterized in that: The visual features in S21 and S31 include text bounding boxes and layout information; the text features in S22 and S31 include character sequences.
8. A unified method for end-to-end text reading and information extraction according to claim 6, characterized in that: Key entities in S32 include dates, amounts, or names; key entity relationships include relationships between dates and amounts.
9. A unified method for end-to-end text reading and information extraction according to claim 6, characterized in that: The structured information in S32 is entity-relationship pairs in JSON format.
10. A unified method for end-to-end text reading and information extraction according to claim 1, characterized in that: The information extraction module in S3 employs a full-parameter fine-tuning strategy and is trained by adjusting the weights based on a dataset of rich visual document images with multiple labels, different document types, and domain knowledge.