Table literal structure recognition method based on heterogeneous architecture

Abstract

The embodiment of the present disclosure provides a table text structure identification method based on a heterogeneous architecture, which comprises the steps of table regularization, table preprocessing, text detection, field extraction and text recognition. The method can perform structured identification on table text pictures, and is mainly aimed at pictures taken by a camera and the like, rather than digitally generated pictures generated by an electronic device. The taken pictures often increase the difficulty of text recognition due to the shooting environment and shooting method. The structured identification refers to extracting important text information with logic from the special structure of the table, and removing other useless information, such as value-added tax invoices and taxi invoices.

Description

Technical Field

[0001] This invention belongs to the field of FPGA image processing, and in particular relates to a method for structured recognition of table text based on heterogeneous architecture. Background Technology

[0002] Structured text recognition of tables refers to the process of converting paper tables into text images using optical devices such as scanners or cameras, extracting text information from these images, establishing structural relationships between text, and extracting the key information needed. This task relies on OCR technology, which converts image information into text information. However, the unique aspect lies in the fact that the text in tables contains unique structural information, such as different attributes like table headers, units, and seals. Different table items also have their corresponding text content. Structured text recognition of tables requires structural analysis of the table format, including the spatial and logical layout of the text, with the ultimate goal of extracting the text content from key areas of the image. However, in real-world images, the complex layout, misaligned spatial arrangement, and complex imaging environments often increase the difficulty of structured recognition. Furthermore, the rapid development of deep learning methods has driven the development of text detection and OCR technologies, laying the foundation for structured recognition technology. However, the dramatic increase in algorithm scale has made efficiency a problem. Processing using a CPU+FPGA heterogeneous architecture can effectively solve this problem. On the other hand, the emergence of high-level synthesis tools like HLS has also provided convenience for development and debugging. Summary of the Invention

[0003] In view of this, this paper provides a method for structured recognition of table text based on heterogeneous architecture. This method extracts important text information from table text contained in images taken in natural environments or scanned by a scanner by performing rule-based processing, preprocessing, text detection, field extraction, and text recognition. This method converts table text images into text, thereby improving the efficiency of extracting text information from images.

[0004] A method for structured text recognition in tables based on heterogeneous architecture, the method comprising:

[0005] Step 1: Obtain the first image to be processed and the corresponding type information of the image, and pre-create a table template to place the first image (0);

[0006] The first image is adjusted using a convolutional neural network model until it is horizontally readable, and if distortion exists in the first image, it is corrected.

[0007] Step 2: The first image is preprocessed, including at least shadow removal, stamp removal, image enhancement, and table border removal.

[0008] Step 3: Text detection. Multiple text regions in the first image are detected using the Convolutional Neural Network (CNN) method. It is determined whether the distance between any two text regions is less than a preset value. If so, the adjacent text regions are concatenated; otherwise, no processing is performed.

[0009] Step 4: All the text regions are processed by a recurrent neural network model to extract fields. Before extraction, the recurrent neural network is preset with text content settings, and multiple second images are output.

[0010] Step 5: Extract all text information from the second image.

[0011] Beneficial effects:

[0012] This invention includes a detailed recognition process and employs deep learning methods to improve the algorithm's accuracy and robustness. Deep learning transforms the extraction of important text from a table into intelligent extraction, converting a natural language processing problem into an image processing problem. A heterogeneous architecture of CPU+FPGA is used to deploy the algorithm. The FPGA portion handles the most computationally intensive neural network model operations, while the CPU portion handles other simpler processing and control scheduling. Furthermore, the algorithm running on the FPGA is written in C / C++, and HLS tools are used for debugging and development on the FPGA, significantly shortening the development and debugging cycles and making it more user-friendly for software designers. Because FPGAs have higher concurrency than CPUs, the deep learning methods involved in this invention are developed using Vivado's HLS tool. The deep learning steps are distributed and processed on the FPGA, while other parts are processed on the CPU, which acts as the host and is responsible for the transmission and scheduling of the entire process. This invention employs a parallel pipeline design, fully utilizing the parallel characteristics of FPGAs, and the use of HLS tools further reduces development and debugging cycles. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the overall process of the present invention, in which the gray box represents the part deployed on the FPGA and the rest deployed on the CPU;

[0015] Figure 2 This is a diagram illustrating template alignment for VAT invoice-type form images within a standardized workflow.

[0016] Figure 3 This is a schematic diagram of the recurrent neural network model in the field extraction process;

[0017] Figure 4 Displaying the structured recognition results for VAT invoice-type table images. Detailed Implementation

[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0021] The table text structure recognition method based on heterogeneous architecture of the present invention includes the following steps:

[0022] Step 1: Obtain the first image to be processed and its corresponding type information, and pre-create a table template for the first image. Adjust the first image using a convolutional neural network model until it is horizontally readable. If distortion exists in the first image, correct it. Specifically, for example, place the image in the center of the template and correct table distortion in the image. If distortion exists, determine if it does; if so, use template matching algorithms for correction. Table template regularization processing includes steps such as table classification, table rotation, and table template alignment. Table classification refers to classifying the tables contained in the image, such as VAT invoices, taxi receipts, medical receipts, etc. Rotation refers to rotating the image until the tables are horizontally readable. The table template processes the image using traditional image processing methods, matching the image with the template and correcting any table distortion in the image.

[0023] Step 2: Preprocess the first image, including at least shadow removal, stamp removal, image enhancement, and table border removal. Table preprocessing involves using traditional image processing methods to quickly preprocess the table and improve the accuracy of subsequent steps. This includes preprocessing methods such as shadow removal, stamp removal, table border extraction, and image enhancement, all deployed on the CPU.

[0024] Step 3: Text detection. Multiple text regions in the first image are detected using the Convolutional Neural Network (CNN) method. If the distance between any two text regions is less than a preset value (e.g., 5mm-10mm), the adjacent text regions are concatenated; otherwise, no processing is performed.

[0025] Step 4: All text regions are processed by a recurrent neural network model to extract fields. Before extraction, the recurrent neural network is pre-set with text content (e.g., invoice tax number, taxpayer, etc.), and multiple second images are output.

[0026] Step 5: Extract all text information from the second image.

[0027] As a specific implementation method provided in this case, the original drawing is photographed in a natural environment or scanned by a scanner to obtain a first image. The first image contains tabular text. The tabular text in the first image is identified and extracted, and pre-set text information, such as useless information like the paper number of an invoice, is removed.

[0028] As a specific implementation method provided in this case, both the convolutional neural network model and the recurrent neural network model deep learning algorithm are used to recognize and detect multiple text regions on the FPGA through the Vivado HLS (FPGA development tool) development tool, while the remaining steps are developed and deployed on the CPU side.

[0029] As a specific implementation method provided in this case, the convolutional neural network model classifies and rotates the table in the first image, and processes it according to the classification method, wherein:

[0030] The output of the convolutional neural network model for processing the first image includes a first category and a second category. The first category represents the image type, such as a value-added tax invoice, a taxi receipt, or a medical receipt. The second category represents the image rotation angle. The processing of the first image by the convolutional neural network model is performed on an FPGA (hardware with the same function as a CPU).

[0031] As a specific implementation method provided in this case, step 3, text detection, includes:

[0032] The regularized and preprocessed table text image is input into a convolutional neural network model to obtain text regions. Each text region is represented by a set of four-bit coordinates. Each set of coordinates represents the coordinates of a specified position in a text region (top left corner) and the corresponding length and width of the text region. The convolutional neural network model is trained on the first image using an algorithm including at least CTPN, SegLink, TextBoxes, PSENet, Mask TextSpotter, and CRAFT. The output results are merged and segmented to obtain multiple text regions.

[0033] As a specific implementation method provided in this case, all coordinates are sorted to form a coordinate sequence, and the coordinate sequence is input into a recurrent neural network model to filter and obtain all text regions;

[0034] The recurrent neural network consists of a bidirectional LSTM layer and a linear classifier layer. The output is a set of node classification results, representing the category of each node in the input coordinate sequence. Nodes in the coordinate sequence are selected based on their categories, and the corresponding text regions are then identified. Extracting text information from all the second images involves using one of the following deep learning text recognition methods: EATEN, CRNN, or CRAFT.

[0035] This example uses a common PC as the CPU host, with an Ubuntu 16.04 operating system. The FPGA uses the JFM7VX690T development kit and is connected to the host via a PCIe port. The test image is a VAT invoice table image, and the result is the text information of important fields such as invoice code, seller name, and seller identification contained in the invoice.

[0036] like Figure 1 The overall flowchart shown in this example includes the following steps:

[0037] Step 1: Table regularization, including table classification, table rotation, and table template alignment. Table classification and rotation use a convolutional neural network model to classify the image. The model output consists of two parts: one representing the image category (in this example, VAT invoice), and the other representing the image rotation angle (e.g., 0 degrees, 90 degrees, 180 degrees, and 270 degrees). This rotation brings the image to a horizontally readable state. This part of the model is deployed on an FPGA. Table template alignment uses traditional image processing methods to match the image with a template, correcting distortions in the table within the image. Figure 2 The diagram shows the template matching process, which is deployed on the CPU.

[0038] Step 2: Table preprocessing. The table is quickly preprocessed using traditional image processing methods to improve the accuracy of subsequent steps. These methods include shadow removal, stamp removal, table border extraction, and image enhancement, and are deployed on the CPU.

[0039] Step 3: Text Detection. The regularized and preprocessed table text image is input into the text detection model. The result is represented by a set of 4-bit coordinates. Each set of coordinates represents the coordinates of the top-left corner of each text region in the image, as well as the length and width of the text region. The text detection model is trained on a dataset using text detection algorithms including but not limited to CTPN, SegLink, TextBoxes, PSENet, Mask TextSpotter, and CRAFT. The model is deployed on an FPGA. The model output is then post-processed through merging, segmentation, and other methods to obtain the final detection result.

[0040] Step 4: Field extraction. The final text detection results obtained in Step 3 are sorted according to their coordinate positions, resulting in a series of coordinate sequences. These sequences are used as input to the neural network model. The recurrent neural network consists of a bidirectional LSTM layer and a linear classifier layer, such as... Figure 3 The model diagram shown outputs a set of node classification results, representing the category of each node in the input coordinate sequence. Nodes in the coordinate sequence are selected based on their categories, thus identifying each text detection region. Figure 4 The image shows a graphical representation of the filtered text detection region on the original image, which is deployed on an FPGA.

[0041] Step 5: Perform character recognition on the important text regions extracted in Step 4. Character recognition methods include, but are not limited to, deep learning character recognition methods such as EATEN, CRNN, and CRAFT. The final result is a logically consistent text extracted from the table text image, such as... Figure 4 The text marked under each red box is the result of the table text structure recognition.

[0042] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for structural text recognition in tables based on heterogeneous architecture, characterized in that, Includes the following steps: Step 1: Obtain the first image to be processed and its corresponding type information, and pre-create a table template for placing the first image; The first image is adjusted using a convolutional neural network (CNN) model to achieve a horizontally readable state. If distortion exists in the first image, it is corrected. Both the CNN and recurrent neural network (RNN) deep learning algorithms are implemented on an FPGA using the Vivado HLS development tool for recognizing and detecting multiple text regions. The CNN model also classifies and rotates tables in the first image, processing them according to their classification. The output of the CNN model on the first image includes a first category and a second category, where the first category represents the image type and the second category represents the image rotation angle. The processing of the first image by the CNN model is performed on the FPGA. Step 2: The first image is preprocessed, including at least shadow removal, stamp removal, image enhancement, and table border removal. Step 3: Text detection. Multiple text regions in the first image are detected using a Convolutional Neural Network (CNN). It is determined whether the distance between any two text regions is less than a preset value. If so, the adjacent text regions are concatenated; otherwise, no processing is performed. The regularized and preprocessed table text image is input into the convolutional neural network model to obtain the text region. The text region is represented by a set of four-bit coordinates, each set of coordinates representing the coordinates of a specified position in the text region and the length and width of the text region. The convolutional neural network model is trained on the first image by an algorithm including at least CTPN, SegLink, TextBoxes, PSENet, Mask TextSpotter, and CRAFT. The output results are merged and segmented to obtain multiple text regions. All coordinates are sorted to form a coordinate sequence, which is then input into the recurrent neural network model to filter out all text regions. The recurrent neural network consists of a bidirectional LSTM structure and a linear classifier, and outputs a set of node classification results, which represent the category of each node in the input coordinate sequence. Based on the category, the nodes in the coordinate sequence are filtered out, and the corresponding text regions are selected. Step 4: All the text regions are processed by a recurrent neural network model to extract fields. Before extraction, the recurrent neural network is preset with text content settings, and multiple second images are output. Step 5: Extract all text information from the second image.

2. The method for structured text recognition in tables based on heterogeneous architecture according to claim 1, characterized in that, The first image is obtained by taking a picture of the original pattern in a natural environment or by scanning it with a scanner. The first image contains table text. The table text in the first image is identified and extracted, and the text information set by the preset is removed.

3. The method for structured text recognition in tables based on heterogeneous architecture according to claim 1, characterized in that, Extracting all text information from the second image involves using one of the following deep learning text recognition methods: EATEN, CRNN, or CRAFT.

Images