Table region identification and extraction method and system for full-frame-line table

Abstract

The invention discloses a table region recognition and extraction method and system for a full-frame table, and the method comprises the steps: generating a table document image data set, constructing a deep learning model which comprises an image sharing coding layer, a column region recognition decoding layer and a table total region recognition decoding layer, a picture passes through an image sharing coding layer and then is respectively input into two branches of a column area identification decoding layer and a table area identification decoding layer to obtain masks of a table and a column area; determining the ordinate of each row separation line through the pixel level accumulated value of the table document image; and carrying out model training, inputting the table image into the trained model, carrying out identification extraction on the text content of the table area in the document image by using optical character identification, and displaying the text content in an Excel format. According to the method, the problems that the text content in the table area in the image and other non-structural documents is difficult to recognize and extract and the recognition and extraction effect of the text content contained in the table is poor due to low table structure recognition accuracy are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of table detection and table structure recognition, and in particular to a table area recognition and extraction method and system for a full-frame table. Background Art

[0002] The work of identifying and extracting text content in table areas mainly focuses on table area detection, table structure recognition, and text character content recognition. The current mainstream methods are as follows: 1. Use heuristic rules or simple machine learning algorithms, relying on visual information such as lines and text blocks obtained through image preprocessing and document analysis, or relying on some text information in PDF encoding to detect the area where the table is located and complete table structure recognition; 2. Use table detection as a specific application in the field of semantic segmentation or target detection, and use deep learning methods for table detection. Due to the special structure of the table, both rows and columns can be used as targets for detection, and separators between rows and columns can be used as targets for detection. Use deep learning methods such as target detection, image segmentation, and graph neural networks to identify table structures.

[0003] The disadvantages of the existing technology are as follows: (1) The table recognition method based on heuristic rules is relatively complex to design, and it is difficult to obtain high accuracy in table recognition in various scenarios, and the robustness is relatively poor; (2) The target detection and recognition of table rows and columns in the deep learning method has achieved a high recognition accuracy rate on the table structure, but for the text content contained in the table, the recognition effect of the whole row or column alone is not good; (3) At present, the research on table detection and structure recognition is mostly aimed at structured documents such as PDF or HTML, in which the position structure and text content of the table are displayed with corresponding codes. With the widespread use of mobile phones and scanners to capture and upload files, the need to extract tabular information in unstructured document images has become more urgent. This includes accurately detecting the table area in the image, and then detecting and extracting the information in the rows and columns of the detected table, making recognition and extraction more difficult. In addition, the final recognition results of the existing table detection and recognition methods can only be displayed in the form of accuracy or command line, which brings great inconvenience to operators. They can neither be used directly nor update or modify the table data on this basis. Summary of the invention

[0004] The purpose of the present invention is to overcome the technical defects of the prior art, solve the problems of low accuracy of table area detection and structure recognition in non-structured document images, poor recognition of text content contained in tables, and inability to save and display table area recognition and extraction results in the form of structured documents, and propose a table area recognition and extraction method and system for fully framed tables.

[0005] In order to solve the above technical problems, the present invention proposes a table area recognition and extraction method for a full-frame table, comprising the following steps:

[0006] S1, generate a table document image dataset;

[0007] S2. Construct an end-to-end deep learning model of TDEM, including an image sharing encoding layer, a column region recognition decoding layer, and a table total region recognition decoding layer composed of a VGG-19 pre-trained neural network. After the image passes through the image sharing encoding layer, it is input into two different branches of the column region recognition decoding layer and the table region recognition decoding layer to obtain masks of the table total region and column region.

[0008] S3, converting the images in the table document image data set into a horizontal projection distribution histogram of image pixels, and determining the vertical coordinate of each row separator line by the horizontal cumulative value of the pixels;

[0009] S4. Perform model training, input the table image into the trained model, use Tesseract optical character recognition to identify and extract the text content in the table area of ​​the document image, and display it in Excel format.

[0010] Furthermore, generating a table document image data set includes:

[0011] Convert documents containing full-frame tables from PDF / HTML format to jpg, png or bmp image formats, and collect and organize the converted document images into a full-frame table document image dataset.

[0012] Furthermore, in the TDEM end-to-end deep learning model, the image sharing coding layer is specifically:

[0013] The fully connected layers of VGG-19 are replaced with two convolutional layers, both of which use the ReLU activation function, followed by a dropout layer with a dropout rate of 0.8, and then two different branches of the decoder network are attached.

[0014] Furthermore, the column region identification decoding layer includes: a convolution layer conv7 column and a 1x1 convolution conv8column layer, after the 1x1 convolution conv8 column layer, a dropout layer with a ReLU activation function and the same dropout probability as the image shared coding layer, and upsampling the feature map using fractional stride convolution; the upsampled feature map is combined with the fourth pooling layer of the image shared coding layer, and then combined with the third pooling layer of the image shared coding layer of the same dimension, and then the feature map is enlarged to the original image size.

[0015] Furthermore, the table total area recognition decoding layer uses an additional 1x1 convolution layer conv7 table, the output of the onv7 table layer is upscaled using fractional stride convolution and combined with the fourth pooling layer of the image shared encoding layer of the same dimension, the combined feature map is upscaled again and then combined with the third pooling layer of the image shared encoding layer, and then the feature map is upscaled to the original image size.

[0016] Furthermore, the vertical coordinate of each row separation line is determined by the pixel level accumulation value as follows:

[0017] Let N represent the number of rows of image pixels in the table area. For 1≤i≤N, select all i that satisfy A(i)>minHor and store them in the array H[y]; the threshold minHor is determined by max(A(i))*p;

[0018] Further filter H[y] based on the threshold lineHor. If the difference between several coordinates is less than lineHor, then select the median as the final vertical coordinate of the row separator and store the final vertical coordinate in the array finlH[y];

[0019] Where i represents the number of image rows that are being subjected to pixel-level projection accumulation calculations. 1≤i≤N, M represents the number of columns of image pixels in the table area, U ij Represents the value of each pixel in the table area, p is a hyperparameter, and the threshold lineHor represents the maximum thickness of the row separator line;

[0020] The hyperparameters and thresholds are selected by comparing the output predicted table row separator line position with the actual table row separator line position in the original table text image, observing whether the predicted row separator line has a mismatch with the actual row separator line, and then adjusting the hyperparameters and thresholds.

[0021] Furthermore, during the model training process, the image sharing coding layer repeatedly trains the gradients received from the column region identification decoding layer and the table region identification decoding layer, while the column region identification decoding layer and the table region identification decoding layer are trained independently.

[0022] According to another aspect of the specification, the present invention provides a table area recognition and extraction system for a full-frame table, the system comprising: a data set generation module, a table column and total area recognition module, a table row area recognition module, a training module and an optical character recognition module;

[0023] The data set generation module is used to generate a table document image data set;

[0024] The table column and total area recognition module uses the TDEM end-to-end deep learning model to output masks of the table total area and column area;

[0025] The table row region recognition module converts the image in the table document image data set into a horizontal projection distribution histogram of image pixels, and determines the vertical coordinate of each row separator line through the pixel horizontal cumulative value;

[0026] The training module is used to train the TDEM end-to-end deep learning model and the algorithms in the table row region recognition module.

[0027] The optical character recognition module recognizes and extracts the text content in the table area in the document image and displays it in Excel format.

[0028] According to another aspect of the specification, the present invention provides a table area recognition and extraction device for a full-frame table, and when the processor executes the executable code, the table area recognition and extraction method for a full-frame table is implemented.

[0029] According to another aspect of the specification, the present invention provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method for identifying and extracting a table area for a full-frame table is implemented.

[0030] Beneficial effects:

[0031] The present invention saves non-structured documents of different formats in a unified manner as image formats, thereby avoiding the problem that the application of recognition algorithms is limited due to different formats.

[0032] For document images containing tables, a table area recognition and extraction model based on cell segmentation is designed to solve the problem of difficulty in identifying the table structure and text content in the image during table recognition.

[0033] The present invention utilizes the intrinsic relationship between the two interdependent tasks of table detection and table structure recognition, that is, the table and the column have a common area, and the accurate positioning of the table area can greatly improve the results of column detection. The model of the present invention utilizes a basic network that is initialized using pre-trained VGG-19 features. The model shares the encoding layer of VGG-19 for table and column detectors, while the decoders for the two tasks are separate. The shared common layer is repeatedly trained with the gradients received from the table and column detectors, while the decoders are trained independently. The semantic information about the basic data types is then used to further improve the model performance. Accurate detection and recognition of table areas and table column areas is achieved, solving the problem that traditional methods cannot accurately detect table areas and accurately identify table structures at the same time.

[0034] The recognition and extraction results of the present invention are presented to the operator in Excel format, which is intuitive and easy to modify. Compared with the traditional heuristic-based table detection and recognition method, which usually uses manually designed rules to screen and recognize the table area in the document, the present invention will greatly reduce the probability of errors and mistakes when obtaining the table structure and text content features, reduce the error propagation or accumulation, and greatly improve the effect of identifying and extracting the text content in the table area. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is an overall principle block diagram of a table area recognition and extraction method for a full-frame table provided according to an embodiment of the present invention.

[0036] Figure 2 4 is a flowchart of a method for identifying and extracting a table area for a full-frame table provided according to an embodiment of the present invention.

[0037] Figure 3 is a schematic diagram of an end-to-end deep learning neural network TDEM provided according to an embodiment of the present invention.

[0038] Figure 4 Schematic diagram of a training module of a system provided according to an embodiment of the present invention.

[0039] Figure 5 It is a schematic diagram of electronic device interaction during model training according to an embodiment of the present invention.

[0040] Figure 6 It is a schematic diagram of the execution of text content recognition and extraction in a table area provided by an embodiment of the present invention.

[0041] Figure 7 It is a schematic diagram of electronic device interaction when performing text content recognition and extraction in a table area according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be used as a limitation of the present invention.

[0043] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0044] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0045] The overall principle block diagram of the table area recognition and extraction method for the full-frame table provided by the present invention is as follows: Figure 1 As shown in the figure, the table area recognition and extraction method for the full-frame table includes two sub-models in the training stage:

[0046] First, the document images containing full-framed tables that have undergone image preprocessing are annotated using labelImg image annotation software to construct a full-framed table document image dataset;

[0047] Input the annotated full-frame table document image into the constructed VGG-19 pre-trained neural network for training to obtain the image shared encoding layer;

[0048] The image shared coding layer is input into the TDEM end-to-end deep learning model for training to obtain the table region detection and column region recognition models;

[0049] Binarize the full-frame table document image and generate the corresponding pixel horizontal projection distribution histogram through horizontal projection accumulation calculation;

[0050] Input the pixel horizontal projection distribution histogram into the table row separator line positioning algorithm model for training, and obtain the table row area recognition model;

[0051] The table area detection and column area recognition models identify and extract the table area and column area in the table document image, and the table row area recognition model identifies and extracts the table row area in the table document image;

[0052] By combining the two sub-models to detect the table document image, the extraction results at the cell level of the table area are obtained;

[0053] The extracted table cells are input into the Tesseract optical character recognition module to obtain accurate table text content recognition results, and the final table recognition and extraction results are stored in Excel format.

[0054] The specific steps of the text semantic feature extraction method for equipment testing provided by the present invention are as follows: Figure 1 As shown, the description is as follows:

[0055] 101) Steps for generating a table document image dataset: converting a document containing a full-frame table from PDF / HTML format into a jpg, png or bmp image format, and collecting and organizing the converted document images into a full-frame table document image dataset.

[0056] 102) Table area detection model and table column area recognition model training steps: Annotate the table area and table column area of ​​the document image in the full-frame table document image dataset, and utilize the inherent interdependence between the two tasks of table detection and table structure recognition to build an end-to-end deep learning model TDEM based on the VGG-19 neural network, and train the model on the annotated full-frame table document image dataset.

[0057] 103) Table row region recognition model training steps: Build a table row region recognition model based on horizontal projection, and train the model on the annotated full-frame table document image dataset.

[0058] 104) Steps for building a full-frame table area recognition and extraction system: input the trained table area detection model, table column area recognition model and table row area recognition model respectively, and combine them with the Tesseract optical character recognition module to obtain a full-frame table area recognition and extraction system based on cell segmentation.

[0059] 105) Full-frame table recognition and extraction input step: input the document image to be recognized and extracted into the constructed full-frame table area recognition and extraction system.

[0060] 106) Full-frame table recognition, extraction and output step: Based on the trained full-frame table area recognition and extraction system, the text content contained in the table area in the document image is obtained, and the processing results are stored in Excel format for display.

[0061] Specifically, the steps for generating a full-frame table document image dataset are as follows:

[0062] The steps of processing the table document image include: converting the PDF / HTML document into an image in jpg, png or bmp format by taking a screenshot or changing the file saving format, annotating the table area, table column area and cell text content area in the document image by labelImg image annotation software, converting the document containing the table into an annotated table document image, and collecting and arranging it to obtain a full-frame table document image data set.

[0063] Specifically, the end-to-end deep learning network model for accurate extraction of tabular data (TDEM) exploits the intrinsic relationship between the two interdependent tasks of table detection and table structure recognition, that is, tables and columns have common areas, and accurate positioning of table areas can greatly improve the results of column detection. The model uses a base network that is initialized with pre-trained VGG-19 features. The model shares the encoding layer of VGG-19 for table and column detectors, while the decoders for the two tasks are separate. The shared common layers are repeatedly trained with the gradients received from the table and column detectors, while the decoders are trained independently. The semantic information about the basic data types is then used to further improve the model performance.

[0064] Specifically, the different branch decoder mechanisms for the table area and column area of ​​the TDEM end-to-end deep learning model are explained as follows:

[0065] Our proposed model uses the pre-trained VGG-19 layers as the base network. The fully connected layers of VGG-19 (layers after pool5) are replaced with two (1x1) convolutional layers. Both convolutional layers (conv6) use the ReLU activation function, followed by a dropout layer with a dropout rate of 0.8 (e.g. Figure 1conv6+dropout as shown). After this layer, two different branches of the decoder network are attached. This is based on the intuition that the column region is a subset of the table region. Therefore, the single encoder network can utilize the features of both the table and column regions to filter out the active regions more accurately. The output of the (conv6+dropout) layer is assigned to the two decoder branches respectively. In each branch, additional layers are attached to filter out the corresponding active regions. In the table branch of the decoder network, an additional (1x1) convolutional layer conv7 table is used before upscaling the image using a series of fractional stride convolutional layers. The output of the conv7 table layer is also upscaled using fractional stride convolutions and combined with a pool4 pooling layer of the same dimension. Similarly, the combined feature map is upscaled again and then combined with a pool3 pooling layer. Finally, the final feature map is upscaled to meet the size of the original image. In the other branch for detecting columns, there is an additional convolutional layer conv7 column with a dropout layer with ReLU activation function and the same dropout probability. After the (1x1) convolutional (conv8column) layer, the feature map is upsampled using fractional stride convolutions. The upsampled feature maps are combined with a pool4 pooling layer and then a pool3 pooling layer of the same dimension. After this layer, the feature maps are upscaled to the original image. In both branches, multiple (1x1) convolutional layers are used before the transpose layer. The intuition behind using (1x1) convolutions is to reduce the dimensionality of the feature maps (channels) for class prediction of pixels, since the number of channels of the output layer (output of the encoder network) must be equal to the number of classes (assigning the channel with the maximum probability to the corresponding pixel) followed by upsampling. Therefore, the output of both branches of the computational graph produces masks for the table and column regions.

[0066] In detail, the tabular data accurate extraction neural network model (TDEM) is as follows Figure 3 As shown. The Tabular Data Extraction Precision Neural Network Model (TDEM) receives a single input image and generates two different semantically labeled output images for tables and columns respectively. The model shares the encoding layers of VGG-19 for table and column detectors, while the decoders for the two tasks are separate. The shared common layers are repeatedly trained with the gradients received from the table and column detectors, while the decoders are trained independently. Semantic information about the underlying data types is then exploited to further improve the model performance. Using VGG-19 pre-trained on the ImageNet dataset as the base network allows the exploitation of prior knowledge of low-level features gained by training on ImageNet.

[0067] In detail, the table row dividing line positioning algorithm is as follows.

[0068]

[0069] After the table area detected by TDEM is binarized, the horizontal projection and accumulation calculation of the image pixel values ​​are performed to obtain the horizontal distribution histogram of the binarized pixel values ​​of the table area. The table row separator line is composed of continuous black pixel values, and its horizontal pixel accumulation value is much larger than other areas of the table. Then, the vertical coordinate of each row separator line is determined by the table row separator line positioning algorithm.

[0070] Specifically, the table row separator positioning algorithm is explained as follows:

[0071] 1) Let N represent the number of rows of pixels in the table area image. For 1≤i≤N, select all i that satisfy A(i)>minHor and store them in the array H[y]. The threshold minHor can be determined by max(A(i))*p.

[0072] 2) Further filter H[y] based on the threshold lineHor. If the difference between several coordinates is less than lineHor, then select the median as the final vertical coordinate of the row separator and store the final vertical coordinate in the array finlH[y].

[0073] i represents the number of image rows that are being subjected to pixel-level projection accumulation calculations. 1≤i≤N, M represents the number of columns of image pixels in the table area, U ij Represents the value of each pixel in the table area, p is a hyperparameter which we set to 0.7 in the algorithm, and the threshold lineHor represents the maximum thickness of the row separator line which we set to 4 in the algorithm.

[0074] The specific method of selecting the values ​​of the hyperparameters and thresholds is as follows: by comparing the output predicted table row separator line position with the position of the actual table row separator line in the original table text image, observe whether the predicted row separator line has misjudgment (a very thick row separator line is identified as multiple row separator lines) and missed prediction (two close row separator lines are identified as one row separator line) and other problems that do not match the actual row separator line, and then adjust the hyperparameter p and threshold lineHor in the table row separator line positioning algorithm. Through continuous testing and training of the input original table text image, the appropriate hyperparameter p and threshold lineHor are determined respectively, so that the table row separator line positioning algorithm model has good prediction results when dealing with all images in the full-frame table document image dataset. The hyperparameter p = 0.7 and threshold lineHor = 4 in the above text are the parameter values ​​determined after training.

[0075] According to the above method embodiment, the present invention also provides a table area recognition and extraction system for a full-frame table, the system comprising: a data set generation module, a table column and total area recognition module, a table row area recognition module, a training module and an optical character recognition module;

[0076] The data set generation module is used to generate a table document image data set;

[0077] The table column and total area recognition module uses the TDEM end-to-end deep learning model to output masks of the table total area and column area;

[0078] The table row region recognition module converts the image in the table document image data set into a horizontal projection distribution histogram of image pixels, and determines the vertical coordinate of each row separator line through the pixel horizontal cumulative value;

[0079] The training module is used to train the TDEM end-to-end deep learning model and the algorithms in the table row region recognition module.

[0080] The optical character recognition module recognizes and extracts the text content in the table area in the document image and displays it in Excel format.

[0081] Figure 4 A schematic diagram of the training module is given, including a table document image preprocessing module, a table data accurate extraction deep learning model TDEM training module, and a table row area recognition model training module.

[0082] The function of the table document image preprocessing module is to convert PDF / HTML documents into images in jpg, png or bmp formats by taking screenshots or changing the file saving format, and to annotate the table area, table column area and cell text content area in the document image through labelImg image annotation software, and then convert the document containing the table into annotated table document images, and collect and sort them to obtain a full-frame table document image dataset.

[0083] The function of the TDEM training module of the deep learning model for precise extraction of tabular data is to initialize the pre-trained VGG-19 neural network features using the input full-frame table document image dataset. The encoder of the model is common in the table area detection task and the table column area recognition task. It is followed by two decoder branches for segmenting the table area and the columns within the table area. The TDEM deep learning model for precise extraction of tabular data is constructed and the model is trained on the full-frame table document image dataset.

[0084] The function of the table row area recognition model training module is to accumulate the horizontal projections of the pixel values ​​of the input full-frame table document image after binarization to obtain the table image pixel horizontal projection distribution histogram, and input the table image pixel horizontal projection distribution histogram into the hyperparameters and thresholds in the table row segmentation line positioning algorithm for training.

[0085] Figure 5 A schematic diagram of electronic device interaction during model training in a table area recognition and extraction device for a full-frame table is provided, including: a full-frame table document image data set storage; a model training algorithm processor; a model output interface; and a model training algorithm storage for storing a model training program. After the device is powered on and the model training program is run through the processor, the following steps are performed:

[0086] The unlabeled table document images stored in the full-frame table document image dataset storage are labeled with the table area, table column area and cell text content area in the document image using labelImg image labeling software to construct a labeled full-frame table document image dataset.

[0087] The annotated full-frame table document image dataset is input into the constructed table area detection and table column area recognition model based on the pre-trained VGG-19 neural network for training, and an end-to-end deep learning model TDEM is obtained.

[0088] The annotated full-frame table document image stored in the full-frame table document image data set memory is input into the table row area recognition model to obtain the vertical coordinate of the table row dividing line.

[0089] The vertical coordinate of the word table row segmentation line is input into the end-to-end deep learning model TDEM to obtain the full-frame table area recognition and extraction model based on cell segmentation.

[0090] Output all models and their model parameters to the model output interface.

[0091] Figure 6 A schematic diagram of the execution of the table area text content recognition and extraction is given, and the functions are: inputting a trained model, inputting a document image of the table text content to be extracted, and inputting all model parameters; based on the trained full-frame table area recognition and extraction system, the recognition and extraction results of the table area text content contained in the input document image are obtained.

[0092] Figure 7A schematic diagram of electronic device interaction when performing table area text content recognition and extraction in a table area recognition and extraction device for a full-frame table is provided, comprising: a full-frame table document image data set storage; a model input interface; a table area text content recognition and extraction algorithm processor; a data storage; a display; and a table area text content recognition and extraction algorithm storage for storing a table area text content recognition and extraction algorithm program. After the device is powered on and the table area text content recognition and extraction algorithm program is run through the processor, the following steps are performed:

[0093] Extracting a document image of text content in a table area to be identified from a full-frame table document image data set storage;

[0094] Acquire trained models from the model input interface, and send these models together with the document image of the table area text content to be identified and all model parameters to the table area text content identification and extraction algorithm processor;

[0095] The table area text content recognition and extraction algorithm processor recognizes and extracts the text content of the table area in the document image based on the trained table data accurate extraction deep learning model TDEM, table row area recognition model and Tesseract optical character recognition module;

[0096] The table area text content recognition and extraction algorithm processor outputs the recognition and extraction results of the table area text content in the document image to the data storage.

[0097] The data storage device outputs the text recognition and extraction results of the table area in the document image in Excel format to the display.

[0098] The neural network model needs to be trained on a dataset of full-frame table document images. For the production and preparation of the dataset, a simple and convenient data organization specification is designed.

[0099] Compared with the prior art, the present invention can overcome the difficulty in detecting and identifying tables in non-structured documents, and achieve a table structure recognition effect with an accuracy rate of about 96%; it can further identify the text content contained in the table area, with a recognition accuracy rate of about 89%; it can effectively solve the problem that the final operating results of the existing table detection and recognition algorithms cannot be stored in a structured document format, and our final recognition and extraction results are presented to the operator in Excel format, which is intuitive and easy to modify. Compared with the traditional heuristic-based table detection and recognition method that usually uses manually designed rules to screen and identify table areas in documents, the present invention will greatly reduce the probability of errors and mistakes when obtaining table structure and text content features, reduce the error propagation or accumulation, and greatly improve the effect of identifying and extracting text content in table areas.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A table area recognition and extraction method for a full-frame table, characterized in that: The steps include: S1, generate a table document image dataset; S2. Construct an end-to-end deep learning model of TDEM, including an image sharing encoding layer, a column region recognition decoding layer, and a table total region recognition decoding layer composed of a VGG-19 pre-trained neural network. After the image passes through the image sharing encoding layer, it is input into two different branches of the column region recognition decoding layer and the table region recognition decoding layer to obtain masks of the table total region and column region. S3, converting the images in the table document image data set into a horizontal projection distribution histogram of image pixels, and determining the vertical coordinate of each row separator line by the horizontal cumulative value of the pixels; S4. Perform model training, input the table image into the trained model, use Tesseract optical character recognition to identify and extract the text content in the table area of ​​the document image, and display it in Excel format.

2. The method for identifying and extracting a table area for a full-frame table according to claim 1, characterized in that: The generating of the table document image data set comprises: Convert documents containing full-frame tables from PDF / HTML format to jpg, png or bmp image formats, and collect and organize the converted document images into a full-frame table document image dataset.

3. The method for identifying and extracting a table area for a full-frame table according to claim 1, characterized in that: In the TDEM end-to-end deep learning model, the image shared coding layer is specifically: The fully connected layers of VGG-19 are replaced with two convolutional layers, both of which use the ReLU activation function, followed by a dropout layer with a dropout rate of 0.8, and then two different branches of the decoder network are attached.

4. The method for identifying and extracting a table area for a full-frame table according to claim 3, characterized in that: The column region identification decoding layer includes: a convolution layer conv7 column and a 1x1 convolution conv8 column layer, after the 1x1 convolution conv8 column layer, a dropout layer with a ReLU activation function and the same dropout probability as the image shared coding layer, and a feature map is upsampled using a fractional stride convolution; the upsampled feature map is combined with the fourth pooling layer of the image shared coding layer, and then combined with the third pooling layer of the image shared coding layer of the same dimension, and then the feature map is enlarged to the original image size.

5. The method for identifying and extracting a table area for a full-frame table according to claim 3, characterized in that: The table total area recognition decoding layer uses an additional 1x1 convolution layer conv7 table. The output of the onv7 table layer is upscaled using fractional stride convolution and combined with the fourth pooling layer of the image shared encoding layer of the same dimension. The combined feature map is upscaled again and then combined with the third pooling layer of the image shared encoding layer. The feature map is then upscaled to the original image size.

6. The method for identifying and extracting a table area for a full-frame table according to claim 1, characterized in that: The vertical coordinate of each row separation line is determined by the pixel horizontal accumulation value as follows: Let N represent the number of rows of image pixels in the table area. For 1≤i≤N, select all i that satisfy A(i)>minHor and store them in the array H[y]; the threshold minHor is determined by max(A(i))*p; Further filter H[y] based on the threshold lineHor. If the difference between several coordinates is less than lineHor, then select the median as the final vertical coordinate of the row separator and store the final vertical coordinate in the array finlH[y]; Where i represents the number of image rows that are being subjected to pixel-level projection accumulation calculations. M represents the number of columns of image pixels in the table area, U ij Represents the value of each pixel in the table area, p is a hyperparameter, and the threshold lineHor represents the maximum thickness of the row separator line; The hyperparameters and thresholds are selected by comparing the output predicted table row separator line position with the actual table row separator line position in the original table text image, observing whether the predicted row separator line has a mismatch with the actual row separator line, and then adjusting the hyperparameters and thresholds.

7. The method for identifying and extracting a table area for a full-frame table according to claim 1, characterized in that: During the model training process, the image sharing coding layer is repeatedly trained with the gradients received from the column region identification decoding layer and the table region identification decoding layer, while the column region identification decoding layer and the table region identification decoding layer are trained independently.

8. A table area recognition and extraction system for a full-frame table for implementing the method described in any one of claims 1 to 7, characterized in that: The system includes: a data set generation module, a table column and total area recognition module, a table row area recognition module, a training module and an optical character recognition module; The data set generation module is used to generate a table document image data set; The table column and total area recognition module uses the TDEM end-to-end deep learning model to output masks of the table total area and column area; The table row region recognition module converts the image in the table document image data set into a horizontal projection distribution histogram of image pixels, and determines the vertical coordinate of each row separator line through the pixel horizontal cumulative value; The training module is used to train the TDEM end-to-end deep learning model and the algorithms in the table row region recognition module. The optical character recognition module recognizes and extracts the text content in the table area in the document image and displays it in Excel format.

9. A table area recognition and extraction device for a full-frame table, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a table area recognition and extraction method for a full-frame table according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a table area recognition and extraction method for a full-frame table according to any one of claims 1 to 7 is implemented.

Images