Method, computer system, and computer program product for improved table processing

Abstract

In a first aspect, the invention relates to a computer-implemented method for improved gridless table processing. In a second and third aspect, the invention relates to a computer system and computer program product for improved gridless table processing. In a fourth aspect, the invention relates to the use of any of the methods, systems, or products for document parsing.

Description

[Technical Field]

[0001] The present invention relates to methods, systems and computer program products for improved table processing and their use for document parsing. [Background technology]

[0002] Methods, systems, and computer program products for processing tables are known in the art.

[0003] Such table processing is particularly useful for document parsing of fixed-layout documents such as PDF documents. Parsing tables from image or non-image PDF documents can be a challenging task. Various open-source and licensed PDF parsing tools available on the market struggle to handle tables correctly.

[0004] In particular, tables without a grid separating columns and rows, i.e., gridless tables, pose a particular challenge to parsing tools. Documents containing such gridless tables are often referred to in the literature as "semi-structured documents." These documents pose a challenge to parsing tools because existing tools have limited automated detection techniques. As a result, analysts must manually select and extract tables when parsing documents. Furthermore, existing parsing tools cannot extract data or information contained within multi-line or merged cells in tables.

[0005] U.S. Patent Application Publication No. 2020-0-089-946 (Patent Document 1) describes a tool for extracting tabular data from electronic documents. The tool generates grid structure information for the tabular data. This grid structure is combined with text associated with the tabular data to obtain a gridded table. The inventors note that the tool provided in Patent Application Publication No. 2020-0-089-946 is inaccurate, particularly when generating grids for tables that contain only horizontal or vertical lines. In addition, the tool does not work when generating grids for gridless tables.

[0006] There remains a need in the art for improved handling of tables, particularly gridless tables, and for improved parsing of documents containing these tables.

[0007] The present invention aims to solve at least some of the technical problems associated with methods, systems and computer program products known in the art. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] U.S. Patent Application No. 2020 0 089 946 Summary of the Invention

[0009] In a first aspect, the invention relates to a method for gridless table processing as set forth in claim 1.

[0010] In a second aspect, the invention relates to a computer system for gridless table processing as claimed in claim 12.

[0011] In a third aspect, the invention relates to a computer program product for gridless table processing as claimed in claim 13.

[0012] In a fourth aspect, the invention relates to the use of any of a method, system or computer program product as claimed in claim 14 for document parsing.

[0013] The present invention is advantageous in that it allows for simple and accurate processing of gridless tables. By extension, it makes semi-structured documents containing such gridless tables easier to parse. Further advantages of the present invention are discussed below through the description, examples, and figures.

[0014] Preferred embodiments of the present invention are discussed in claims 2 to 12 and 15 and throughout the description, examples and figures. [Brief explanation of the drawings]

[0015] [Figure 1] 1 shows a circuit schematic diagram of the overall workflow of the present invention comprising processing of gridless tables and associated documents. [Figure 2] Illustrates localization prediction of gridless tables in documents using a deep learning model, specifically a convolutional neural network. [Figure 3] Illustrates localization prediction of gridless tables in documents using a deep learning model, specifically a convolutional neural network. [Figure 4] Illustrates localization prediction of gridless tables in documents using a deep learning model, specifically a convolutional neural network. [Figure 5] Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 6] Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 7] Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 8]Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 9] Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 10] Illustrates table grid prediction for extracted gridless tables using generative adversarial networks, specifically conditional GANs. [Figure 11] 1 illustrates preferred pre- and post-dilatation image processing steps for improved processing of gridless tables. [Figure 12] 1 illustrates preferred pre- and post-dilatation image processing steps for improved processing of gridless tables. [Figure 13] 1 illustrates preferred pre- and post-dilatation image processing steps for improved processing of gridless tables. [Figure 14] 1 illustrates preferred pre- and post-dilatation image processing steps for improved processing of gridless tables. [Figure 15] 1 illustrates preferred pre- and post-dilatation image processing steps for improved processing of gridless tables. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention relates to a computer-implemented method, computer system and computer program product for improved gridless table processing, and the use of any of the methods, systems or products for document parsing. In the following, the invention is described in detail, preferred embodiments are discussed, and the invention is illustrated by non-limiting examples.

[0017] Unless otherwise defined, all terms used in disclosing the present invention, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art to which this invention belongs. As a further guide, term definitions are included to better understand the teachings of the present invention. As used herein, the following terms have the following meanings:

[0018] As used herein, "a," "an," and "the" refer to both singular and plural references unless the context clearly dictates otherwise. By way of example, "a compartment" refers to one or more compartments.

[0019] As used herein, "comprise," "comprising," and "comprising of" are synonymous with "include," "including," "includes," or "contain," "containing," or "contains," and are inclusive or open-ended terms that specify the presence of what follows, e.g., composition, but do not exclude or preclude the presence of additional, non-recursive features, elements, or steps known in the art or disclosed therein.

[0020] While the terms "one or more" or "at least one," such as one or more components or at least one component of a group of components, are themselves clear, by way of further illustration, the terms specifically encompass reference to any one of the components, or any two or more of the components, such as, for example, any ≧3, ≧4, ≧5, ≧6, ≧7, etc., of the members, or even all of the components.

[0021] Unless otherwise defined, all terms used in disclosing the present invention, including technical and scientific terms, have the meanings commonly understood by those of ordinary skill in the art to which this invention belongs. As a further guide, definitions of terms used herein are included to better understand the teachings of the present invention. The terms or definitions used herein are provided solely to aid in the understanding of the present invention.

[0022] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, although they may. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, as would be apparent to one of ordinary skill in the art from this disclosure. Furthermore, while some embodiments described herein include some features but not other features included in other embodiments, combinations of features from different embodiments are meant to form different embodiments within the scope of the present invention and as would be understood by one of ordinary skill in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0023] Moreover, terms such as "first," "second," and "third" in the description and claims, unless specified otherwise, are used to distinguish between similar elements and do not necessarily describe a sequential or chronological order. The terms so used are interchangeable under appropriate circumstances, and it is understood that the embodiments of the invention described herein are capable of operating in sequences other than those described or illustrated herein.

[0024] In a first aspect, the present invention relates to a computer-implemented method for improved table processing. The method preferably comprises providing a fixed-layout electronic document including a gridless table. The method preferably comprises training a deep learning model (DLM) with training data associated with a plurality of training-related electronic documents. Preferably, each of the plurality of training-related electronic documents includes a gridless training table and an associated label indicating a bounding box of the training table. The method preferably comprises determining, by the trained DLM, a bounding box of the gridless table in the fixed-layout electronic document. The method preferably comprises extracting an image of the gridless table from the fixed-layout electronic document based on the determined bounding box. The method preferably comprises processing the extracted image by performing at least a dilation step. The method preferably comprises training a conditional generative adversarial network (cGAN) including a generator neural network (GNN) and a discriminator neural network (DNN). Preferably, the cGAN is trained with training data including a set of real pairs. Preferably, wherein each of the sets of actual pairs includes a training-related dilated gridless table image and a corresponding table-grid image. Preferably, wherein training the cGAN comprises a plurality of training steps. Preferably, wherein each of the training steps comprises one of the following steps a) or b), and step c): a) Providing the DNN with real pairs obtained from a set of real pairs. b) providing the DNN with generated pairs including candidate false grid images and corresponding training-related inflated gridless table images obtained from the set of real pairs, wherein the candidate false grid images are obtained by the GNN by modifying the corresponding training-related inflated gridless table images using random data vectors; c) determining by the DNN an indication of whether the actual pair or generated pair provided to the DNN is a pair generated by the GNN; Preferably, the training of the cGAN comprises multiple iterative training cycles of the GNN and the DNN. Preferably, each of the multiple iterative training cycles of the GNN and the DNN comprises at least one of the multiple training steps. Preferably, during each training cycle, a loss function associated with each GNN or DNN is minimized until an associated stable cycle predicate is satisfied. Preferably, the cGAN is trained by minimizing a joint loss of the loss functions associated with the GNN and the DNN until a joint loss stable predicate is satisfied. Preferably, the method further comprises a step of determining a table grid image for the extracted image processed by the trained GNN, which preferably comprises the trained cGAN. Preferably, the method further comprises a step of combining the determined table grid image and a gridless table from the provided fixed-layout electronic document into a gridded image of the gridless table.

[0025] In a second aspect, the present invention relates to a computer system for improved table processing, preferably wherein the computer system is configured to perform a computer-implemented method according to the first aspect of the invention.

[0026] In a third aspect, the present invention relates to a computer program product for improved table processing, preferably wherein the computer program product comprises instructions that, when executed by a computer, cause the computer to perform the computer-implemented method according to the first aspect of the invention, preferably wherein the computer is a computer system according to the second aspect of the invention.

[0027] In a fourth aspect, the present invention relates to the use of a computer-implemented method for improved table processing according to the first aspect of the invention, a computer system according to the second aspect of the invention, or a computer program product according to the third aspect of the invention, for generating a parsed document from a fixed-layout electronic document.

[0028] The present invention provides a computer-implemented method, computer system, and computer program product for improved table processing, and the use of any of the methods, systems, or products for generating parsed documents from fixed-layout electronic documents. Those of ordinary skill in the art will understand that the method may be implemented in a computer program product and executed using a computer system. It will also be apparent to those of ordinary skill in the art that the improved table processing can be used to parse tables from documents. Therefore, in the following, the four aspects of the present invention will be treated together.

[0029] The present invention aims to improve table processing, particularly the processing of gridless tables. While the provided solution can improve the processing of any table, including or lacking any grid structure known in the art, the present invention is particularly directed to improved processing of gridless tables. Such tables are particularly difficult to detect in electronic fixed-layout documents. Furthermore, information contained therein, such as hierarchy, is difficult to extract. Therefore, the present invention aims to provide, more particularly, a method, system, and computer program product for conveniently, efficiently, and reliably extracting tabular data from electronic documents while preserving their format and structure (or hierarchy). The improved processing of gridless tables according to the present invention can be advantageously used in different fields, such as digital character extraction (DCE), image enhancement, optical character recognition (OCR), document layout analysis, field detection, field parsing, etc.

[0030] An object of the present invention is to generate a parsed document from a fixed-layout electronic document. In this regard, from a provided fixed-layout electronic document, a determined table grid image and a gridless table are combined into a gridded image of the gridless table. This gridded image of the gridless table may or may not be provided as a separate image. This gridded image of the gridless table may or may not be superimposed on the fixed-layout electronic document or its image. Preferably, generating a parsed document from the fixed-layout electronic document includes a step of optical character recognition (OCR) of text associated with table data from the gridded image of the gridless table.

[0031] "Parsing," as used herein, refers to a term known in the art and should preferably be understood as analyzing and / or separating a data stream, such as a fixed-layout electronic document, into more easily processable components, with the purpose of facilitating certain transformations, such as OCR, performed on the data stream.

[0032] "Tabular data" as used herein refers to a term known in the art and should preferably be understood as any representation of data in the form of an appropriate and well-defined format and structure (or hierarchy) to enhance the visual representation, interpretation and extraction of data. Tabular data helps to maintain the hierarchical structure of data and also allows for classification of the data represented therein. Tabular data can include structured data arrangements such as grid structures, tables, etc. In the case of gridless tables, the structured data arrangement is not visible. Optionally, tabular data can comprise a flow chart.

[0033] As used herein, "electronic document" refers to a term known in the art and should preferably be understood as any electronic medium capable of containing one or more types of content therein. Such electronic documents can be in different formats, such as portable document format files (having a .PDF extension), documents generated using word processing software (e.g., files having a .doc or .docx extension), web text documents (.HTML or .htm extension), Adobe PostScript documents (.ps extension), etc. Furthermore, electronic documents may or may not contain different types of content, for example, unstructured data arrays such as text data, images, etc., and structured data arrays such as tabular data, histograms, etc. Additionally, electronic documents may or may not contain a single tabular data array (e.g., a single table) or multiple tabular data arrays.

[0034] As used herein, the term "fixed-layout electronic document" refers to a term known in the art and is preferably understood to mean any electronic medium capable of containing one or more types of content, the one or more types of content having a fixed layout. Such fixed-layout electronic documents can be in different formats, such as searchable or non-searchable portable document format files (having a .PDF extension), bitmap files (having a .bmp extension), portable network graphics files (having a .png extension), Joint Photographic Experts Group files (having a .jpeg extension), tagged image file formats (having a .tiff extension), etc. Preferably, the fixed-layout electronic document is a PDF document, abbreviated as PDF. Such PDFs are widely used and facilitate communication (e.g., sending and receiving) of tabular data between different individuals, and are convenient for interpreting and extracting information from the tabular data. However, when it is necessary to extract tabular data from a PDF, such as copying tabular data from a PDF, the tabular data loses its format and structure (or hierarchy), and the extracted data becomes unusable.

[0035] To extract images of gridless tables from a provided fixed-layout electronic document, a simple embodiment of the present invention provides training a deep learning model (DLM) with training data relating to a plurality of training-related electronic documents, preferably each of which includes a gridless training table and an associated label indicating a bounding box of the training table.

[0036] Advantageously, the present invention enables automatic detection of tabular data from electronic documents, thereby mitigating problems associated with manual user selection of tabular data. In this manner, the time and effort required for manual data selection can be reduced, thereby improving user efficiency. Furthermore, the present invention enables gridless tables to be extracted from fixed-layout electronic documents such that the hierarchy of the tabular data is maintained after extraction. It will be appreciated that maintaining the hierarchy of the tabular data after extraction allows for maintaining the integrity of the information represented in the tabular data, thereby enabling convenient, reliable, and efficient extraction and interpretation of the tabular data.

[0037] "Deep learning," as used herein, refers to a term known in the art and should preferably be understood as a machine learning technique comprising a network of nodes (such as artificial neurons) (such as an artificial neural network (ANN), a recurrent neural network (RNN), a convolutional neural network (CNN), etc.) capable of semi-supervised or supervised learning from sample electronic documents containing various forms of tabular data therein. In such an example, the DLM can be trained to detect the location of tabular data within electronic documents, such as by providing a plurality of training related sample electronic documents containing gridless training tables and associated labels indicating the bounding boxes of the training tables.

[0038] As used herein, the term "bounding box" refers to a term known in the art and is preferably understood to refer to a rectangular shape generated over an area of ​​an electronic document containing a gridless table, the rectangular shape completely encompassing the gridless table. It should be understood that the area of ​​a single bounding box generated corresponds to the entire area of ​​a page of the electronic document covered by a single tabular data. For example, the area of ​​a single bounding box may correspond to the area of ​​a page of the electronic document if the given tabular data is represented to completely cover the page of the electronic document. However, most tabular data typically covers a smaller area within the page of the electronic document, such as within 50% of the page of the electronic document. Optionally, if a single tabular data exists on multiple pages of the electronic document, the area of ​​a single bounding box may span multiple pages. In such an example, the generated bounding box corresponds to an area larger than the entire area of ​​each page of the electronic document.

[0039] Preferably, the bounding box comprises coordinates corresponding to the vertices of the bounding box and coordinates corresponding to the width and height of the bounding box. Thus, the bounding box may comprise the coordinates: tx; ty; tw; th. Alternatively, the bounding box comprises coordinates corresponding to each vertex of the bounding box. Thus, the bounding box may comprise the coordinates: (tx1, ty1); (tx2, ty2); (tx3, ty3); (tx4, ty4).

[0040] To extract an image of a gridless table from a provided fixed-layout electronic document, a simple embodiment of the present invention further provides extracting the image of the gridless table from the fixed-layout electronic document based on the determined bounding box. Preferably, the extracted image corresponds to an image of an area within the bounding box of the fixed-layout electronic document. The extracted image may or may not be associated with one or more image file formats, including, but not limited to, .jpg, .png, .bmp, .gif, etc.

[0041] Preferably, a plurality of training-related electronic documents, each including a gridless training table and an associated label indicating a bounding box of the training table, are obtained by: providing a plurality of DLM training-related fixed-layout electronic documents, each of the DLM training-related fixed-layout electronic documents including a gridless table, the plurality of DLM training-related fixed-layout electronic documents being convertible to a markup language; · converting a plurality of training-related fixed-layout electronic documents into corresponding documents in a markup language; identifying, in each corresponding markup language document, table tags that are associated with gridless tables in the associated training-related fixed-layout electronic document; and Associating with each of a plurality of training-related fixed-layout electronic documents a label for a bounding box of the gridless table, the label being based at least in part on the corresponding identified table tag.

[0042] According to a preferred embodiment, a plurality of training-related electronic documents, each including a gridless training table and an associated label indicating the bounding box of the training table, are obtained by: providing a plurality of PDF documents each containing a gridless table; Convert multiple PDF documents to HTML documents; identifying, in each of the HTML documents, table tags that are associated with gridless tables in the associated PDF document; and Associating with each of the plurality of PDF documents a label for a bounding box of the gridless table, the label being based at least in part on the corresponding identified table tag.

[0043] As used herein, "markup language" refers to a term known in the art and should preferably be understood as a file format suitable for annotating a document in a manner syntactically distinct from the corresponding text, and is used to format the corresponding text without the markup language being displayed when the document is processed for display. The use of the markup language may or may not need to be limited to formatting the corresponding text. The markup language may or may not have additional functionality. Examples of markup languages ​​include the following file formats: troff, nroff, TeX, Scribe, GML, SGML, HTML, XML, XHTML, and other XML-based applications. The use of a markup language to obtain table tags is easy to implement and provides qualitative information about the location of (gridless) tables in an electronic document. From this location information, the bounding box of the (gridless) table is easily derived. Preferably, HTML is used as the markup language. HTML is widely used.

[0044] Preferably, the DLM for determining the bounding box of a gridless table in a fixed-layout electronic document is one or more of an artificial neural network (ANN), a recurrent neural network (RNN), or a convolutional neural network (CNN). According to a preferred embodiment, the DLM is a CNN. Different DLM architectures can be trained to determine the bounding box of a gridless table in a fixed-layout electronic document. However, most techniques are not robust to extreme viewpoint and background variations. CNNs are known to be extremely robust to background and viewpoint variations in object detection and classification tasks, making them well suited for determining the bounding box of a gridless table in a fixed-layout electronic document with a variable layout, such as a scientific publication.

[0045] Preferably, the loss function associated with training the DLM to determine bounding boxes for gridless tables in fixed-layout electronic documents is based at least in part on the Intersection Over Union (IOU) loss function. IOU can be calculated as the area of ​​the intersection divided by the area of ​​the union of the two boxes. IOU must be greater than or equal to 0 and less than or equal to 1. To predict a bounding box, the IOU must be approximately 1 between the predicted bounding box and the ground truth bounding box. More preferably, the loss function associated with training the DLM to determine bounding boxes for gridless tables in fixed-layout electronic documents is based at least in part on the Binary Cross Entropy (BCE) loss function.

[0046] The CNN extracted image is processed according to a simple embodiment of the present invention. Preferably, the CNN extracted image processing comprises at least a dilation step. Dilation of the extracted image aligns objects present in the gridless table, e.g., gridless text information, with a required and / or uniform pixel density.

[0047] It will be appreciated that by performing such morphological expansion of objects in a gridless table so that the objects have a uniform pixel density, the complexity associated with processing objects having varying pixel densities is reduced.

[0048] As used herein, "morphological processing" refers to a term known in the art and is preferably to be understood as an image processing technique in which pixels of an image are modified based on the shape of an object represented in the image. Furthermore, morphological processing can employ a structuring element, such as a sliding window, that is used to detect the shape of the object, and then modify the pixels associated with the detected object. As used herein, "structuring element" refers to a term known in the art and is preferably to be understood as a template having a predetermined shape, such as a rectangle, that is used to perform morphological processing.

[0049] "Dilation" or "morphological dilation" as used herein refers to terms known in the art and should preferably be understood as an operation in which pixels are added to the boundary of an object detected in an image.

[0050] According to a preferred embodiment, processing with a DLM extracted image further comprises the following steps before the dilation step: Converting the extracted image into a grayscale image; Applying binarization to the grayscale image by adaptive Gaussian method. Here, the dilation step is performed on the binarized grayscale image.

[0051] According to a preferred embodiment, processing the extracted image further comprises, after the dilation step, the following steps: obtaining an outline of the dilated object in the dilated image obtained by the dilation step; Inflating the outline containing the image; and A step of applying binarization to the dilated outline containing the image by means of an adaptive Gaussian method.

[0052] According to a particularly preferred embodiment, processing the extracted image comprises the following steps: Converting the extracted image into a grayscale image; · Applying binarization to the grayscale image by adaptive Gaussian method; Dilation of the binarized grayscale image; A step of obtaining the outline of the dilated object in the dilated thresholded grayscale image; Inflating the outline containing the image; and A step of applying binarization to the dilated outline containing the image by means of an adaptive Gaussian method.

[0053] As used herein, the term "grayscale image" refers to a term known in the art and should preferably be understood as a monochromatic image containing pixels having different shades of gray. It will be understood that different shades of gray are formed by combinations of varying proportions of black and white, such as, for example, a dark shade of gray with a maximum proportion of black and a minimum proportion of white, and a light shade of gray with a minimum proportion of black and a maximum proportion of white. Such conversion of the extracted image to a grayscale image minimizes the complexity that may arise during processing of a color image, such as by employing computer vision-based image processing techniques, e.g., morphological operations such as dilation and erosion. Grayscale images also associate less pixel information with the image compared to RGB (red, green, blue) images, and conversion of the received image to a grayscale image further aids in processing, such as faster processing.

[0054] "Binarization," as used herein, refers to a term known in the art and should preferably be understood as classifying pixels of an image, for example, a grayscale image, into two groups to obtain a binary image having only the two colors, where the pixels are classified based on a threshold intensity value. Pixels having intensities less than the threshold intensity value (such as pixels associated with lighter shades of gray) are classified into the first group, and pixels having intensities greater than the threshold intensity value (such as pixels associated with darker shades of gray) are classified into the second group. As a result, a grayscale image having pixels of various shades of gray is converted into a binarized grayscale image having only black and white colors, where the black pixels correspond to pixels of the grayscale image for all intensity values ​​less than the threshold intensity value, and the white pixels correspond to pixels of the grayscale image for all intensity values ​​greater than the threshold intensity value. Furthermore, by binarizing the image, for example, the grayscale image, the binarized grayscale image is separated such that the background of the binarized grayscale image, which is associated with white space, is separated from the foreground of the binarized grayscale image, which is associated with the text and grid structure of the tabular data.

[0055] As used herein, the term "adaptive Gaussian method" refers to a term known in the art and should preferably be understood as a binarization method that employs intensity values ​​of pixels adjacent to a given pixel (such as neighboring pixels) to classify the given pixel into a first group or a second group. Furthermore, the adaptive Gaussian method considers a weighted sum of intensity values ​​of pixels adjacent to the given pixel and the distance of the neighboring pixels from the given pixel to classify the given pixel into a first group or a second group. In addition, the adaptive Gaussian technique divides an image, e.g., a grayscale image, into multiple sub-images based on variations in the image (e.g., based on variations in the background of the grayscale image), and then considers a dynamic threshold intensity value for each sub-image based on a weighted average of intensity values ​​of pixels adjacent to a given pixel in the sub-image and a constant value. It will be understood that performing binarization of an image, e.g., a grayscale image, by considering a dynamic threshold intensity value, i.e., by employing the adaptive Gaussian technique, can improve the clarity and accuracy associated with the binarized image.

[0056] According to a preferred embodiment, processing the extracted image with DLM further comprises an erosion step at least after the dilation step, and most preferably, processing the extracted image with DLM further comprises an erosion step after each dilation step performed.

[0057] "Erosion" or "morphological erosion," as used herein, refers to terms known in the art and should preferably be understood as the operation of removing pixels from the boundary of an object detected in an image, such as by removing the outermost layer of pixels associated with the object in the image. Subsequent morphological dilation and erosion of objects in the gridless table image further reduces the complexity associated with processing the objects.

[0058] To determine the table grid image from the DLM processed extracted image, a simple embodiment of the present invention further provides training a conditional generative adversarial network (cGAN) comprising a generator neural network (GNN) and a discriminator neural network (DNN).

[0059] Generative adversarial networks (GANs) are a branch of unsupervised machine learning implemented by a system in which two neural networks, a GNN and a DNN, compete against each other in a zero-sum game framework. The two neural networks are created and learn (i.e., train) through their competition. The first neural network is implemented as a generator system and is called a generator neural network (GNN). The second neural network is implemented as a discriminator system and is called a discriminator neural network (DNN). The GNN starts with random inputs and attempts to generate synthetic or fake images. The DNN receives realistic images along with the synthetic images from the GNN. The DNN then outputs a binary decision indicating its prediction of whether the image received from the GNN is real or synthetic (i.e., fake). As the DNN improves at discriminating between real and synthetic images, the GNN improves at generating images to fool the DNN. The GNN and DNN are trained iteratively by minimizing their associated loss functions. For example, equilibrium is reached when a GNN can no longer fool a DNN. There is no need to explicitly include domain knowledge. Image features are learned automatically. The loss function of a GNN is learned and not predefined. In conditional GANs (CGANs), instead of generating samples from random inputs, a GNN generates output images conditional on the input images. Such systems can then be trained to learn a mapping from the input image space to the output or generated image space. This training process consists of learning an optimal set of values ​​for multiple parameters that determine the mapping. In other words, the mapping can be thought of as a function mapping with parameters that are adjustable and learned based on a set of training samples.

[0060] Preferably, the cGAN is a pixel-to-pixel (pix2pix) GAN. The pix2pix model is a type of cGAN in which the generation of an output image is conditional on an input, in this case, a processed, i.e., dilated, image of the extracted image by the DLM. The DNN is provided with both such processed and target images and must determine whether the target is a plausible transformation of the processed image. The GNN is trained via an adversarial loss, which prompts the GNN to generate plausible images in the target domain. The GNN is also updated via an L1 loss, which is measured between the generated image and the expected output image. This additional loss prompts the GNN model to create a plausible translation of the source image. More preferably, the GNN in the pix2pixGAN is a CNN. Even more preferably, the GNN in the pix2pixGAN is a CNN with a U-net architecture. The U-net is a CNN architecture for fast and accurate image segmentation.

[0061] Preferably, the cGAN is trained with training data comprising a set of real pairs, each set of real pairs preferably comprising a training-related inflated gridless table image and a corresponding table-grid image. Preferably, training the cGAN comprises a plurality of training steps, each of which preferably comprises one of the following steps a) or b) and step c): a) providing the DNN with real pairs obtained from a set of real pairs; b) providing the DNN with generated pairs including candidate false grid images and corresponding training-related inflated gridless table images obtained from the set of real pairs, where the candidate false grid images are obtained by the GNN by modifying the corresponding training-related inflated gridless table images using random data vectors; c) determining by the DNN an indication of whether the actual pair or generated pair provided to the DNN is a pair generated by the GNN; Preferably, the cGAN training comprises multiple iterative training cycles of the GNN and DNN. Preferably, each of the multiple iterative training cycles of the GNN and DNN comprises at least one of the multiple training steps. Preferably, during each training cycle, a loss function associated with each GNN or DNN is minimized until an associated stability cycle predicate is satisfied. Preferably, the cGAN is trained by minimizing a combined loss of the GNN or DNN associated loss function until a combined loss stability predicate is satisfied.

[0062] Preferably, each of the training related dilated gridless table images of the set of actual pairs is obtained by dilating a gridless table image.

[0063] Similar to the extracted images by DLM, the performance of the present invention, and specifically the performance of cGAN, improves when training-related dilated gridless table images are available, as discussed below.

[0064] More preferably, each of the training related dilated gridless table images of the set of actual pairs is obtained by: Converting gridless table images into grayscale images; applying binarization to the grayscale image by an adaptive Gaussian method; and Dilate the binarized grayscale image.

[0065] More preferably, each of the training related dilated gridless table images of the set of actual pairs is obtained by: Converting gridless table images into grayscale images; · Applying binarization to the grayscale image by adaptive Gaussian method; Dilation of the binarized grayscale image; Obtaining the outline of the dilated object in the dilated binary grayscale image; Inflating the outline containing the image; and A step of applying binarization to the dilated outline containing the image by means of an adaptive Gaussian method.

[0066] Preferably, a set of actual pairs, each comprising a training related inflated gridless table image and a corresponding table grid image, is obtained by: providing a plurality of cGAN training-related fixed-layout electronic documents, each of the fixed-layout electronic documents including a gridless table, the plurality of cGAN training-related fixed-layout electronic documents being convertible to a markup language; · converting a plurality of training-related fixed-layout electronic documents into corresponding documents in a markup language; identifying, in each of the corresponding markup language documents, table tags associated with gridless tables in the associated training-related fixed-layout electronic document; · for each gridless table, determining a table grid image based at least in part on the corresponding identified table tag; obtaining an image of a gridless table from each of the plurality of cGAN training-related fixed-layout electronic documents; and Dilating each of the obtained gridless table images.

[0067] More preferably, an image of the gridless table is obtained from each of the provided pdf documents by: determining, by the trained DLM, a bounding box for a gridless table in each of the cGAN training related fixed-layout electronic documents; and Extracting an image of a gridless table from each of the cGAN training related fixed-layout electronic documents based on the determined corresponding bounding box.

[0068] According to a preferred embodiment, a set of actual pairs, each comprising a training-related inflated gridless table image and a corresponding table grid image, is obtained by: providing a plurality of pdf documents each containing a gridless table; Convert multiple PDF documents to HTML documents; - identifying in each of the html documents table tags that are associated with gridless tables in the associated pdf document; · for each gridless table, determining a table grid image based at least in part on the corresponding identified table tag; obtaining an image of a gridless table from each of said pdf documents; and Dilating each of the obtained gridless table images.

[0069] According to a further preferred embodiment, an image of the gridless table is obtained from each of the provided pdf documents by: determining the bounding box of the gridless table in each pdf document by the trained DLM; and Extracting an image of the gridless table from each of said pdf documents based on the determined corresponding bounding box.

[0070] Similar to the training data for DLM, the use of a markup language to obtain table tags is easy to implement and provides qualitative information about the location of the invisible grid of a gridless table in an electronic document. From this information, a table grid image of the gridless table can be easily derived. Preferably, HTML is used as the markup language because of its widespread use.

[0071] To determine a gridded image of the gridless table, a simple embodiment of the present invention further provides the steps of: determining a table grid image for the processed extracted image by a trained GNN comprising a trained cGAN; and combining the determined table grid image and the gridless table from the provided fixed-layout electronic document with the gridded image. The table grid image and the gridless table can be combined to obtain the gridded image using any technique known in the art, such as image merging or superimposing one image on the other. The technique used may or may not depend on the file format of the image used in implementing the present invention.

[0072] As with any machine learning technique (MLT), the quality of the generated output depends heavily on the quality of the training data on which the MLT model is trained. In the case of a trained cGAN, a lack of training data may or may not result in poorly or corrupted determined table grid images. Combining such poorly or corrupted determined table grid images and gridless tables with gridded images may or may not impair processing and / or parsing. Image restoration techniques known in the art can be used to improve the quality of poorly or corrupted determined table grid images.

[0073] According to a preferred embodiment, the present invention further comprises training an artificial neural network (ANN) with training data comprising training pairs, each including a training-related table grid image and a corresponding corrupted table grid image. Preferably, wherein the ANN is an autoencoder. According to a preferred embodiment, the present invention further comprises determining a restored table grid image from the corrupted table grid image by a trained GNN decision table grid image constructed by the trained ANN. Preferably, wherein the determined restored table grid image and the gridless table from the provided fixed-layout electronic document are combined into a gridded image of the gridless table.

[0074] Example

[0075] The present invention is further described by the following non-limiting examples which further illustrate the present invention and are not intended to, nor should they be construed to, limit the scope of the invention.

[0076] Example 1: Parsing a Document

[0077] This example provides a general overview of the workflow according to the present invention. See FIG.

[0078] FIG. 1 provides a schematic overview of the overall workflow of the present invention. This workflow comprises processing a gridless table and its associated document. According to this embodiment, a fixed-layout electronic document (1) containing a gridless table is provided, which is a PDF document (1′), which is converted into an image file (1″). This image file is fed to a trained CNN (2). The training of this DLM is discussed in Example 2. Using this trained CNN, an image (3) of the gridless table is extracted and then processed by performing at least a dilation step (4). The processing of the extracted image is discussed in Example 4. The processed extracted image (5) is fed to a trained GNN, which comprises a trained GAN (6). The training of this GAN (6) is discussed in Example 3. Using the trained GNN, a table grid image (7) for the processed extracted image is determined. The table grid image is then superimposed (8) on the extracted image of the gridless table to obtain a gridded image of the gridless table (9). This gridded image is fed into an OCR tool (10) to obtain the tabular information in a gridless table in a word-processable format (11).

[0079] Example 2: Training of DLM

[0080] This embodiment relates to the training of DLM, particularly CNN. Please refer to FIGS. 2 to 4.

[0081] Figures 2 and 3 illustrate the acquisition of training data for training a DLM. The training data relates to a plurality of training-related electronic documents (1), each of which includes a gridless training table (12) and an associated label (13) that indicates the bounding box of the training table. By training a DLM with such training data, the trained DLM can determine the bounding box (15) of the gridless table (12) within the fixed-layout electronic document (1).

[0082] In this example, the DLM is a CNN, specifically based on YOLO, a publicly available tool. YOLO is a powerful neural network that draws bounding boxes around detected objects in an image. Darknet is an open-source framework used to train neural networks and serves as the base for YOLO. In this example, a YOLOV3 network is used to identify gridless tables.

[0083] To train the YOLOV3 network, we first converted image-based PDF documents into HTML output. We then chopped up the HTML content and parsed out tables. We then detected and painted table tags from the HTML source code. This allowed us to determine the coordinates of the table in the PDF document, as seen in Figure 2, thereby forming the training set fed to YOLOV3. From this information, we determined the x,y coordinates of the table as well as its height and width.

[0084] YOLOV3 uses IOU and BCE as loss functions, and logistic regression for object localization. IOU can be calculated as the area of ​​the intersection divided by the area of ​​the union of the two boxes. IOU must be between 0 and 1, inclusive. To be able to predict a bounding box, IOU needs to be approximately 1 between the predicted bounding box and the ground truth bounding box. Currently, the cell is offset from the top left of the image by (cx,cy), and the bounding box has a width Pw and height Ph ahead.

[0085] The predictions of the YOLOV3 network correspond as follows:

number

[0086] The loss function of the YOLOV3 network is shown below.

number

[0087] Figure 4 illustrates the iteration of localization prediction, for example to determine the bounding box of a gridless table in an image. During training, the bounding box predictions (15', 15'', 15''') are iteratively refined until the difference between the bounding box of the gridless table and the ground truth (16) satisfies a predicate.

[0088] Example 3: Training a GAN

[0089] This embodiment relates to training a GAN, particularly a cGAN. Please refer to Figures 5 to 10.

[0090] Figure 5 illustrates a general overview of the GAN architecture. A GAN is implemented by a system in which two neural networks, a GNN (14) and a DNN (21), compete against each other in a zero-sum game framework. The two neural networks are created and trained through competition. The first neural network is implemented as a generator system and is called the generator neural network (GNN). The second neural network is implemented as a discriminator system and is called the discriminator neural network (DNN). The GNN starts with random inputs (17) and attempts to generate synthetic or fake images (18). The DNN receives realistic images along with the synthetic images from the GNN (19; 20). The DNN then outputs a binary decision indicating its prediction of whether the image received from the GNN is real or fake (22). As the DNN improves its ability to discriminate between real and synthetic images, the GNN improves its ability to generate images that deceive the DNN. GNNs are trained by minimizing their associated loss functions. The loss functions of GNNs are learned and not predefined. DNNs are trained by minimizing their associated loss functions (23). GNNs and DNNs are trained iteratively by minimizing their associated loss functions. For example, equilibrium is reached when the GNN can no longer fool the DNN. There is no need to explicitly include domain knowledge. Image features are learned automatically.

[0091] In this example, the GAN uses a PatchGAN architecture. The binary cross entropy method is used as the loss function for the DNN. Furthermore, to train the model, the output images of YOLOV3 (Example 2) are simulated, thereby providing coordinate information on a gridless table. These images are preprocessed using computer vision dilation and passed to the DNN as real images, while the output images of the GNN are passed as fake images. The trainable parameter of the DNN is set to false, and the loss function uses binary cross entropy. Finally, all images are passed to the GAN, with losses set to binary cross entropy and mean absolute error. The GAN in this example is a cGAN and is therefore conditioned on its input. The overall loss function of the GAN consists of a generator loss function and a classifier loss function. The classifier reduces the real and generated images to patches and calculates the entropy loss for each patch. Since this example uses a patch GAN architecture, the overall loss function of the GAN corresponds to the following:

number

[0092] Figures 6 to 10 illustrate table grid prediction for a gridless table extracted using a GAN. Figure 6 is an inflated image of the gridless table (3). Figure 7 is the output of the GAN. This output constitutes a corrupted grid image (7'). This corrupted grid image can be restored using image restoration techniques to obtain a restored table grid image (7''). This restored table grid image can be superimposed with a restored table grid image to obtain a gridded image of the gridless table (9). This gridded image can be parsed using OCR to obtain word readable information of the gridless table (11).

[0093] Example 4: Image Dilation

[0094] This embodiment relates to processing before and after image dilation. Please refer to Figs. 11 to 15.

[0095] Figures 11 through 15 illustrate preferred image processing steps before and after dilation to improve the processing of gridless tables. The original color image (24) is converted to a grayscale image. After conversion, adaptive binarization is applied to the grayscale image because a static threshold is not used and every document maintains different font styles, font sizes, indentations, calligraphy, etc. Thus, a binarized grayscale image (25) is obtained. This image highlights the table header font. These steps also help dynamically infer the threshold value for image pixels. The binarized value is calculated using a weighted sum of neighboring values ​​using a Gaussian window. The binarized grayscale image is then dilated and eroded. Thus, a dilated image (26) of the binarized grayscale image is obtained. Erosion and dilation require two inputs: the image and a structuring element. The structuring element is called a kernel. During dilation, the kernel performs convolution and max pooling, thereby expanding the area covered by the text data. During erosion, the kernel performs the exact opposite operation. That is, convolution and min-pooling are performed to reduce the appearance of the text data. After the dilation step, the outline of the dilated object in the dilated image of the previous dilation step (27) is obtained. Then, dilation is applied to the outline containing the image, and the dilated outline containing the image is binarized again to obtain the binarized dilated outline containing the image (28).

[0096] The present invention is not limited to any of the implementations described above, and it is envisaged that several modifications can be made to the presented working examples without reviewing the scope of the appended claims. The method according to the invention can be implemented in many different ways without departing from the scope of the invention.

Claims

1. 1. A computer-implemented method for improved table processing, the method comprising: Providing a fixed-layout electronic document that includes a gridless table; training a deep learning model (DLM) with training data associated with a plurality of training-related electronic documents, each of which includes a gridless training table and an associated label indicating a bounding box of the training table; determining a bounding box of the gridless table in the fixed-layout electronic document using the trained DLM; extracting an image of the gridless table from the fixed-layout electronic document based on the determined bounding box; processing the extracted image by performing at least a dilation step; The method comprises: training a conditional generative adversarial network (cGAN) comprising a generator neural network (GNN) and a discriminator neural network (DNN) with training data comprising a set of real pairs each comprising a training-related dilated gridless table image and a corresponding table grid image, wherein training the cGAN comprises a plurality of training steps, each of which comprises: a) providing said DNN with actual pairs obtained from said set of actual pairs; or b) providing the DNN with generated pairs including candidate false grid images and corresponding training-related inflated gridless table images obtained from the set of real pairs, the candidate false grid images being obtained by the DNN by modifying the corresponding training-related inflated gridless table images using random data vectors; and c) determining by the DNN an indication of whether the actual pair or the generated pair provided to the DNN is a pair generated by the GNN; The method comprises the steps of: training the cGAN comprises a plurality of GNN and DNN iterative training cycles, each comprising at least one of the plurality of training steps, wherein during each training cycle, a loss function associated with each GNN or DNN is minimized until an associated stable cycle predicate is satisfied, and the cGAN is trained by minimizing a joint loss of the loss functions associated with the GNN and DNN until a joint loss stable predicate is satisfied; - determining a table grid image for the processed extracted image by the trained GNN included in the trained cGAN; Further provided with The method comprises: training an artificial neural network (ANN) with training data comprising training pairs, each comprising a training related table grid image and a corresponding corrupted table grid image; determining, by the trained ANN, a restored table grid image from the determined table grid image of the trained GNN; Further provided with the determined restored table grid image and the gridless table from the provided fixed-layout electronic document are combined into a gridded image of the gridless table; The method further comprises providing the gridded image to an OCR tool to generate a parsed document from the fixed-layout electronic document to obtain tabular information of the gridless table in a word-processable format. method.

2. 2. The method of claim 1, wherein the step of processing the extracted image includes, prior to the dilation step: converting the extracted image to a grayscale image; and applying binarization to the grayscale image by an adaptive Gaussian method; wherein the dilation step is performed on the binarized grayscale image.

3. 3. The method of claim 1, wherein the step of processing the image comprises, after the dilation step: obtaining an outline of the dilated object in the dilated image by the dilation step; dilating the outline to include an image; and applying binarization to the dilated outline containing the image by an adaptive Gaussian method; The method further comprises:

4. A method comprising the steps of claims 2 and 3.

5. The method according to any one of claims 1 to 4, wherein the step of processing the extracted image comprises an erosion step after the dilation step.

6. 6. The method of claim 1, wherein the set of actual pairs, each comprising a training-related inflated gridless table image and a corresponding table grid image, comprises: providing a plurality of PDF documents, each containing a gridless table; converting the plurality of PDF documents into HTML documents; identifying, in each of the HTML documents, a table tag associated with the gridless table in the associated PDF document; for each of the gridless tables, determining a table grid image based at least in part on the corresponding identified table tag; obtaining an image of the gridless table from each of the PDF documents; dilating each of the obtained images of the gridless table; A method obtained by

7. 7. The method of claim 6, wherein from each of the provided PDF documents, the image of the gridless table is: determining a bounding box of the gridless table in each PDF document using the trained DLM; extracting an image of the gridless table from each of the PDF documents based on the determined corresponding bounding boxes; A method obtained by

8. 8. The method of claim 1, wherein the plurality of training-related electronic documents, each comprising a gridless training table and an associated label indicating a bounding box of the training table, comprises: providing a plurality of PDF documents, each containing a gridless table; converting the plurality of PDF documents into HTML documents; identifying, in each of the HTML documents, a table tag associated with the gridless table in the associated PDF document; Associating a label of the bounding box of the gridless table with each of the plurality of PDF documents, the label being based at least in part on the corresponding identified table tag; The method obtained by

9. 9. The method of claim 1, wherein the DLM is one or more of an artificial neural network (ANN), a recurrent neural network (RNN), or a convolutional neural network (CNN).

10. The method of any one of claims 1 to 9, wherein the DLM is a CNN.

11. A computer system for improved table processing, the computer system being configured to perform the computer-implemented method of any of claims 1 to 10.

12. A computer program product for improved table processing, comprising instructions that, when said computer program product is executed by a computer, cause said computer to perform the computer-implemented method of any of claims 1 to 10.

13. 11. The method of claim 1, wherein generating a parsed document from the fixed-layout electronic document comprises performing optical character recognition (OCR) of text associated with tabular data from the gridded image of the gridless table.

14. 12. A method of using the computer system of claim 11 to generate a parsed document from the fixed-layout electronic document.

15. 15. The method of claim 14, wherein generating a parsed document from the fixed-layout electronic document comprises performing optical character recognition (OCR) of text associated with tabular data from the gridded image of the gridless table.

16. 13. A method of using the computer program product of claim 12 to generate a parsed document from the fixed-layout electronic document.

17. 17. The method of claim 16, wherein generating a parsed document from the fixed-layout electronic document comprises performing optical character recognition (OCR) of text associated with tabular data from the gridded image of the gridless table.

Images