Automated depiction and extraction of tabular data using machine learning

Abstract

A computer-implemented method for automatically extracting tabular data from images using a machine learning model (122) includes receiving a set of images of tabular data and a set of labeled data respectively corresponding to the images of tabular data. The method further includes training a first neural network to delineate cells (440) of the tabular data using the labeled data, and training a second neural network to determine content of the cells (440) in the tabular data using the labeled data. The method further includes, upon receiving an input image (112) containing first tabular data without any labeled data, generating an electronic output corresponding to the first tabular data by determining a structure of the first tabular data using the first neural network and extracting content of the first tabular data using the second neural network.

Description

Technical Field

[0001] This invention generally relates to computer technology, and more specifically to a content recognition system for automatically determining the structure of information presented as tabular data and further determining the content within the tabular data. Background Technology

[0002] A vast amount of literature (including books, papers, articles, blogs, reports, etc.) is generated in any field (such as medicine, law, engineering, science, business, etc.). Additionally, literature is generated as part of business, such as invoices, quotations, account statements, contracts, etc. All such literature is typically written for human-to-human exchange without any plans for machine understanding. For distinction, documents are described as “natural language” documents, as opposed to documents or files written for machine readability and understanding. With the advent of artificial intelligence and other advances in machine learning, machines such as computers can analyze corpora of information from literature to provide insights undetectable to humans. Various document parsing and analysis systems exist that facilitate the interpretation of the content of literature and provide functions such as search capabilities. In many published documents, important information is contained in tables included within the document. This is especially true for medical literature (such as clinical studies), where much information about results across different groups is contained only in these tables and not in the rest of the text.

[0003] Currently, these tables are difficult or impossible to parse because there is no standard format or structure. Humans are very good at this, but machines cannot yet understand the structure of tables to extract meaning. Therefore, it is necessary to solve the above-mentioned problems in this field. Summary of the Invention

[0004] From one perspective, the present invention provides a computer-implemented method, system, and computer program product for determining the content of tabular data, comprising: performing a training operation of training a machine learning model using a processor's machine learning algorithm, the training operation comprising: receiving a set of images of tabular data and a set of labeled data corresponding to the images of the tabular data respectively; training a first neural network using the labeled data to depict the tabular data from the set of images as units; and training a second neural network using the labeled data to determine the content of the units in the tabular data from the set of images; receiving an input image containing labeled data of the first tabular data but without indicating the structure of the first tabular data; and generating an electronic output corresponding to the first tabular data by using the first neural network to determine the structure of the first tabular data and using the second neural network to extract the content of the first tabular data.

[0005] In another aspect, the present invention provides a computer program product for determining the contents of tabular data, the computer program product comprising a computer-readable storage medium readable by processing circuitry, and storing instructions for execution by the processing circuitry to perform a method for performing the steps of the present invention.

[0006] In another respect, the present invention provides a computer program stored on a computer-readable medium and loadable into the internal memory of a digital computer, the computer program including software code portions for performing the steps of the present invention when the program is run on the computer.

[0007] According to one or more embodiments of the present invention, a computer-implemented method for automatically extracting tabular data from images using a machine learning model includes receiving a set of images containing tabular data and a set of labeled data corresponding to the images of the tabular data. The method further includes training a first neural network using the labeled data to depict the tabular data as units, and training a second neural network using the labeled data to determine the content of the units in the tabular data. The method also includes, upon receiving an input image containing the first tabular data without any labeled data, determining the structure of the first tabular data using the first neural network and extracting the content of the first tabular data using the second neural network to generate an electronic output corresponding to the first tabular data.

[0008] According to one or more embodiments of the present invention, a system includes a memory and a processor coupled to the memory. The processor is programmed with a machine learning algorithm to execute a method including training a machine learning model. The training includes receiving a set of images of tabular data and a set of labeled data corresponding to the images of the tabular data, respectively. The training also includes using the labeled data to train a first neural network to depict the tabular data from the set of images as cells. The training further includes using the labeled data to train a second neural network to determine the content of the cells in the tabular data from the set of images. The method further includes, after training, generating an electronic output corresponding to the first tabular data by, upon receiving an input image containing labeled data that includes the first tabular data but does not indicate the structure of the first tabular data, determining the structure of the first tabular data using the first neural network and extracting the content of the first tabular data using the second neural network, upon receiving labeled data containing the first tabular data but not indicating the structure of the first tabular data.

[0009] According to one or more embodiments of the present invention, a computer program product includes a memory storage device having computer-executable instructions stored thereon. When executed by a processor, the computer-executable instructions cause the processor to perform a method including training a machine learning model. The training includes receiving a set of images of tabular data and a set of labeled data corresponding to the images of the tabular data, respectively. The training also includes using the labeled data to train a first neural network to depict the tabular data from the set of images as units. The training further includes using the labeled data to train a second neural network to determine the content of the units in the tabular data from the set of images. The method, after training, further includes: upon receiving an input image containing labeled data that indicates the structure of the first tabular data but does not indicate the structure of the first tabular data, determining the structure of the first tabular data using the first neural network and extracting the content of the first tabular data using the second neural network to generate an electronic output corresponding to the first tabular data.

[0010] Embodiments of the present invention facilitate the use of machine learning to automatically parse and understand tabular data in digital documents, and subsequently use the documents for comprehensive analysis. According to one or more embodiments of the invention, tabular data in a document is interpreted even though the tabular data is presented in various layouts, styles, information types, and formats, and there is no explicit description of the encoding / formatting information regarding the structure of the tabular data. Therefore, embodiments of the present invention promote improvements to automatic content recognition systems that currently cannot automatically interpret presented tabular data.

[0011] Additional technical features and advantages are achieved through the technology of this invention. Embodiments and aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, reference is made to the specific embodiments and accompanying drawings. Attached Figure Description

[0012] The details of the exclusive rights claimed herein are specifically pointed out and explicitly asserted in the claims at the conclusion of the specification. The foregoing and other features and advantages of embodiments of the invention will become clear from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A system for content recognition according to one or more embodiments of the present invention is described;

[0014] Figure 2 A sample input image in image format is depicted in the example scene;

[0015] Figure 3 The electronic output of tabular data from a marked-up input image is depicted in the example scenario;

[0016] Figure 4A flowchart is depicted for a method for training a machine learning model to extract tabular data from an input image according to one or more embodiments of the present invention;

[0017] Figure 5 An example structure of a machine learning model according to one or more embodiments of the present invention and a data flow for training the machine learning model are described.

[0018] Figure 6 A computer system according to one or more embodiments of the present invention is described;

[0019] Figure 7 A cloud computing environment according to one or more embodiments of the present invention is described; and

[0020] Figure 8 A model layer according to one or more embodiments of the present invention is depicted.

[0021] The illustrations depicted herein are illustrative. Many variations can be made to the illustrations or operations described herein without departing from the scope of the invention. For example, actions can be performed in different orders, or actions can be added, deleted, or modified. Furthermore, the term "coupled" and its variations describe a communication path between two elements and do not imply a direct connection between the elements without any intervening elements / connections. All such variations are considered part of the specification.

[0022] In the following detailed description of the accompanying drawings and disclosed embodiments, the various elements shown in the figures are provided with reference numerals of two or three numbers. With a few exceptions, the leftmost number(s) of each reference numeral(s) corresponds to the drawing in which its element was first illustrated. Detailed Implementation

[0023] Public documents include important information presented in tabular form (“tables”), i.e., tabular data included in documents(s). Documents may include information presented in tabular form, where the table organizes information across multiple rows and columns. Such tabular data facilitates the structured presentation of information, the summarization of key information, and the presentation of results and / or observations. Typically, tabular data is included in a document as an image without any corresponding information describing the structure used to tabulate the data. This structure indicates how the data is depicted, for example, as rows, columns, cells, and other such components of a table.

[0024] In many cases, understanding the information in tables is crucial for analyzing various documents within a literature body. Therefore, in machine learning, it is essential to read and understand tabular data to perform comprehensive analysis using literature. However, interpreting tabular data in literature presents technical challenges because tabular data is presented in various layouts, styles, information types, and formats, and there is no explicit description of the encoded / formatted information regarding the structure of the tabular data. While humans are adept at reading such diverse tabular data layouts and styles, interpreting tabular data remains a technical challenge. Some examples of differences in tables can include varying row and column heights, merging of first neural network memory units, different numbers of columns, different numbers of rows within different columns, or boundaries distinguishing different types of units, etc.

[0025] For example, in the case of medical literature such as clinical studies, information about results across different groups is included in such tables. It should be understood that medical literature is merely an example; tabular data is presented in literature from various other fields, such as engineering, science, business, law, etc., and embodiments of the invention are not limited to using documents from any particular field.

[0026] Furthermore, in many cases, data presented in tabular format within a document is not present in the remaining text of the document. Therefore, for automatic content recognition systems that automatically parse available documents to interpret their content, there is a technical challenge in decrypting such tabular data. Currently, parsing such tabular data is a technical challenge because there is no standard format or structure for tabular data. Data is often tabulated for human understanding, and humans are generally adept at parsing such data, which is formatted in tabular format using a self-organizing structure and does not necessarily require a description of the format.

[0027] Embodiments of the present invention address such technical challenges and facilitate machine autonomous understanding of the structure (or format) of tabular data in order to extract its meaning. One or more embodiments of the present invention facilitate training machine learning (ML) models to receive tabular data without any description of the structure, description, or labels, thereby specifying the format of the tabular data, identifying discrete columns, rows, and / or cells, and extracting the content included in the table (e.g., text, images, etc.). In one or more embodiments of the present invention, the extracted content is output in a structured and meaningful format.

[0028] In one or more embodiments of the invention, the ML model receives tabular data in the form of a digital image of a table, without any specified depiction or labels. For example, the digital image can be any digital image file format, such as bitmap, Joint Image Experts Group (JPEG), Portable Web Graphics (PNG), or any other format. In one or more examples, the image of the tabular data is extracted from a document, which in turn is any digital content format, such as Portable Document Format (PDF), DOC, etc. The ML model can then identify discrete columns, rows, and cells, and extract the text included therein in a structured and meaningful manner. In one or more embodiments of the invention, the extracted data is output using a computer-readable format such as Hypertext Markup Language (HTML), Extended Markup Language (XML), or any other such computer-readable format.

[0029] Figure 1 A system 100 for content recognition according to one or more embodiments of the present invention is described. System 100 may be... Figure 8 The content recognition workload 96 is described herein. Alternatively, as indicated herein, system 100 may be an adjunct system for content recognition. System 100 particularly includes a content recognition device 120 that receives an input image 112 of tabular data from an electronic document 110. The received input image 112 does not have any accompanying structural information describing the format of the tabular data in the input image 112.

[0030] Content recognition device 120 uses ML model 122 to parse the content of input image 112 and generates electronic output 130 including tabular data of markup format 132. The tabular data of markup format 132 includes a description of the structure of the tabular data, such as describing the cells in the tabular data. For example, the described format describes rows and columns of tabular data that divide the data into multiple cells. In one or more embodiments of the invention, content recognition device 120 may also extract text from input image 112 and include it in electronic output 130. The extracted text can be presented so that its meaning can be determined based on the relationships between cells in the tabular data.

[0031] Training data 140 is used to train an ML model 122 to parse the content of an input image 112. Training data 140 includes the extracted image 142 as table data paired with labeled data 144 corresponding to the structure and content of the table data in the extracted image 142. Training data 140 includes multiple (e.g., thousands, millions, etc.) such corresponding pairs of the extracted image 142 as table data and the labeled data 144 corresponding to the table structure and content. Training data 140 is used by the ML model to refine the identification of the table structure and content in a given image (such as input image 112) of table data without requiring a corresponding description of the table data structure.

[0032] Figure 2 Example input image 112 in the example scene is depicted, and Figure 3 The electronic output 130 in the example scenario is depicted. Figure 2 In the illustrated example, input image 112 includes columns 410 and rows 420. As shown, one or more rows may include sub-rows 430. Rows 420 and columns 410 form multiple cells 440. The structure of the tabular data is depicted in the electronic output 130 as a markup format 132, such as... Figure 3 As shown in the figure. The description in markup format 132 describes the column description 510 corresponding to column 410. Column description 510 indicates the number of columns 410. Column description 510 also describes the header of column 410. For example, in other instances where column 410 has other attributes (such as sub-columns), column description 510 may include other characteristics to describe column 410.

[0033] Figure 3 The markup format 132 shown also includes a line depiction 520 corresponding to line 420. Line depiction 520 describes multiple lines 420 in the input image 112. Line depiction 520 also includes a header for line 420. Line depiction 520 also includes a sub-description 530 for line 420 that includes one or more sub-lines 430.

[0034] also, Figure 3 The electronic output 130 shown includes output from... Figure 2 The text 540 extracted from unit 440 shown. According to... Figure 2 The input image 112 shown depicts a structure to include the extracted text 540.

[0035] Figure 4A flowchart illustrating a method 600 for training an ML model 122 of a content recognition device 120 to extract tabular data from an input image 112, according to one or more embodiments of the present invention, is shown. Method 600 includes training the ML model 122 using training data 140 at block 610. The training data includes a training image 142 having tabular data and corresponding training labeled data 144 describing the structure and content of the tabular data in the training image 142. In one or more examples, the training labeled data 144 may be multiple XML files providing labels for the tabular data in the corresponding image file or a document file including the training image 142.

[0036] Training labeled data 144 provides the ML model 122 with a normalized, structured representation of the tabular data in training images 142. In one or more embodiments of the invention, training images 142 are generated by converting tabular data from multiple documents (e.g., PDF files) into image files (e.g., JPEGs). In one or more examples, such conversion can be performed automatically, thereby generating training data 140 itself without human effort. Training data 140 provides multiple (thousands, millions) samples that provide training images 142 accompanied by ground truth representations of the tabular data in labeled representation 144.

[0037] In one or more embodiments of the present invention, the ML model 122 may have a deep learning network architecture that can be trained with training data 140. Figure 5 An example structure of an ML model 122 and a data flow for training the ML model 122 are described according to one or more embodiments of the present invention.

[0038] In one or more embodiments of the present invention, ML model 122 is a deep neural network learning architecture that uses an encoder-decoder model with multiple layers having convolutional neural network layers (CNN), attention modules, and recurrent neural network layers (RNN). In one or more embodiments of the present invention, ML model 122 includes an encoder 710 and two decoder sets, a first decoder set 720 and a second decoder set 730.

[0039] Encoder 710 includes one or more CNNs that analyze training images 142 and use a set of convolutional filters to capture visual features of the training images 142. The parameters of the convolutional filters are updated via backpropagation during training. The visual features are collectively referred to as feature maps. Each pixel in a feature map is a high-dimensional feature vector that describes the pattern of a corresponding local patch (e.g., 16x16 pixels) in the training image 142. Encoder 710 forwards the feature maps to a first decoding set 720 and a second decoding set 730.

[0040] Decoding sets 720 and 730 may include one or more units of an RNN and a focus module. In one or more examples, the RNN may be implemented as a Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) unit. The first decoding set 720 may be used to determine the structure of the tabular data in the training image 142 (and the input image 112). Therefore, the first decoding set 720 may also be referred to as a structure-decoding neural network set. The second decoding set 730 may be used to determine the content of one or more units in the tabular data of the training image 142 (and the input image 112). Therefore, the second decoding set 730 may also be referred to as a content-decoding neural network set.

[0041] The structure decoding neural network ensemble (720) includes a structure attention module 722 and a structure decoder 724, the structure decoder 724 being an RNN. The structure attention module 722 is a neural network that learns how to assign different degrees of focus to different parts of a feature map encoded from the training image 142 (and, the input image 112) to interpret structure. A higher degree of focus in a region of the feature map allows the structure decoder 724 to utilize more information from that region. The degree of focus at a given location in the feature map is determined by multiplying the set of weight parameters by the feature map itself and the “hidden state” of the structure decoder 724. In an RNN, in the context of a recursive layer, the “hidden state” is a value shared during recursion to provide a representation of the previous input. The weight parameters of the attention module 722 are adjusted via backpropagation during training, as will be described further. The attention module 722 analyzes the feature map of the training image 142 to determine the degree of focus. The structure decoder RNN 724 also includes its own weight parameters updated via backpropagation. The structure decoder RNN 724 and the structure attention module 722 are repeatedly trained together to adjust the weight parameters of the two networks (structure decoder RNN 724 and structure attention module 722).

[0042] The weights are updated to minimize the difference between the estimated label representation 730 generated by the ensemble of the structure decoding neural network (720) and the known table structure 144 for the training image 142. (As...) Figure 5As depicted, the labeled representation estimate is compared with the tabular structure 144 to compute a loss function 740. The loss function 740 indicates the difference between the extracted structure and known structures in the training data 144. The loss function 740 is provided as feedback to the structure decoder RNN 724, the structure attention module 722, and the encoder 710. Alternatively, a structure accuracy score is computed based on this difference and provided as feedback. The weights of the structure decoder RNN 724, the structure attention module 722, and the encoder 710 are updated based on the structure accuracy score so that the labeled representation estimate 730 is closer to the tabular data 144. In one or more examples, the structure accuracy score is compared with a predetermined accuracy threshold. If the structure accuracy score exceeds the predetermined structure accuracy threshold, the set of structure decoding neural networks 720 is labeled as trained. Alternatively, if the difference between the structure accuracy score and the predetermined structure accuracy threshold is greater than a certain threshold, training of the set of structure decoding neural networks 720 continues, i.e., the weights are adjusted.

[0043] An ensemble of content decoding neural networks 730 is trained and operates in a similar manner to the ensemble of structure decoding neural networks 720. The ensemble of content decoding neural networks 730 includes a content attention module 732 and a content decoder 734, which is an RNN. The content attention module 732 is a neural network that learns how to assign different degrees of focus to different parts of the feature map encoded from the training image 142 (and, the input image 112) to interpret content. A higher degree of focus in a certain region of the feature map allows the content decoder 734 to utilize more information from that region. The degree of focus at a given location in the feature map is determined by multiplying the set of weight parameters by the feature map itself, as well as the hidden states of the structure decoder 724 and the content decoder 734. The weight parameters of the content attention module 732 are adjusted via backpropagation during training, as will be described further. For example, the content attention module 732 analyzes the feature map of the training image 142 to be used by the content decoder RNN 734. The content decoder RNN 734 also includes its own weight parameters updated via backpropagation. The content decoder RNN734 and the content attention module 732 are repeatedly trained together to adjust the weight parameters of the two networks (content decoder RNN734 and content attention module 732).

[0044] The weights are updated to minimize the difference between the representation estimate 730 generated as a set of content decoding neural networks and the known content in the tabular data for the training images 142. (As...) Figure 5As shown, the label representation estimate 730 is compared with the tabular data 144 to calculate a loss function 740. The loss function 740 indicates the difference between the extracted content and the known content in the training data 144. The loss function 740 is provided as feedback to the content decoder RNN 734, the content attention module 732, and the encoder 710. Alternatively, a content accuracy score is calculated based on this difference and provided as feedback. The weights of the content decoder RNN 734, the content attention module 732, and the encoder 710 are updated based on the content accuracy score to make the label representation estimate 730 closer to the tabular data 144. In one or more examples, the content accuracy score is compared with a predetermined accuracy threshold. If the accuracy score exceeds the predetermined accuracy threshold, the set of content decoding neural networks 730 is marked as trained. Alternatively, if the accuracy score differs from the predetermined accuracy threshold by more than a certain threshold, training of the set of content decoding neural networks 730 continues, i.e., the weights are adjusted.

[0045] Therefore, to depict tabular data presented in image format, in one or more embodiments of the present invention, the ML model 120 includes three neural networks—an encoder 710, a structure attention module 722, and a structure decoder 724. The encoder 710 determines feature maps from training images, and these feature maps are forwarded to the structure attention module 722. The structure attention module 722 determines the degree of focus from the feature maps. The structure decoder uses the degree of focus and the feature maps to determine a descriptive estimate of the structure of the tabular data. Based on the difference between the estimate and the labeled representation from the training data 144, the weights of the encoder 710, structure attention module 722, and structure decoder 724 are updated / adjusted.

[0046] Additionally, the ML model 120 includes a content attention module 732 and a content decoder 734 for extracting content from tabular data, both of which work in conjunction with the encoder 710. Here, the content attention module 732 uses feature maps determined by the encoder 710 to determine the degree of focus for content extraction. The content decoder 734 then uses the feature maps and the degree of focus to determine a content estimate. The difference between the content estimate and the actual content from the training data is used to adjust the weights of the encoder 710, the content attention module 730, and the content decoder 734. In one or more examples, the content attention module 732 uses input from the structural decoder 724. In one or more examples, all neural networks in the ML model 120 are trained in combination.

[0047] Return to reference Figure 4 The flowchart of method 600 shows that, in one or more embodiments of the present invention, method 600 further includes receiving an input image 112 comprising tabular data having an unknown structure at block 620. In other words, the content recognition device 120 is unaware of the structure of the tabular data in the input image.

[0048] In box 630, content recognition device 120 uses a trained ML model 122 to determine a depiction of the structure of tabular data in input image 112. Specifically, a structure decoding neural network facilitates the identification of the depiction of the tabular data based on weights set during training. This depiction includes labeled representations of cells 440 in input image 112. The depiction also includes labeled representations of columns 410 and rows 420 from the tabular data. The depiction also identifies boundaries that may exist between sub-rows and sub-columns in the tabular data.

[0049] Furthermore, method 600 includes using the set of content decoding neural network 730 in box 640 to extract content from tabular data in input image 112.

[0050] Method 600 further includes merging the extracted content and the depiction of tabular data from input image 112 at box 650. This merging includes adding the extracted content to a markup representation of the depicted structure of the tabular data. Therefore, in one or more embodiments of the invention, a markup file is generated including tags based on a markup language used to characterize the tabular structure (see...). Figure 3 The tag file also includes the extracted content embedded within the tags.

[0051] In this way, ML model 122 is used to automatically analyze input image 112 to identify the structure of tabular data from the input image and further extract the content of the tabular data. Furthermore, a markup format is used to represent the tabular data. It should be noted that although a markup format is used throughout the examples described herein, in one or more embodiments of the invention, any other electronic formatting or protocol (such as comma-separated volumes (CSV)) or any other machine-readable format may be used to represent the tabular data.

[0052] Turn now Figure 6A computer system 800 is generally illustrated according to an embodiment. The computer system 800 may be an electronic computer framework comprising and / or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein. The computer system 800 can be easily extended, scalable, and modularized, with the ability to be changed to different services or to reconfigure certain features independently of other features. The computer system 800 may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, the computer system 800 may be a cloud computing node. The computer system 800 can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system 800 can be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may reside in local and remote computer system storage media, including memory storage devices.

[0053] like Figure 6 As shown, the computer system 800 has one or more central processing units (CPUs) 801a, 801b, 801c, etc. (collectively or commonly referred to as processors 801). Processors 801 can be single-core processors, multi-core processors, computing clusters, or any number of other configurations. Processors 801 (also referred to as processing circuitry) are coupled to system memory 803 and various other components via system bus 802. System memory 803 may include read-only memory (ROM) 804 and random access memory (RAM) 805. ROM 804 is coupled to system bus 802 and may include a basic input / output system (BIOS) that controls certain basic functions of computer system 800. RAM is a read-write memory coupled to system bus 802 for use by processor 801. System memory 803 provides temporary memory space for instructions during operation. System memory 803 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0054] Computer system 800 includes an input / output (I / O) adapter 806 and a communication adapter 807 coupled to a system bus 802. I / O adapter 806 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 808 and / or any other similar component. I / O adapter 806 and hard disk 808 are collectively referred to herein as mass storage device 810.

[0055] Software 811 executing on computer system 800 may be stored in mass storage device 810. Mass storage device 810 is an example of a tangible storage medium readable by processor 801, wherein software 811 is stored as instructions for execution by processor 801 to operate computer system 800, such as those described below with respect to the various figures. Examples of computer program products and the execution of such instructions are discussed in more detail herein. Communication adapter 807 interconnects system bus 802 with network 812, which may be an external network, enabling computer system 800 to communicate with other such systems. In one embodiment, system memory 803 and a portion of mass storage device 810 jointly store an operating system, which may be any suitable operating system, such as z / OS or AIX from IBM, to coordinate... Figure 6 The functions of the various components shown are illustrated.

[0056] Additional input / output devices are shown connected to system bus 802 via display adapter 815 and interface adapter 816. In one embodiment, adapters 806, 807, 815, and 816 may be connected to one or more I / O buses, which are connected to system bus 802 via an intermediate bus bridge (not shown). Display 819 (e.g., screen or display monitor) is connected to system bus 802 via display adapter 815, which may include a graphics controller to improve the performance of graphics-intensive applications and a video controller. Keyboard 821, mouse 822, speakers 823, etc., may be interconnected to system bus 802 via interface adapter 816, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Therefore, as Figure 6 The computer system 800 configured therein includes processing capabilities in the form of a processor 801, storage capabilities including system memory 803 and mass storage device 810, input devices such as keyboard 821 and mouse 822, and output capabilities including speaker 823 and display 819.

[0057] In some embodiments, the communication adapter 807 may use any suitable interface or protocol (such as an Internet Small Computer System Interface) to transmit data. The network 812 may be a cellular network, radio network, wide area network (WAN), local area network (LAN), or the Internet. External computing devices may connect to the computer system 800 via the network 812. In some examples, the external computing device may be an external web server or a cloud computing node.

[0058] It should be understood that Figure 6 The block diagram is not intended to indicate that the computer system 800 will include Figure 6 All components shown. Conversely, computer system 800 may include... Figure 6 Any suitable fewer or additional components not shown herein (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, the embodiments described herein with respect to computer system 800 can be implemented with any suitable logic, wherein the logic mentioned herein may include any suitable hardware (e.g., processor, embedded controller, or application-specific integrated circuit, etc.), software (e.g., applications, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0059] In one or more examples, embodiments of the invention may be implemented using cloud computing technology. It should be understood that although this disclosure includes a detailed description of cloud computing, implementations of the teachings cited herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0060] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0061] The features are as follows:

[0062] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0063] Extensive network access: Capabilities are available through the network and accessible via standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0064] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0065] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up quickly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0066] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0067] The service model is as follows:

[0068] Software as a Service (SaaS): Provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices through a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities; a possible exception is limited user-specific application configuration settings. Platform as a Service (PaaS): Provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage, but have control over the deployed applications and, possibly, the configuration of the application hosting environment.

[0069] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0070] The deployment model is as follows:

[0071] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0072] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0073] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0074] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0075] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0076] Now for reference Figure 7 This describes an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. Nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as the aforementioned private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 7 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0077] See now Figure 8 This demonstrates a cloud computing environment of 50 ( Figure 7 This provides a set of functional abstractions. It should be understood beforehand. Figure 8 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0078] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0079] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0080] In one example, management layer 80 can provide the functions described below. Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure the required service level is met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources based on anticipated future needs according to the SLA.

[0081] Workload tier 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this tier include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and content recognition 96.

[0082] Therefore, one or more embodiments of the present invention facilitate machine autonomous understanding of unstructured tables from various documents available in electronic formats. Embodiments of the present invention provide technical solutions to technical challenges in the field of computing technology, improving computing technology by enabling machines to autonomously perform such functions.

[0083] Furthermore, embodiments of the present invention provide practical applications by facilitating the automation of systems such as corpus conversion, comparison, and conformance. For example, a corpus conversion service could be a system supported by machine learning trained to parse tabular data. This system uses Optical Character Recognition (OCR) to locate text in a document image and uses handcrafted features to parse the table. To train the system, the user needs to manually annotate the positions of columns, rows, and cells. As described herein, embodiments of the present invention operate without requiring explicit annotation of table elements. Instead, embodiments of the present invention use structured representations as supervision. Therefore, embodiments of the present invention provide improved training and improved systems suitable for end-to-end training and multi-task training. Moreover, existing comparison and conformance systems use a set of manually defined rules to define the table layout and use OCR technology to extract content. Embodiments of the present invention facilitate such extraction by using end-to-end processing that avoids some errors accumulated from different processing steps in comparison and conformance systems.

[0084] Furthermore, embodiments of the present invention improve the content recognition system by including tabular data processing and, in some cases, improving the performance of tabular data processing. As previously mentioned, a large amount of information can be obtained in unstructured tabular formats within multiple electronic documents. Such tabular data includes information such as health insurance coverage information, information published in scientific literature, etc.

[0085] Embodiments of the present invention can feed in an image representation of a table that is user-friendly. Embodiments of the present invention automatically learn how to identify the table layout (structure and content) in an end-to-end manner and determine when to extract text. Furthermore, embodiments of the present invention provide a comprehensive parsing and depiction of the table structure, rather than simply extracting the content of each cell from the table as a single unit. Additionally, embodiments of the present invention can parse and depict multi-column / multi-row tables. Even further, embodiments of the present invention parse table data without explicitly annotating one or more elements in the table's structure (e.g., the positions of columns, rows, and cells).

[0086] Therefore, embodiments of the present invention only require the extracted image of the table to intelligently recognize its structure and content as reusable markup (HTML / XML) format. Thus, embodiments of the present invention offer advantages over existing OCR-based systems because they facilitate the recognition of the structure of the table image and the generation of a programmed table of elements, which is not achieved by OCR-based systems.

[0087] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of this invention.

[0088] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or protrusions in slots with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0089] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0090] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit system including, for example, a programmable logic circuit system, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by personalizing the electronic circuit system using state information from the computer-readable program instructions in order to perform aspects of this invention.

[0091] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0092] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0093] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0095] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements over those found in the market, or to enable those skilled in the art to understand the embodiments described herein.

[0096] Various embodiments of the invention are described herein with reference to the accompanying drawings. Alternative embodiments of the invention may be designed without departing from the scope of the invention. In the following description and drawings, various connections and positional relationships (e.g., above, below, adjacent, etc.) are illustrated between elements. Unless otherwise specified, these connections and / or positional relationships may be direct or indirect, and the invention is not intended to be limiting in this respect. Therefore, the connection of entities may refer to direct or indirect connections, and the positional relationship between entities may be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process with additional steps or functions not described in detail herein.

[0097] The following definitions and abbreviations will be used to interpret the claims and description. As used herein, the terms “comprising,” “including,” “containing,” “comprising,” “having,” “covering,” or “covering,” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0098] Furthermore, the term "exemplary" is used herein to mean "used as an example, illustration, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments or designs. The terms "at least one" and "one or more" can be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" can be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0099] The terms “about,” “substantially,” “roughly,” and their variations are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of application submission. For example, “about” could include a range of ±8%, 5%, or 2% of a given value.

[0100] For the sake of brevity, conventional techniques relating to the manufacture and use of aspects of the present invention may or may not be described in detail herein. Specifically, various aspects of the computing systems and specific computer programs used to implement the different technical features described herein are well known. Consequently, for the sake of brevity, many conventional implementation details are only briefly mentioned or omitted entirely herein, without providing well-known system and / or process details.

Claims

1. A computer-implemented method, comprising: The processor's machine learning algorithms are used to perform training operations for training a machine learning model, the training operations including: A set of images receiving tabular data and a set of labeled data corresponding to the images of the tabular data; The labeled data is used to train a first neural network to depict the tabular data from the set of images as units, wherein training the first neural network includes: parsing a first image from the set of images to match portions of the tabular data from the first image as specified by corresponding first labeled data from the set of labeled data; and updating one or more weights from the first neural network based on structural accuracy calculated by comparing the portions with the corresponding first labeled data; and The labeled data is used to train a second neural network to determine the contents of the cells in the tabular data from the set of images; Receive an input image containing labeled data of the first table without indicating the structure of the first table data; and By using the first neural network to determine the structure of the first table data and using the second neural network to extract the content of the first table data, an electronic output corresponding to the first table data is generated.

2. The computer-implemented method according to claim 1, wherein training the first neural network further includes: The one or more weights from the first neural network are updated based on the fact that the accuracy of the structure is lower than a predetermined threshold. as well as If the accuracy of the structure exceeds the predetermined threshold, the first neural network is marked as trained.

3. The computer-implemented method according to claim 1 or 2, wherein training the second neural network comprises: Content is extracted from a second image from the set of images to match the content of the table data from the second image as specified by the corresponding second tag data from the set of tag data.

4. The computer-implemented method according to claim 3, wherein training the second neural network further comprises: The content accuracy of the second neural network is calculated by comparing the content extracted from the second image with the content from the corresponding second labeled data; One or more weights from the second neural network are updated based on the content accuracy being lower than a predetermined threshold; as well as If the accuracy of the content exceeds the predetermined threshold, the second neural network is marked as trained.

5. The computer-implemented method according to any one of claims 1 or 2, wherein generating the electronic output further comprises: The structure to be depicted is merged with the content extracted from the input image.

6. The computer-implemented method of claim 5, wherein the electronic output comprises the content extracted from the input image separated according to the structure using a predetermined markup language.

7. A computer system, comprising: Memory; as well as A processor coupled to the memory, the processor using machine... The learning algorithm is programmed and configured to execute a method, the method comprising: Training a machine learning model, wherein the training includes: A set of images receiving tabular data and a set of labeled data corresponding to the images of the tabular data; The labeled data is used to train a first neural network to depict the tabular data from the set of images as units, wherein training the first neural network includes: parsing a first image from the set of images to match portions of the tabular data from the first image as specified by corresponding first labeled data from the set of labeled data; and updating one or more weights from the first neural network based on structural accuracy calculated by comparing the portions with the corresponding first labeled data; and The labeled data is used to train a second neural network to determine the contents of the cells in the tabular data from the set of images; Receive an input image containing labeled data of the first table without indicating the structure of the first table data; and By using the first neural network to determine the structure of the first table data and using the second neural network to extract the content of the first table data, an electronic output corresponding to the first table data is generated.

8. The computer system of claim 7, wherein training the first neural network further comprises: The one or more weights from the first neural network are updated based on the fact that the accuracy of the structure is lower than a predetermined threshold. as well as If the accuracy of the structure exceeds the predetermined threshold, the first neural network is marked as trained.

9. The computer system of claim 7 or 8, wherein training the second neural network comprises: Content is extracted from a second image from the set of images to match the content of the table data from the second image as specified by the corresponding second tag data from the set of tag data.

10. The computer system of claim 9, wherein training the second neural network further comprises: The content accuracy of the second neural network is calculated by comparing the content extracted from the second image with the content from the corresponding second labeled data; One or more weights from the second neural network are updated based on the content accuracy being lower than a predetermined threshold; as well as If the accuracy of the content exceeds the predetermined threshold, the second neural network is marked as trained.

11. The computer system of claim 7 or 8, wherein generating the electronic output further comprises: The structure to be depicted is merged with the content extracted from the input image.

12. The computer system of claim 11, wherein the electronic output comprises the content extracted from the input image separated according to the structure using a predetermined markup language.

13. A computer program product for determining the contents of tabular data, the computer program product comprising: A computer-readable storage medium, which is readable by processing circuitry and stores instructions for execution by the processing circuitry to perform the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being loadable into the internal memory of a digital computer, the computer program including a software code portion that, when the program is run on a computer, is configured to perform the method according to any one of claims 1 to 6.

Images