A multimodal network designed with layout in mind for document comprehension.

Abstract

A technique is provided for a layout-aware multimodal network for document understanding. In one technique, word data representations are identified based on words extracted from an image of a document. Based on that image, table features of one or more tables in the document are identified. Based on those table features, one or more table data representations are identified. These word data representations and one or more table data representations are input into a machine learning model to generate a document data representation for that document. The task is performed based on that document data representation. In a related technique, instead of the one or more table data representations, one or more layout data representations are identified based on a set of layout features of the document identified based on the image, and these are input into the machine learning model.

Description

Technical Field

[0001] This disclosure generally relates to machine learning, and more particularly, to multimodal networks for better understanding of documents.

Background Art

[0002] Background Automated document processing is a common endeavor and poses numerous challenges. Some of these challenges arise from the existence of many different types of documents such as tax returns, job applications, resumes, invoices, passports, driver's licenses, technical papers, and essays. Each type of document has a different structure and layout, which makes it difficult for automated systems to properly extract the content and metadata of the document, identify key-value pairs, and classify the extracted content. Furthermore, there can be significant variations within a single document type, such as the many different ways a resume can be formatted.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Automation technologies have been developed because manual processing, extraction, and classification are infeasible for the large volume of documents typical enterprises intake and process. One of those technologies includes a machine-learned document understanding model that generates a data representation (or embedding) for a document based on the content of the document such as the phrases detected within the document. However, such document understanding models have relatively low performance in document classification and other end tasks (e.g., key-value pair extraction). One possible reason for that low performance is the complex layout that documents typically follow.

[0004] Another issue associated with current automated document processing (e.g., in the field of key-value extraction) is model regression. For example, it is difficult for current document understanding models to improve a single key without regressing on other keys.

[0005] The methods described in this section are pursued methods, but not necessarily methods that have been conceived or pursued previously. Therefore, unless otherwise stated, no method described in this section should be presumed to qualify as prior art simply because it is included in this section. [Brief explanation of the drawing]

[0006] [Figure 1] This block diagram shows an exemplary model architecture for improving document comprehension in one embodiment. [Figure 2] This block diagram shows an exemplary portion of an exemplary document and an exemplary pair of input texts from different blocks or sections of the exemplary document in one embodiment. [Figure 3] This block diagram shows an exemplary pre-training path for matching overall layout consistency in one embodiment. [Figure 4] This block diagram shows an exemplary pre-training path for table content detection in one embodiment. [Figure 5] This is a block diagram showing an exemplary fine-tuning path in one embodiment. [Figure 6] In one embodiment, this is a flowchart illustrating an exemplary process for leveraging a document understanding model. [Figure 7] This is a block diagram showing a computer system in which embodiments of the present invention may be implemented. [Figure 8] This is a block diagram of a basic software system that can be used to control the operation of a computer system. [Modes for carrying out the invention]

[0007] Detailed explanation In the following description, many specific details are provided for illustrative purposes and to provide a comprehensive understanding of the invention. However, it will be apparent that the invention can be implemented without these specific details. In other instances, well-known structures and apparatus are shown in block diagrams to avoid unnecessarily obscuring the invention.

[0008] General Overview Systems and methods are provided for generating and leveraging more accurate document understanding models. In one technique, the document understanding model includes new input formats to improve paragraph / block-level layout understanding. In particular, embeddings of outputs from table detection models and / or document layout detection models carry rich, holistic-level layout information and are input into the document understanding model. In related techniques, quasi-supervised learning tasks for paragraph / block-level layout consistency are added to improve model performance for documents with complex layouts.

[0009] The embodiments improve computer-related technologies. In particular, the embodiments improve document understanding by considering the characteristics of tables and / or the overall layout of the document. As a result, machine learning tasks such as document classification and keyword extraction are improved with increased accuracy and fewer errors, especially with respect to documents with relatively complex layouts.

[0010] Model Architecture Figure 1 is a block diagram showing an exemplary model architecture 100 for improving document comprehension in one embodiment. The model architecture 100 includes four extractors 112, 114, 116, and 118, each generating its own data representation (122-128) of the data extracted by the corresponding extractor. While four extractors are shown, the model architecture may include fewer or more extractors. For example, the model architecture 100 may include only one of the extractors 112 or 114. As another example, the model architecture 100 may include only one of the extractors 116 or 118.

[0011] The model architecture 100 also includes a document understanding model 130 that generates document data representations (or embeddings) 132, assuming data representations 122-128 are inputs. Finally, the model architecture 100 includes an end task 140 that takes the document embeddings 132 as input and generates outputs that vary depending on the end task 140. An example of an end task 140 is the classification of documents input to the extraction units 112-118, which recognize named entities, key-value pair extractions, and document-based questions and answers. These embeddings can also be used in document matching and retrieval.

[0012] The digital image 110 of the document is input to the extraction units 112-118. A document may have a single page or multiple pages. If a document has multiple pages, then a different digital image may exist for each page. The digital image 110 may be one of several digital image formats, such as JPEG or PNG. Each of the extraction units 112-118 includes an analysis unit and a data representation generation unit. Each analysis unit analyzes different aspects of the digital image 110.

[0013] Image Feature Extraction Unit The analysis unit of the image feature extraction unit 112 analyzes the pixel-level information of the digital image 110. The analysis unit detects layout components (e.g., paragraphs, figures) as objects from the digital image 110. The analyzer also pays attention to other features in different layers, such as edges and lines, and words and paragraphs.

[0014] The image features extracted by the analysis unit of the image feature extraction unit 112 are input to the data representation generation unit of the image feature extraction unit 112. Each data representation generation unit generates a data representation or "embedding". The data representation (or embedding) may be a one-dimensional vector containing multiple values ​​of size N, or a two-dimensional matrix of size N × M. The data representation generation unit may include a model such as a neural network trained using one or more machine learning techniques.

[0015] Text extraction unit The text extraction unit 114 includes an optical character recognition (OCR) component (a type of analysis unit) that extracts individual text items, such as words or phrases. Each text item is associated with an embedding in a mapping from text items to embeddings. This mapping may be limited to certain words or phrases, or may intentionally exclude common words or phrases, such as stop words. The text items may be adapted (or limited) to a specific subject area, such as a tax return, passport, or invoice. If the text extraction unit 114 can access multiple mappings, then there may be overlaps in the text items included in those multiple mappings.

[0016] In the mapping from text items to embeddings, the text embeddings may be pre-trained text embeddings, such as embeddings already generated based on a corpus of text. These text embeddings can be updated when training the document understanding model 130. In this way, the text embeddings do not start as randomly generated embeddings and can therefore converge more quickly.

[0017] Table Feature Extraction Unit The table feature extraction unit 116 includes an analyzer that detects any table included in the digital image 110 and determines one or more features of the detected table for each detected table. Exemplary features of a table include the coordinates of the table, the width of the table, the height of the table, the number of columns of the table, and the number of rows of the table.

[0018] The coordinates of the table may include vertical and horizontal offsets to the digital image 110 in which the table is placed. The offset may be the number of pixels from the origin (e.g., the upper left corner of the digital image 110), or may be the ratio of a set A of table features included in a table that may include a 4 - set coordinate system with one for each corner of the table. Alternatively, a pair of table features included in a table may include only a 2 - set coordinate system where one is for one corner of the table and the other corresponds to the opposite corner of the table. The other corners may be derived based on these 2 - set coordinate systems.

[0019] The width and height of the table may be evaluated in one or more units such as the number of pixels or the ratio in the overall width or height of the digital image 110. For example, the width of the table may be 5% of the width of the digital image 110, indicating that the table is relatively narrow. As another example, the height of the table may be 50% of the height of the digital image 110, indicating that the table is relatively tall or prominent.

[0020] The table feature extraction unit 116 includes a data representation generation unit that generates a table data representation (or embedding) 126 based on the identified table features. For example, the data representation generation unit may receive the identified table features as input and include a neural network that generates an embedding. The neural network may be trained by a number of training examples combined with training by other neural networks included in other data representation generation units including the document understanding model 130.

[0021] Overall Layout Feature Extraction Unit The overall layout feature extraction unit 118 includes an analysis unit that analyzes the digital image 110 from an overall perspective in order to identify the layout features of one or more components of the digital image 110. Examples of layout components include figures, paragraphs, and tables. Examples of features of layout components include the coordinates of the layout components, the size of the layout components, and the trained embeddings of the layout components. Layout features and table features focus on different levels of detail. For example, layout features focus on the entire table, while the related table features focus on the rows / cells within the table.

[0022] The overall layout feature extraction unit 118 also includes a data representation generation unit that generates a data representation for the digital image 118 based on the detected layout features. The data representation generation unit receives the layout features as input and outputs a data representation.

[0023] Document Understanding Model In certain embodiments, the document understanding model 130 is limited to receiving a number of data representations / embeddings from each extraction unit. For example, the document understanding model 130 receives a single embedding from the image feature extraction unit 112, up to 500 embeddings from the text extraction unit 114, up to 5 embeddings from the table feature extraction unit 116, and a single embedding from the overall layout feature extraction unit 118. If more phrases are detected in the digital image 110 than the maximum number of phrases, then one or more phrase embeddings (more than the maximum number) are not generated or sent to the document understanding model 130. For example, if 600 phrases are detected, then 100 of the most common phrases (such as determined in a corpus of text) are ignored and the phrase embeddings for those phrases are not sent to the document understanding model 130.

[0024] In some cases, the data representation from the extraction unit (for a digital image) may be less than the maximum number of embeddings from the extraction unit (for that digital image). For example, two tables may be detected in digital image 110, but the document understanding model 130 may be configured to receive up to five embeddings from the table feature extraction unit 116. The three possible embeddings corresponding to the three tables that are not detected in digital image 110 (because they do not exist) may be empty or filled with zeros.

[0025] The document understanding model 130 includes a neural network that takes data representations (from extraction units 112-118) as input and outputs a document data representation or embedding 132 representing the digital image 110. The document embedding 132 is input to an end task 140, which outputs a value corresponding to a task of a type such as a binary classifier (e.g., whether the digital image 110 contains a certain attribute or feature) or a document classifier, and the document (represented by the digital image 110) may belong to one of several classes. The end task 140 is not limited to any particular configuration and may include multiple inputs in addition to the document embedding 132. For example, the end task 140 may include a logistic regression model that accepts another neural network and / or other inputs.

[0026] Pre-training In the context of document comprehension, “pre-training” refers to training a model to acquire a general semantic understanding of the content from documents. In this phase, data (e.g., tens of millions of documents) is fed to a document comprehension model 130, whose weighting is automatically adjusted so that it correctly predicts the content from the documents. However, pre-training is performed before the main training phase, in which the output of the document comprehension model 130 is input to the end task 140.

[0027] In the pre-training phase, one or more self-managed tasks are performed on input documents. Each self-managed task consists of two parts: data entry and labeling. In self-managed tasks, labels are generated automatically without human intervention. An example of a self-managed task is a masked language model (MLM) where words are randomly masked from a sentence. The masked words become the labels for this task, and the model is tuned to predict those masked words.

[0028] Matching overall layout consistency In one embodiment, a self-managed task is performed to enable the document understanding model 130 to acquire overall layout consistency. This self-managed task is referred to as “overall layout consistency matching.” In this task, the whole model is required to predict, for each pair of input texts (e.g., two different words), whether the input texts in that pair come from the same layout block. Each input text in the pair may be distinguished from other possible input texts by a space or punctuation such as a period, comma, colon, or semicolon. This task helps the whole model capture the layout information embedded in the document.

[0029] In one embodiment, to perform the task of matching overall layout consistency, the training data generation unit automatically generates training examples based on the analysis of segmented documents. The training data generation unit may be implemented in software, hardware, or any combination of software and hardware. A segmented document is one that has already been segmented or divided into separate sections or blocks. An example of a segmented document is an HTML document with HTML tags that define different sections, such as paragraph tags, table tags, and section tags. Another example of a segmented document is a PDF document with defined sections (excluding PDFs containing only images). Other examples include Word documents and documents in any markup language such as XML, TeX, and LaTeX.

[0030] The training data generation unit selects two pairs of input text (such as words or phrases) and generates labels indicating whether the two input texts come from different sections. If the two input texts come from the same section of a segmented document (for example, identified based on the same set of tags or pathnames), the training data generation unit generates a label indicating that the two input texts come from the same section. An example of such a label is "1". Conversely, if the two input texts do not come from the same section of a segmented document, the training data generation unit generates a label indicating that the two input texts do not come from the same section. An example of such a label is "0". For each pair of input texts and their corresponding labels, the training data generation unit generates a single training instance that is added to a set of training data for the model.

[0031] The training data generation unit can generate many training examples. For example, after selecting a single word or phrase contained within a section of a segmented document, the training data generation unit can generate training examples for each other word or phrase contained within that section, which could amount to over a hundred training examples (all with the same positive label). The training data generation unit can also generate another training example for each word or phrase found outside the section (for the selected word or phrase), which could be a much larger number of training examples, each with a negative label.

[0032] Figure 2 is a block diagram showing a part 200 of an exemplary document and a pair of exemplary input texts from different blocks or sections of the exemplary document in one embodiment. Part 200 includes Figure 210, Figure Section 220, and Non-Figure Section 230. Figure Section 220 and Non-Figure Section 230 are different sections in the same document. In this example, the training data generation unit selects two words, namely "plasma" from Figure Section 220 and "dog" from Non-Figure Section 230. The training data generation unit identifies that "plasma" and "dog" are from different sections. This identification may be based on section tags (or path names) associated with the two sections from which the selected words originate.

[0033] Figure 3 is a block diagram showing an exemplary pre-training path 300 for matching overall layout consistency in one embodiment. The pre-training path 300 includes similar elements such as the model architecture 100. The pre-training path 300 includes a set of extractors 310, such as a visual extractor, an OCR extractor, a layout extractor, and / or a table extractor. Thus, the set of extractors 310 may include any one of these extractors or any combination thereof. The pre-training path 300 may also include a document understanding model 320, a word-level layout prediction unit 330, ground truth 340, and a layout evaluation unit 350. Thus, while the final use case including the document understanding model 320 could be document classification or recognition of named entities, this pre-training phase includes predicting whether two words come from the same section or block.

[0034] All tokens / phrases extracted from the document are input into the model simultaneously, and the document comprehension model 320 outputs the embeddings for each token (and other additional embeddings about the entire document as an overall representation). For pre-training, the trainer samples pairs of embeddings (displaying tokens) according to layout labels and sends the concatenated embeddings to the classifier (in this example, the phrase-level layout prediction unit 330). The classifier's prediction (output) may be within the range of 0 to 1, indicating whether the tokens in the pair come from the same block / section.

[0035] The layout evaluation unit 350 compares the prediction with the corresponding case labels from the ground truth 340. Based on the output 352 of this comparison, the weights in the document understanding model 320 (and optionally in one or more models in a set of extraction units 310) are updated, for example, using backpropagation.

[0036] Table content detection In one embodiment, a self-managed task is performed to assist the document understanding model 130 when capturing table contents from a document. This self-managed task is referred to as “table contents detection.” Table contents detection leverages the overall model to predict at the word level whether a particular phrase is from a table. Furthermore, the overall model may be used to predict whether the particular phrase is from the table header or from the table contents. The table header may include column names and the table name, while the table contents include any content within the range of a row or column in the table, but nothing outside of that range.

[0037] The training data generation unit may use documents in a well-defined table to generate training examples for training the document comprehension model 130. Again, examples of such documents include HTML and PDF documents having tags or table information that allow the training data generation unit to identify whether a text item in the document is part of a table or not, or, in a relevant embodiment, whether a text item in the document is part of a table header, part of the content of the table, or not part of the table.

[0038] Figure 4 is a block diagram showing an exemplary pre-training route 400 for table content detection in one embodiment. The pre-training route 400 includes similar elements to the pre-training route 300. The pre-training route 400 includes a set of extraction units 410, such as a visual extraction unit, an OCR extraction unit, a layout extraction unit, and a table extraction unit. The pre-training route 400 also includes a document understanding model 420, a word-level table label prediction unit 430, ground truth 440, and a table content evaluation unit 450. Thus, while the final use case involving the document understanding model 420 could be document classification or recognition of named entities, this pre-training phase includes predicting whether a text item is from a table, and whether it is from a table header, from table content, or from no part of the table.

[0039] Assuming training cases include token embeddings and labels, the output of the document comprehension model 420 is input to the word-level table-level prediction unit 430, which predicts whether the token (corresponding to the token embedding) came from a table (or whether the token came from a table header, a table content, or no table). The table content evaluation unit 450 compares this prediction with the labels of the training cases coming from ground truth 440. Based on the output 452 of this comparison, the weighting of one or more models in one set of extraction units 410 within the document comprehension model 420 (or optionally) is updated, for example, using backpropagation.

[0040] In the relevant example, the training example includes two token embeddings, such as whether the corresponding tokens belong to the same row, column, and cell in the same table, and multiple labels, each corresponding to a different task. The evaluation of each row, column, and cell is a separate task (head). Therefore, the output of the document understanding model 420 may include multiple predictions, such as multiple values ​​between 0 and 1, each corresponding to a different task.

[0041] Fine adjustment A pre-trained document understanding model acts as an engine that provides the ability to understand document content. Task-specific heads are designed and added "on top of" (or immediately after) the document understanding model to predict according to the task. Examples of such tasks include document classification, recognition of named entities, and key-value extraction.

[0042] Figure 5 is a block diagram showing an exemplary fine-tuning path 500 in one embodiment. The fine-tuning path 500 includes similar elements, such as the pre-training path 400. The fine-tuning path 500 includes an input document 502 (e.g., an image) and a set of extraction units 510, such as a visual extraction unit, an OCR extraction unit, a layout extraction unit, and / or a table extraction unit. The fine-tuning path 500 also includes a document understanding model 520, a task head 530, and a controlled training component 540. The controlled training component 540 can utilize training data, including label data or ground truth. As shown in Figure 5, the weight of the task head 530 is updated in 542 based on the evaluation of each training instance to align with the objectives of controlled training. The weight of the (pre-trained) document understanding model 520 is also assigned before the fine-tuning process. In the relevant embodiment, the weight of the document understanding model 520 is also adjusted to align with the objectives of the task.

[0043] Thus, the document understanding model 520 (including the weights generated for the document understanding model 520 during the pre-training process) can be used when training different task heads. The weights of the document understanding model 520 can be further modified during the fine-tuning process for each of these different task heads. Therefore, if the weights of the document understanding model 520 are not modified during the fine-tuning corresponding to different tasks, then the same document understanding model 520 is used for those different tasks. Conversely, if the weights of the document understanding model 520 are modified during the fine-tuning corresponding to different tasks, then different versions of the document understanding model 520 (where "different versions" means different weights) are used for the different tasks.

[0044] Exemplary process Figure 6 is a flowchart illustrating an exemplary process 600 for leveraging a document understanding model in one embodiment. Process 600 may be implemented in software, hardware, or any combination of software and hardware. For example, process 600 may be implemented by one or more computer processes executing software code written in a sophisticated programming language such as Java® or C++. Embodiments are not limited to any programming language or whether specific hardware is used.

[0045] In block 610, an image of the document is identified. The image may originate from one of several data sources, such as an image database, or from a remote computing device such as a scanning device, a digital camera, or a lap computer.

[0046] In block 620, a set of text items to be extracted from the image is identified based on the image. Thus, block 620 may precede an OCR step that automatically recognizes the text items in the image. The text items may be words or phrases containing multiple words.

[0047] In block 630, text item embedding is generated based on the set of text items. Block 630 may be executed by an OCR extraction unit.

[0048] Although not shown in Figure 6, process 600 may also include identifying a set of image features based on an image whose features can be generated by the image feature extraction unit. Subsequently, process 600 may also include generating an image embedding based on the set of image features.

[0049] In block 640, based on the image, a set of table features of one or more tables in the document is identified. One or more tables may have already been recognized using an OCR process or another process for automatically recognizing tables in a document. Such recognition may include recognizing intersecting parallel horizontal and parallel vertical lines and identifying that characters are detected within those lines.

[0050] In block 650, one or more table embeddings are generated based on a set of table features. Block 650 may be executed by a table feature extractor and may include one or more neural networks.

[0051] In block 660, text item embeddings and one or more table embeddings are input into a machine learning model to generate document embeddings for the document. (If there are other embeddings, such as image embeddings, then those embeddings are also input into the machine learning model to generate those image embeddings.) Thus, the machine learning model may have already been trained before block 610. Such training may include training on one or more self-managed tasks and different specific end tasks.

[0052] In block 670, a task is performed based on document embedding. This task may be a classification task, a key-value extraction task, or a named entity recognition task.

[0053] Process 600 includes generating and leveraging one or more table embeddings, while embodiments may further, or alternatively, include generating and leveraging overall layout embeddings based on the features of components detected in the image, such as paragraphs and figures.

[0054] Hardware Overview According to one embodiment, the techniques described herein are implemented by one or more application-specific computing devices. The application-specific computing device may include one or more digital electronic devices such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are hardwired or permanently programmed to perform the techniques, or may include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such application-specific computing devices may also be combined with custom hardwired logic, ASICs, or FPGAs having custom programming to achieve the techniques. The application-specific computing device may be a desktop computer system, a portable computer system, a portable device, a network device, or any other device that incorporates hardwired or programmable logic to perform the techniques.

[0055] For example, Figure 7 is a block diagram showing a computer system 700 in which an embodiment of the present invention may be implemented. The computer system 700 includes a bus 702 or other transmission mechanism for transmitting information, and a hardware processor 704 coupled to the bus 702 for processing information. The hardware processor 704 may be, for example, a general-purpose microprocessor.

[0056] The computer system 700 may also include main memory 706, which is coupled to bus 702 for storing information or instructions executed by processor 704, such as random access memory (RAM) or other dynamic storage devices. Main memory 706 may be used to store temporary variables or other intermediate information during the execution of instructions by processor 704. Once such instructions are stored in a non-temporary storage medium available to processor 704, the computer system 700 becomes a purpose-specific machine customized to perform the actions specified in those instructions.

[0057] The computer system 700 further includes read-only memory (ROM) 708 or other static storage devices coupled to bus 702 for storing static information and instructions for processor 704. A storage device 710, such as a magnetic disk, optical disk, or solid-state drive, is supplied to and coupled to bus 702 for storing information and instructions.

[0058] The computer system 700 may be connected via bus 702 to a display 712, such as a cathode ray tube (CRT), to display information to the computer user. An input device 714, including alphanumeric and other keys, is connected to bus 702 to transmit information and command selections to the processor 704. Another type of user input device is a cursor control 716, such as a mouse, trackball, or cursor directional keys, for transmitting directional information and command selections to the processor 704 and for controlling the movement of a cursor on the display 712. This input device typically has two degrees of freedom on two axes, namely a first axis (e.g., x) and a second axis (e.g., y), which allow the device to determine its position on a plane.

[0059] Computer system 700 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic in combination with the computer system to make or program the computer system 700 into a machine for a specific purpose. According to one embodiment, the techniques described herein are executed by computer system 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium, such as storage device 710. Execution of the sequence of instructions contained in main memory 706 causes processor 704 to execute the process steps described herein. In alternative embodiments, hardwired electronic circuit configurations may be used instead of, or in combination with, software instructions.

[0060] As used herein, the term “storage medium” refers to any non-temporary medium that stores data and / or instructions that cause a machine to operate in a particular way. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, or solid-state drives, such as storage device 710. Volatile media include dynamic memory, such as main memory 706. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or cartridges.

[0061] A storage medium is distinct from a transmission medium, but can be used in conjunction with it. A transmission medium is involved in the transfer of information between storage mediums. For example, a transmission medium includes coaxial cables, copper wires, and optical fibers, including wires containing bus 702. A transmission medium can also take the form of sound waves or light waves, such as those generated during radio and infrared data communications.

[0062] Various forms of media may be involved in transmitting one or more sequences of one or more instructions to the processor 704 for execution. For example, the instructions may initially be held on a magnetic disk or solid-state drive of a remote computer. The remote computer may load the instructions into dynamic memory and transmit them over a telephone line using a modem. A modem local to the computer system 700 may receive data over the telephone line and may use an infrared transmitter to convert the data into an infrared signal. An infrared detector receives the data transmitted in infrared signal format, and appropriate electronic circuitry may place the data on bus 702. Bus 702 transmits the data to main memory 706, and the processor 704 retrieves and executes the instructions from main memory 706. The instructions received by main memory 706 may be selectively stored on storage device 710 before or after execution by processor 704.

[0063] The computer system 700 also includes a communication interface 718 connected to bus 702. The communication interface 718 provides bidirectional data communication connected to a network link 720 connected to a local network 722. For example, the communication interface 718 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connectivity to a corresponding type of telephone line. As another example, the communication interface 718 may be a local area network (LAN) card that provides data communication connectivity to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface 718 transmits and receives electrical, electromagnetic, or optical signals that carry digital data streams displaying various types of information.

[0064] Network link 720 typically provides data communication to other data devices through one or more networks. For example, network link 720 may provide connection to a host computer 724 or to a data device operated by an Internet service provider (ISP) 726 through a local network 722. ISP 726 then provides data communication services through a worldwide packet data communication network, now commonly referred to as the “Internet” 728. Both the local network 722 and the Internet 728 use electrical, electromagnetic, or optical signals to transmit digital data streams. Signals through these various networks, and signals on network link 720 through the communication interface 718, transmit digital data bidirectionally to the computer system 700, and these signals are exemplary forms of transmission media.

[0065] The computer system 700 can send messages and receive data, including programming code, through the network, network link 720, and communication interface 718. In the case of the internet, server 730 can send request code for an application program through the internet 728, ISP 726, local network 722, and communication interface 718.

[0066] The requested code may be executed by the processor 704 upon receipt and / or stored in the memory device 710 or other non-volatile storage for later execution.

[0067] According to some embodiments, a system including one or more computing devices has means for performing operations including operations enumerated in any method claim. According to a further embodiment, a computer program product, when executed by one or more computing devices, includes instructions causing one or more computing devices to perform operations including operations enumerated in any method claim. According to a further embodiment, a computer-readable medium, when executed by one or more computing devices, includes instructions causing one or more computing devices to perform operations including operations enumerated in any method claim.

[0068] Software Overview Figure 8 is a block diagram of a basic software system 800 that may be used to control the operation of a computer system 700. The software system 800 and its components, including their connections, relationships, and functions, are intended to be typical only and not to limit the implementation of this exemplary embodiment. Other software systems suitable for implementing this exemplary embodiment may have different components, including components with different connections, relationships, and functions.

[0069] A software system 800 is provided to instruct the operation of the computer system 700. The software system 800, which may be stored in system memory (RAM) 706 and fixed storage (e.g., hard disk or flash memory) 710, includes a kernel or operating system (OS) 810.

[0070] OS810 manages the low-level aspects of computer operation, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more application programs, represented as 802A, 802B, 802C...802N, can be loaded (e.g., transferred from fixed storage 710 to memory 706) for execution by system 800. Such applications, or other software intended for use in computer system 700, can also be stored as a pair of downloadable computer executable instructions for download and installation from, for example, an internet location (e.g., a web server, application store, or other online service).

[0071] The software system 800 includes a graphical user interface (GUI) 815 for receiving user commands and data in a graphical manner (e.g., "point-and-click" or "touch gestures"). These inputs can then be processed by the system 800 in response to instructions from the operating system 810 and / or application 802. The GUI 815 also operates to display the results of actions from the OS 815 and application 802, where the user can provide additional input or terminate the session (e.g., log off).

[0072] OS810 runs directly on the bare hardware 820 (e.g., processor 704) of the computer system 700. Alternatively, a hypervisor or virtual machine monitor (VMM) 830 may be inserted between the bare hardware 820 and OS810. In this configuration, VMM830 acts as a software "cushion" or virtualization layer between OS810 and the bare hardware 820 in the computer system 700.

[0073] VMM830 creates and runs one or more virtual machine instances ("guest machines"). Each guest machine contains a "guest" operating system, such as OS810, and one or more applications, such as application802, designed to run on the guest operating system. VMM830 displays the guest operating systems that have virtual operating platforms and manages the execution of those guest operating systems.

[0074] In some cases, VMM830 can enable a guest operating system to function as if it were running directly on the bare hardware 820 of computer system 700. In these cases, the same version of the guest operating system configured to run directly on the bare hardware 820 can also run on VMM830 without modification or reconfiguration. In other words, in some cases, VMM830 can provide the guest operating system with complete hardware and CPU virtualization.

[0075] In other cases, the guest operating system may be specifically designed or configured to run on VMM830 for efficiency. In these cases, the guest operating system is aware that it is running on the virtual machine monitor. In other words, VMM830 can provide paravirtualization to the guest operating system in some cases.

[0076] A computer system process includes the allocation of hardware processor time and memory (physical and / or virtual), the allocation of which is for storing instructions executed by the hardware processor, for storing data generated by the hardware processor executing those instructions, and / or for storing the hardware processor state (e.g., register contents) during hardware processor allocation when the computer system process is not running. A computer system process operates under the control of the operating system and may operate under the control of other programs running on the computer system.

[0077] The basic computer hardware and software described above are for the purpose of illustrating the basic computer components that may be used to perform exemplary embodiments. Such exemplary embodiments are not necessarily limited to any particular computing environment or computing device configuration. Rather, such exemplary embodiments may be implemented in any kind of system architecture or processing environment that a person skilled in the art would understand, based on this disclosure, to be capable of supporting the features and functions of the exemplary embodiments described herein.

[0078] Cloud computing The term “cloud computing” is used herein in general to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid supply and release of resources with minimal administrative effort or service provider interaction.

[0079] Cloud computing environments (sometimes referred to simply as cloud environments or clouds) can be implemented in a variety of different ways to best suit different needs. For example, in a public cloud environment, the underlying computing infrastructure is owned by the organization making the cloud services available to other organizations or the general public. In contrast, private cloud environments are generally intended for use by or within a single organization. Community clouds are intended to be shared by several organizations within a community, while hybrid clouds include two or more clouds (e.g., private, community, or public) linked by data and application portability.

[0080] Generally, the cloud computing model allows some of those responsibilities, which may have previously been provided by an institution's own information technology department, to be instead offered as a service layer within the cloud environment for consumer use (whether internal or external to the institution, depending on the public / private nature of the cloud). Depending on its specific implementation, the exact definition of each cloud service layer, or its components or features, may vary, but common examples include Software as a Service (SaaS), where consumers utilize software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications. In Platform as a Service (PaaS), consumers may utilize software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (e.g., everything under the runtime environment). In Infrastructure as a Service (IaaS), consumers can deploy and run any software application and / or supply processing, storage, network, and other underlying computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (e.g., everything below the operating system layer). In Database as a Service (DBaaS), consumers utilize database servers or database management systems running on cloud infrastructure, while the DBaaS provider manages or controls the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0081] In the features described above, embodiments of the present invention have been described with reference to a number of specific details that may vary depending on the implementation. This specification and the figures should therefore be considered illustrative rather than restrictive. The sole and exclusive scope of the present invention, and the scope intended by the applicant as the scope of the present invention, is the scope of the language and equivalents in the specific form issued by such claims, including any subsequent amendments to the set of claims issued in this application.

Claims

1. It is a method, Identifying multiple generated word data representations based on a set of words extracted from a document image, This includes identifying a set of table features in one or more tables of the document based on the aforementioned image, The set of table features for the table detected in the image of the document is: The coordinates of the aforementioned table, The width of the aforementioned table and, The height of the aforementioned table and, The number of columns in the aforementioned table and, The number of rows in the aforementioned table includes at least two of the following: The aforementioned method, Identifying one or more table data representations generated based on one or more table features in one or more tables detected in the image of the document, To generate a document data representation for the aforementioned document, the plurality of word / phrase data representations and the one or more table data representations are input into a machine learning model, This includes performing a task based on the aforementioned document data representation, The method described above is performed by one or more computing devices.

2. The aforementioned method, Based on the aforementioned image, the layout features of one set of documents are identified, The method further includes generating one or more layout data representations based on the one set of layout features of the document, The aforementioned input includes inputting the one or more layout data representations into the machine learning model. The method according to claim 1, wherein the document data representation is generated based on the one or more layout data representations.

3. The aforementioned machine learning model is the first model, The method further includes automatically generating multiple labels, Each label represents one of several training cases. Each of the aforementioned labels indicates whether the text item in the corresponding training example is from a table in the document. The output of the first model is input to a second model that is different from the first model. The second model predicts whether the input text item is from a table in the document, The method according to claim 1 or 2, further comprising training the first model based on the plurality of training examples.

4. The method according to claim 3, wherein each of the plurality of labels indicates whether the text item is from a table header in the document, from the contents of a table in the document, or not from any table in the document.

5. The method described above, before training the second model, involves, for each of the multiple divided documents, Selecting text items from each of the aforementioned divided documents, The system automatically determines whether the text item is part of a table within each of the partitioned documents. The method according to claim 3, further comprising generating training cases based on the aforementioned identification.

6. The aforementioned training examples are the first set of training examples. The method further includes training a first model based on a first set of training instances, and then training a third model that takes the output of the first model as input based on a second set of training instances different from the first set of training instances. The method according to claim 3, wherein the third model has a first task objective different from the second task objective in the second model.

7. The method according to claim 6, further comprising updating the weighting of the first model based on training of the third model based on the second plurality of training cases.

8. The method further includes identifying one or more image data representations generated based on a set of image features of the images in the document, The aforementioned input includes inputting one or more image data representations into the machine-learned model. The method according to any one of claims 1 to 7, wherein the document data representation is further generated based on the one or more image data representations.

9. The aforementioned method, Based on the aforementioned image, identify a set of words extracted from the aforementioned image, Based on the aforementioned set of words, the plurality of word data representations are generated, Based on the aforementioned image, identify the table features of one or more tables in the document, The method according to any one of claims 1 to 8, further comprising generating one or more table data representations based on the one set of table features in one or more tables detected in the image of the document.

10. The aforementioned multiple word data representations are multiple word embeddings, The aforementioned one or more table data representations are one or more table embeddings, The method according to any one of claims 1 to 9, wherein the document data representation is a document embedding.

11. It is a method, Identifying multiple phrase data representations generated based on a set of phrases extracted from a document image, Identifying one or more layout data representations generated based on a set of layout features of the document identified based on the aforementioned image, In order to generate a document data representation for the aforementioned document, the plurality of word / phrase data representations and the one or more layout data representations are input into a machine learning model, This includes performing a task based on the aforementioned document data representation, The method described above is performed by one or more computing devices.

12. The aforementioned machine learning model is the first model, The method further includes automatically generating multiple labels, Each label represents one of several training cases. Each of the aforementioned labels indicates whether a pair of text items come from the same section in the document. The output of the first model is input to a second model that is different from the machine learning model. The second model predicts whether the input pair of text items are from the same section in the document. The method according to claim 11, further comprising training the second model based on the plurality of training examples.

13. The aforementioned method, Based on the aforementioned image, identify a set of words extracted from the aforementioned image, To generate the multiple phrase data representations based on the aforementioned set of phrases, Based on the aforementioned image, the layout features of the set in the document are identified, The method according to claim 11 or claim 12, further comprising generating one or more layout data representations based on the set of layout features.

14. One or more storage media for storing instructions that, when executed by one or more computing devices, result in the execution of the method described in any one of claims 1 to 13.

15. It is a system, One or more computing devices, A system comprising one or more storage media that, when executed by one or more computing devices, provides instructions that result in the execution of the method described in any one of claims 1 to 13.

Images