A machine learning system for automatic document segmentation and classification

Abstract

A computer-implemented method for automatically splitting and classifying an input document into one or more sub-documents using a machine learning system is described. The machine learning system includes a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem. The method includes receiving visual input representing a plurality of pages of the input document, using the visual segmentation neural network to classify each page of the input document into each of a plurality of templates, determining, for each page of the input document, the final document type to which the page belongs, and using the grouper subsystem to group the plurality of pages of the input document into one or more sub-documents based on (i) each template of each page and (ii) each final document type to which each page belongs.

Description

[Background technology]

[0001] The present disclosure relates generally to the field of document processing, and more particularly to machine learning systems for automatic document segmentation and classification.

[0002] In today's digital age, large volumes of documents are created and managed by organizations and individuals every day. These include, for example, contracts, invoices, legal documents, financial documents, and other important documents critical to the functioning of an organization. Managing and processing these documents can be time-consuming and labor-intensive. Additionally, manual processes can introduce the risk of errors and inconsistencies that can affect the accuracy and reliability of the information stored. Summary of the Invention

[0003] In general, one innovative aspect of the subject matter described herein can be embodied in a computer-implemented method for automatically segmenting and classifying an input document into one or more sub-documents using a machine learning system. The input document includes a plurality of pages. The machine learning system includes a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem. The method includes: receiving a visual input representing a plurality of pages of an input document; classifying each page of an input document into a respective one of a plurality of templates using a visual segmentation neural network; For each page of the input document, Generate a set of text lines from the text content of the page using an optical character recognition subsystem; using a title classifier to process each line of text in the set of lines of text to generate a corresponding confidence score representing the probability that the line of text contains the title of the page; Selecting the text line having the highest confidence score based on the confidence scores of the set of text lines; Determine whether the highest confidence score exceeds a threshold; responsive to determining that the highest confidence score exceeds a threshold, processing the selected lines of text using a document classifier to generate a respective document score for each of a plurality of document types, each respective document score representing a probability that the page belongs to the document type; performing an action including selecting the document type having the highest document score as the final document type to which the page belongs; Using the grouper subsystem, grouping multiple pages of the input document into one or more sub-documents based on (i) each page's respective template and (ii) each page's respective final document type.

[0004] In some implementations, the machine learning system further includes a discriminative extraction neural network. In these implementations, the method includes, in response to determining, for each page of the input document, that the highest confidence score does not exceed a threshold, processing each line of text in the set of lines of text to identify an identification number of the page using the discriminative extraction neural network. In these implementations, grouping the multiple pages of the input document into one or more sub-documents using the grouper subsystem is further based on the identification number of the page.

[0005] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs configured to perform the operations of the methods encoded on computer storage devices.

[0006] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following technical advantages. Improved Accuracy: The described system takes into account both visual and textual features of a document and therefore achieves better performance in document processing tasks compared to existing systems. In particular, by processing both visual and textual features of a document using a specific combination of a visual segmentation neural network, a title classifier, a document classifier, and optionally an identification extraction neural network, the machine learning system described herein can identify the template, document type, and identification number (if present) of each page. Using the identified information, the described system can segment a document into one or more subdocuments with greater accuracy compared to existing systems. This results in a more streamlined, more efficient, and more effective document processing system. Scalability: The described machine learning system can handle large volumes of documents in a short time, making it suitable for use in organizations that must process large volumes of documents daily. For example, the described system can process hundreds or thousands of documents in a few minutes, a task that humans cannot perform manually. Improved Speed: By using a visual segmentation neural network, a title classifier, a document classifier, and optionally a discriminative extraction neural network, the described system is able to process documents faster compared to existing document processing systems, thus saving organizations a significant amount of time and resources. Robustness and Customizability: Machine learning systems are designed and trained to be robust to changes in document format, layout, and style. This means that they can still perform accurately even if document formats change over time. Furthermore, the use of multiple templates and document types allows the machine learning system to learn and adapt to meet the specific needs of an organization. Improved Data Quality: The described machine learning systems can accurately classify each page of a document into the appropriate document type, which results in improved data quality and can reduce the manual effort otherwise required to review and correct errors in existing document processing systems. Additionally, the visual segmentation neural network, title classifier, document classifier, and discriminative extraction neural network enable improved data extraction from documents, allowing organizations to make more informed decisions based on the extracted data. Improved Document Management: The described machine learning system's ability to automatically group sub-documents based on template, document type, and identification number improves the overall management and organization of documents, which helps reduce the time and computational resources required to search for documents among a large number of documents.

[0007] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates an exemplary machine learning system for automatic document segmentation and classification.

[0009] [Figure 2] Here is an example of classifying each page of a document into a corresponding template.

[0010] [Figure 3] 1 illustrates an exemplary process for processing a page to generate a confidence score for each of multiple lines of text within the page.

[0011] [Figure 4] FIG. 2 illustrates an example of a positive set and a negative set of keywords representing a document type.

[0012] [Figure 5A] 1 illustrates an exemplary process for determining whether two consecutive pages belong to the same template. [Figure 5B] 1 illustrates an exemplary process for determining whether two consecutive pages belong to the same template.

[0013] [Figure 6] FIG. 1 is a flow diagram of an example process for processing an input document to divide the input document into one or more sub-documents.

[0014] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0015] This specification describes a machine learning system implemented as a computer program on one or more computers at one or more locations configured to automatically segment and classify an input document into one or more subdocuments. The machine learning system includes a visual segmentation neural network, an optical character recognition (OCR) subsystem, a title classifier, a document classifier, a discriminant extraction neural network, and a grouper subsystem.

[0016] In general, an input document is a collection of interlinked sub-documents, each of which can belong to any of a set of document types.

[0017] For example, in some implementations, the set of document types includes one or more of an invoice, a delivery note, a purchase order, or an email. For example, a five-page input document may include a tax invoice on pages 1 and 2, a delivery note on page 3, and purchase orders on the remainder. As another example, a 300-page input document may include emails on pages 1-57, purchase orders on pages 58-137, delivery notes on pages 138-212, and invoices on the remaining pages.

[0018] In some other implementations, the set of document types includes one or more financial document types, for example, the set of document types includes one or more of a balance sheet, an income statement, a cash flow statement, a statement of owner's equity, an annual report, a quarterly report, a note to financial statements, or a text return report.

[0019] In some other implementations, the set of document types includes one or more legal document types, for example, the set of document types includes one or more of an employment agreement, an independent contractor agreement, a non-disclosure agreement, a loan agreement, a consulting agreement, a partnership agreement, a corporate law agreement, a business agreement, or a sales agreement.

[0020] In some other implementations, the set of document types includes one or more medical document types, for example, the set of document types includes one or more of a doctor's note, a prescription, a vaccination record, a medical record, a discharge summary, a medical test result, a surgery report, a consent form, or an email confirming an appointment.

[0021] The set of document types can be expanded based on the header content and / or title of the document.

[0022] 1 illustrates an exemplary machine learning system 100 for automatic document segmentation and classification. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, which may implement the systems, components, and techniques described below. Machine learning system 100 includes a visual segmentation neural network 104, an OCR subsystem 108, a title classifier 112, a document classifier 114, a discriminant extraction neural network 120, and a grouper subsystem 118.

[0023] To process an input document, system 100 first receives visual input 102 representing multiple pages of the input document. For example, visual input 102 may be a PDF file that is a combination of all scanned images of all pages of the input document.

[0024] The visual segmentation neural network 104 is configured to classify each page in the visual input 102 into a respective template 106 in the set of templates. Each template can be used to generate a different document type. Different templates have different visual appearances. Specifically, each template has one or more corresponding visual features, including, but not limited to, a logo, padding, background, header, footer, chart, and font style.

[0025] The visual segmentation neural network 104 is a convolutional neural network (CNN) that includes one or more convolutional neural network layers. The CNN is trained for a document template recognition task. Typically, each of the one or more convolutional neural network layers includes multiple artificial neurons. The artificial neurons are arranged in a 2D or 3D grid called a filter. Each filter extracts a different type of feature from the input data. For example, from an image, one filter can extract edges, lines, circles, or more complex shapes.

[0026] To classify each page of the input document, the neural network 104 vertically combines the page with the previous page in the document to generate a combined two-page input and provides this combined two-page input to one or more trained CNN layers. The one or more trained CNN layers extract a set of visual features from each page of the combined two-page input and compare the two sets of visual features to determine whether the page and the previous page have the same template. The one or more visual features may include a logo, padding, background, header, footer, chart, or font style.

[0027] If the current page and the previous page have the same visual features, this means they have the same template, and one or more trained convolutional neural network layers associate the current page with the known template of the previous page. If the current page and the previous page have different visual features, this means they have different templates, and one or more trained convolutional neural network layers associate the current page with a new template. For example, as shown in FIG. 5A, one or more convolutional neural network layers 500 of neural network 104 process a combined two-page input 502 and determine that the two pages in input 502 belong to the same template. In contrast, FIG. 5B shows one or more convolutional neural network layers 500 process a combined two-page input 504 and determine that the two pages in input 504 belong to two different templates.

[0028] The result of the process of classifying each page of the input document is shown in Figure 2, where the visual segmentation neural network 104 uses one or more trained convolutional neural network layers to classify pages 1-4 of the input document into template 1 and pages 5 and 6 of the input document into template 2. Figure 2 shows a simplified implementation. In practice, the input document may have tens, hundreds, or thousands of pages.

[0029] The following sequence of operations is performed for each page of the input document.

[0030] For each page, the OCR subsystem 108 is configured to generate a set of text lines 110 from the text content of the page. The title classifier 112 then processes each text line in the set of text lines to generate a corresponding confidence score that represents the probability that the text line contains the title of the page. The title classifier 112 is configured to select the text line with the highest confidence score based on the confidence scores of the set of text lines. In some implementations, the title classifier 112 is a random forest classifier. The process for generating the confidence scores is described in more detail below with reference to FIG. 3.

[0031] The system 100 determines whether the highest confidence score exceeds a threshold. If the highest confidence score exceeds the threshold, which means that the selected text line is highly likely to contain a title, the system 100 processes the selected text line to determine the document type 116 of the page. In particular, the system 100 uses the document classifier 114 to process the selected text line to generate a respective document score for each document type in the set of document types. Each document score represents the probability that the page belongs to that document type. The system 100 selects the document type with the highest document score as the final document type 116 to which the page belongs.

[0032] More specifically, as shown in FIG. 4 , the system maintains (i) a positive set of keywords 402 representing document types that are likely to match the page's final document type, and (ii) a negative set of keywords 404 representing document types that are unlikely to match the page's final document type. For example, for invoice classification, the positive set of keywords 402 may include "invoice," "tax invoice," "order invoice," and "commercial invoice," while the negative set of keywords 504 may include "delivery note," "insurance," "purchase order," and "email." The document classifier 114 is configured to calculate a respective document score for each keyword in the positive set of keywords 402 and the negative set of keywords 404. The document score for each keyword represents the probability that the page belongs to the document type represented by that keyword. The system 100 then selects the keyword with the highest document score. This keyword specifies the final document type 116 to which the page belongs.

[0033] If the highest confidence score does not exceed a threshold, meaning that the selected line of text is unlikely to contain a title, system 100 uses discriminative extraction neural network 120 to process each line of text in the set of lines of text to identify a page's identification number 122. The identification number may be, for example, an invoice number or a purchase number. In particular, discriminative extraction neural network 120 extracts key information from each line of text (e.g., by using spatial dual-modality graph reasoning (SDMGR) or graph methods) and identifies the page's identification number 122 from the extracted key information.

[0034] In some implementations, the system 100 can determine an identification number even when the selected line of text is likely to contain a title.

[0035] Once the system 100 has determined each page's respective template, each page's respective final document type, and optionally each page's respective identification number, the grouper subsystem 118 groups the pages of the input document represented by the visual input 102 into one or more sub-documents 124 based on each page's respective template, each page's respective final document type, and optionally each page's respective identification number.

[0036] The components of the machine learning system 100 are designed and trained to be robust to changes in document format, layout, and style. This means that even if the format of a document changes over time, the system 100 can still perform accurately. Additionally, the use of multiple templates and document types allows the machine learning system to learn and adapt to meet the specific needs of an organization.

[0037] 3 illustrates an exemplary process for processing a page 302 and generating a confidence score for each of multiple lines of text within the page 302. First, the OCR subsystem 108 is configured to extract multiple lines of text 304 from the textual content of the page 302. In particular, the OCR subsystem is a neural network trained to recognize text and extract lines of text from scanned images of the page. The OCR subsystem is trained to analyze the text across many levels (e.g., character level, word level, line level, etc.) and iteratively process the image. It looks for different image attributes, such as curves, lines, intersections, and loops, and combines the results of all these different levels of analysis to obtain a final result that enables the OCR subsystem to recognize the text and extract lines of text from the page 302.

[0038] The system 100 then calculates a set of features for each text line (306). For example, the set of features for each text line may include the x-coordinate, y-coordinate, width w, height h, and matching score of the text line. The system 100 calculates the matching score for each text line by using a fuzzy matching method. Specifically, for each text line and for each of the positive set of keywords and the negative set of keywords, the system 100 calculates a respective score for each keyword in the set, where each score represents the probability that the text line contains the keyword. The system 100 selects the highest score among the respective scores as the matching score for the text line.

[0039] The system 100 provides the calculated features of all text lines as input to the title classifier 112 (which is the random forest classifier 408 in this example). The random forest classifier 408 maintains a set of random trees (e.g., N random trees, where N is a positive integer greater than 2) to classify whether each line of text contains a title. Each random tree selects a respective set of random features from the set of features. Based on the selected respective set of random features, each random tree calculates a respective set of confidence scores, including a corresponding confidence score for each text line, where each corresponding confidence score for each text line represents the probability that the text line contains a title. The random forest classifier calculates a final confidence score for each text line by taking a weighted average of the N corresponding confidence scores calculated for the text line by the N random trees. The final confidence score for each text line represents the final probability that the text line contains a title. The text line with the highest final confidence score is most likely to contain a title.

[0040] FIG. 6 is a flow diagram of an exemplary process for processing an input document to divide the input document into one or more sub-documents.

[0041] For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, a machine learning system suitably programmed in accordance with this specification, such as machine learning system 100 of FIG. 1, can perform process 600.

[0042] The system receives visual input representing multiple pages of an input document, step 602. The visual input may be a PDF file that is a combination of all scanned images of all pages of the input document.

[0043] The system classifies each page of the input document into a respective one of a plurality of templates (step 604). Each template can be used to generate a different document type. Different templates have different visual appearances. Specifically, each template has one or more corresponding visual features, including, but not limited to, a logo, padding, background, header, footer, chart, and font style.

[0044] To classify each page of an input document, the system vertically combines the page with the previous page in the document to generate a combined two-page input and provides this combined two-page input to one or more trained CNN layers. The one or more trained CNN layers extract one or more visual features from the combined two-page input and compare them to determine whether the page and the previous page have the same template. If the page and the previous page have the same template, the one or more trained convolutional neural network layers associate the page with the known template of the previous page. If the page and the previous page have different templates, the one or more trained convolutional neural network layers associate the current page with a new template.

[0045] For each page of the input document, the system performs steps 606-616 as follows.

[0046] The system uses optical character recognition techniques to generate a set of text lines from the text content of the page (step 606).

[0047] The system processes each text line in the set of text lines to generate a corresponding confidence score that represents the probability that the text line contains the title of the page (step 608). For example, the system extracts features for each text line and provides the extracted features as inputs to a random forest classifier. The system then uses the random forest classifier to process the extracted features to generate a corresponding confidence score for each text line using a random forest classification method.

[0048] Based on the confidence scores of the set of text lines, the system selects the text line with the highest confidence score (step 610).

[0049] The system determines whether the highest confidence score exceeds a threshold (step 612).

[0050] In response to determining that the highest confidence score exceeds the threshold, the system processes the selected lines of text using a document classifier to generate a respective document score for each of a plurality of document types (step 614), each document score representing the probability that the page belongs to that document type.

[0051] The system selects the document type with the highest document score as the final document type to which the page belongs (step 616).

[0052] In particular, in some implementations, the system maintains (i) a positive set of keywords representing document types that are likely to match the final document type of the page, and (ii) a negative set of keywords representing document types that are unlikely to match the final document type of the page. The system calculates a respective document score for each keyword in the positive set of keywords and the negative set of keywords. The system selects the keyword with the highest document score. The selected keyword specifies the final document type to which the page belongs.

[0053] In response to determining that the highest confidence score does not exceed a threshold, the system processes each text line in the set of text lines to identify a page identification number. In particular, the system extracts key information for each text line and identifies the page identification number from the extracted key information. In some implementations, the system extracts the key information from each text line by using a spatial dual-modality graph reasoning (SDMGR) method or a graph method.

[0054] The system uses the grouper subsystem to group the pages of the input document into one or more sub-documents based on (i) each page's respective template, and (ii) each page's respective final document type, and / or (ii) each page's respective identification number (step 618). By grouping the pages into one or more sub-documents, the system has successfully divided the input document into one or more sub-documents.

[0055] The system may display one or more sub-documents on a user interface of the system. Alternatively, or in addition, the system may transmit one or more sub-documents to a computing device of a user of the system. Furthermore, the system may store one or more sub-documents in an appropriate location in one or more data storages. The one or more data storages may be available locally or on one or more cloud computing systems.

[0056] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0057] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively, or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiver device suitable for execution by a data processing apparatus.

[0058] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0059] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0060] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or even at all, and which can be stored on storage devices in one or more locations. Thus, for example, an index database can contain multiple collections of data, each of which can be organized and accessed differently.

[0061] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and running on the same one or more computers.

[0062] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0063] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0064] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0065] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on a user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0066] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, such as data servers, or middleware components, e.g., application servers, or front-end components, such as client computers having graphical user interfaces, web browsers, or apps through which users can interact with implementations of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0067] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to user devices for the purpose of displaying the data to and receiving user input from users interacting with the devices functioning as clients. Data generated at the user devices, e.g., results of user interactions, may be received from the devices at the server.

[0068] While the specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in certain combinations, and even initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0069] Similarly, while operations may be illustrated in the figures or recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order illustrated, or in any sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0070] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for automatically segmenting and classifying an input document into one or more sub-documents using a machine learning system, the input document including a plurality of pages, the machine learning system including a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem, the method comprising: receiving a visual input representing the plurality of pages of the input document; classifying each page of the input document into a respective one of a plurality of templates using the visual segmentation neural network; For each page of the input document: generating a set of text lines from the text content of the page using the optical character recognition subsystem; using the title classifier to process each line of text in the set of lines of text to generate a corresponding confidence score representing the probability that the line of text contains the title of the page; selecting a text line having a highest confidence score based on the confidence scores of the set of text lines; determining whether the highest confidence score exceeds a threshold; in response to determining that the highest confidence score exceeds the threshold, processing the selected lines of text using the document classifier to generate a respective document score for each of a plurality of document types, each respective document score representing a probability that the page belongs to that document type; performing operations including selecting the document type having the highest document score as the final document type to which the page belongs; using a grouping subsystem to group the pages of the input document into one or more sub-documents based on (i) the respective template of each page and (ii) the respective final document type to which each page belongs.

2. the machine learning system further includes a discriminant extraction neural network; for each page of the input document, in response to determining that the highest confidence score does not exceed the threshold, processing each line of text in the set of lines of text using the discriminative extraction neural network to identify an identification number for the page; The method of claim 1 , wherein using the grouper subsystem to group the plurality of pages of the input document into the one or more sub-documents is further based on the identification numbers of the pages.

3. and in response to determining that the highest confidence score does not exceed the threshold for each page of the input document, processing each line of text in the set of lines of text using the discriminative extraction neural network to identify the identification number of the page. extracting key information within each line of text using the discriminative extraction neural network; and identifying the identification number of the page from the extracted key information using the identification extraction neural network.

4. Extracting key information from each line of text is The method of claim 3 , comprising extracting key information using spatial dual-modality graph reasoning (SDMGR) or graph methods.

5. The method of claim 1 , wherein the visual input is a PDF file.

6. The method of claim 5 , wherein each page of the PDF file is a scanned image.

7. The method of claim 1 , wherein different templates of the plurality of templates have different visual appearances.

8. 10. The method of claim 1, wherein each of the plurality of templates is used to generate a different document type, and each template has one or more visual features, the one or more visual features including a logo, padding, background, header, footer, chart, or font style.

9. The method of claim 1 , wherein the plurality of document types includes an invoice, a delivery note, a purchase order, an insurance policy, and an email.

10. classifying each page of the input document into a respective one of the plurality of templates using the visual segmentation neural network, vertically combining the page with a previous page in the document to generate a combined two-page input for the visual segmentation neural network, the visual segmentation neural network including one or more trained convolutional neural network layers; using the one or more trained convolutional neural network layers to process the combined two-page input to determine whether the page and the previous page have the same template; 2. The method of claim 1, comprising: in response to determining that the page and the previous page have the same template, associating the page with a known template of the previous page; or in response to determining that the page and the previous page have different templates, associating the current page with a new template.

11. processing the combined two-page input using the visual segmentation neural network to determine whether the page and the previous page have the same template; 11. The method of claim 10, comprising using the one or more trained convolutional neural network layers to extract one or more visual features from the combined two-page input, the one or more visual features comprising a logo, padding, background, or font style.

12. The method of claim 1 , wherein the title classifier is a random forest classifier.

13. processing each line of text in the set of lines of text using the title classifier to generate the corresponding confidence score; extracting features for each line of text and providing the extracted features as inputs to the random forest classifier; and processing the extracted features using the random forest classifier to generate a corresponding confidence score for each line of text using a random forest classification method.

14. The method of claim 13 , wherein the features of each line of text include an x-coordinate, a y-coordinate, a width w, a height h, and a matching score of the line of text.

15. and processing the selected lines of text using the document classifier to generate a respective document score for each of a plurality of document types in response to determining that the highest confidence score exceeds the threshold for each page of the input document. maintaining (i) a set of positive keywords representing document types that are likely to match the final document type of the page, and (ii) a set of negative keywords representing document types that are unlikely to match the final document type of the page; calculating a respective document score for each keyword in the set of positive keywords and the set of negative keywords; 2. The method of claim 1, wherein selecting the document type having the highest document score as the final document type to which the page belongs comprises selecting the keyword having the highest document score, the keyword specifying the final document type to which the page belongs.

16. 1. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for automatically segmenting and classifying an input document into one or more sub-documents using a machine learning system, wherein the input document includes a plurality of pages, and the machine learning system includes a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem, and wherein the operations include: receiving a visual input representing the plurality of pages of the input document; classifying each page of the input document into a respective one of a plurality of templates using the visual segmentation neural network; For each page of the input document: generating a set of text lines from the text content of the page using the optical character recognition subsystem; using the title classifier to process each line of text in the set of lines of text to generate a corresponding confidence score representing the probability that the line of text contains the title of the page; selecting a text line having a highest confidence score based on the confidence scores of the set of text lines; determining whether the highest confidence score exceeds a threshold; in response to determining that the highest confidence score exceeds the threshold, processing the selected lines of text using the document classifier to generate a respective document score for each of a plurality of document types, each respective document score representing a probability that the page belongs to that document type; performing operations including selecting the document type having the highest document score as the final document type to which the page belongs; using a grouper subsystem to group the pages of the input document into one or more sub-documents based on (i) the respective template of each page and (ii) the respective final document type to which each page belongs.

17. The operation is 17. The system of claim 16, further comprising: in response to determining, for each page of the input document, that the highest confidence score does not exceed the threshold, processing each text line in the set of text lines to identify an identification number for the page; and wherein the operation of grouping the plurality of pages of the input document into the one or more sub-documents using the grouper subsystem is further based on the identification number of the page.

18. 17. The system of claim 16, wherein the visual input is a PDF file and each page of the PDF file is a scanned image.

19. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for automatically segmenting and classifying an input document into one or more sub-documents using a machine learning system, wherein the input document includes a plurality of pages, and the machine learning system includes a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem, and the operations include: receiving a visual input representing the plurality of pages of the input document; classifying each page of the input document into a respective one of a plurality of templates using the visual segmentation neural network; For each page of the input document: generating a set of text lines from the text content of the page using the optical character recognition subsystem; using the title classifier to process each line of text in the set of lines of text to generate a corresponding confidence score representing the probability that the line of text contains the title of the page; selecting a text line having a highest confidence score based on the confidence scores of the set of text lines; determining whether the highest confidence score exceeds a threshold; in response to determining that the highest confidence score exceeds the threshold, processing the selected lines of text using the document classifier to generate a respective document score for each of a plurality of document types, each respective document score representing a probability that the page belongs to that document type; performing operations including selecting the document type having the highest document score as the final document type to which the page belongs; using a grouper subsystem to group the plurality of pages of the input document into one or more sub-documents based on (i) the respective template of each page and (ii) the respective final document type to which each page belongs.

20. The operation is 20. The one or more non-transitory computer storage media of claim 19, further comprising: in response to determining, for each page of the input document, that the highest confidence score does not exceed the threshold, processing each line of text in the set of lines of text to identify an identification number of the page; and wherein the operation of grouping the plurality of pages of the input document into the one or more sub-documents using the grouper subsystem is further based on the identification number of the page.

Images