Method, system, and program for generating structured document from unstructured document
The method and system convert unstructured documents into structured documents using natural language processing and ontologies, addressing the challenge of extracting and navigating domain-specific information by maintaining location information, enhancing the efficiency and accuracy of information retrieval.
Patent Information
- Application Number
- JP2022546714
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-07
- Filing Date
- 2021-01-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-01-08
AI Technical Summary
Existing technologies face challenges in efficiently extracting and navigating domain-specific information from complex, unstructured documents such as health insurance plans and retirement plan summaries due to their semi-structured or unstructured nature, making it difficult to interpret and extract relevant information.
A method and system that utilizes domain-specific natural language processing and ontologies to convert unstructured documents into structured documents, maintaining location information, allowing for navigation and verification of extracted information.
Enables efficient extraction and navigation of relevant information within unstructured documents, ensuring accuracy and usability through graphical user interfaces that link extracted content back to its original location, facilitating quick access and verification.
Smart Images

Figure 0007593725000001 
Figure 0007593725000002 
Figure 0007593725000003
Abstract
Description
[Technical field]
[0001] Aspects of the present disclosure relate to extracting content from unstructured documents using natural language processing techniques, and more particularly to techniques for generating structured documents from unstructured documents and processing the structured documents using natural language processing techniques. [Background technology]
[0002] Many real-world behaviors are defined in terms of complex, very long documents or sets of documents. These documents may generally be unstructured or semi-structured, with related information contained in paragraphs of text content or in tables or other organizational structures that group similar information together. For example, in a health care context, an employer may offer various health plans to employees, and the various health plans may be summarized in a summary plan document that identifies various characteristics of the health plans offered to the employee, in a table or other organizational structure. This plan summary document may further include additional information that may be applicable to all plans offered to the employee or to a particular plan offered to the employee, and this additional information may be maintained in unstructured text (e.g., paragraphs, bulleted lists, etc.). In another example, retirement plans may be summarized in a plan summary document that outlines relevant information about the various retirement plans offered to the employee, in a combination of tabular and unstructured text. In a further example, various documents in an industrial environment may contain summaries of requirements for the physical goods to be manufactured, such as sizes, materials, tolerances, etc., as well as further detailed information contained in unstructured text.
[0003] Typically, such documents may contain a significant amount of domain-specific information. For example, a health plan summary document may contain information specific to the domain of the health plan, such as copayment information, in-network / out-of-network definitions, etc. A retirement plan summary document may contain information specific to the domain of the retirement plan, such as minimum age requirements for participation, entitlement information, etc. Although these documents contain a significant amount of domain-specific information, these documents may be difficult to interpret and extract information from. Summary of the Invention
[0004] One aspect of the present disclosure provides a method for extracting information from an unstructured document. The method typically includes receiving a source document from which information is to be extracted. The source document is converted into a text document file. Information is extracted from the text document file based on a domain-specific natural language processing engine and a domain-specific ontology. The extracted information is linked to location information in the source document to generate a structured document. The structured document typically includes the extracted information and the linked location information.
[0005] Another aspect of the present disclosure provides a system for extracting information from unstructured documents. The system typically includes a processor and a memory containing instructions that, when executed by the processor, perform operations for extracting information from unstructured documents. The operations typically include receiving a source document from which information is to be extracted. The source document is converted into a text document file. Information is extracted from the text document file based on a domain-specific natural language processing engine and a domain-specific ontology. The extracted information is linked to location information within the source document to generate a structured document. The structured document typically includes the extracted information and the linked location information.
[0006] Yet another aspect of the present disclosure provides a computer-readable medium having instructions stored thereon that, when executed by a processor, perform operations for extracting information from an unstructured document. The operations typically include receiving a source document from which information is to be extracted. The source document is converted into a text document file. Information is extracted from the text document file based on a domain-specific natural language processing engine and a domain-specific ontology. The extracted information is linked to location information in the source document to generate a structured document. The structured document typically includes the extracted information and the linked location information.
[0007] One aspect of the present disclosure provides a method for presenting information extracted from an unstructured document in a graphical user interface. The method typically includes receiving an unstructured document and a structured document containing information extracted from the unstructured document and location information associated with the extracted information. The unstructured document is rendered in a first pane and a graphical rendering of the structured document is rendered in a second pane. The graphical rendering may typically be a structure in which content from the structured document is displayed in a hierarchical format. Each element in the structured document is linked to the rendered unstructured document based on location information included in the structured document.
[0008] Another aspect of the disclosure provides a system. The system typically includes a processor and a memory having instructions stored thereon that, when executed by the processor, perform operations for presenting information extracted from an unstructured document in a graphical user interface. The method typically includes receiving an unstructured document and a structured document that includes information extracted from the unstructured document and location information associated with the extracted information. The unstructured document is rendered in a first pane and a graphical rendering of the structured document is rendered in a second pane. The graphical rendering may typically be a structure in which content from the structured document is displayed in a hierarchical format. Each element in the structured document is linked to the rendered unstructured document based on location information included in the structured document.
[0009] Yet another aspect of the disclosure provides a computer-readable medium having stored thereon instructions that, when executed by a processor, perform operations for presenting information extracted from an unstructured document in a graphical user interface. The operations typically include receiving an unstructured document and a structured document containing information extracted from the unstructured document and location information associated with the extracted information. The unstructured document is rendered in a first pane and a graphical rendering of the structured document is rendered in a second pane. The graphical rendering may typically be a structure in which content from the structured document is displayed in a hierarchical format. Each element in the structured document is linked to the rendered unstructured document based on location information included in the structured document. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 illustrates an exemplary network computing environment in which structured documents are generated from unstructured documents and information is extracted from the unstructured documents using natural language processing on the unstructured documents and the structured documents according to one embodiment. [Diagram 2] FIG. 2 illustrates exemplary operations for extracting information from structured documents and unstructured documents using natural language processing according to one embodiment. [Diagram 3] FIG. 1 illustrates exemplary operations for presenting information extracted from an unstructured document in a graphical user interface according to one embodiment. [Figure 4] FIG. 1 illustrates an exemplary graphical user interface in which an unstructured document and a structured document containing information extracted from the unstructured document are displayed according to one embodiment. [Diagram 5] FIG. 10 illustrates an exemplary graphical user interface in which visual effects are rendered regarding information extracted from an unstructured document, according to one embodiment. [Figure 6]FIG. 1 illustrates an exemplary system in which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Unstructured documents are documents in which information is typically stored in a free-form manner that is not dictated by an underlying structure or relationships between the information stored in the document. Unstructured domain-specific documents, such as institutional summary descriptions, technical specifications, etc., are typically voluminous documents that are difficult to understand and contain a significant amount of domain-specific information. To extract information from these domain-specific documents, the documents may be converted from an image-based format (e.g., portable document format) to a plain text file, and the plain text file may be processed by a natural language processing engine or other analytical engines, or both, to extract the relevant information from the document. However, converting a document to a plain text file may result in some information loss. For example, location information that may be associated with content within a document, such as the location of cells in a table and the associated text within each cell, may be lost in converting a document in an image-based format to a plain text file. Because location information and other relevant information may be lost in converting a document from an image-based format to a plain text file, it may be difficult to map the information extracted from the plain text file back to the original document and to generate a navigable structure that can be used to traverse the document.
[0012] Aspects of the present disclosure provide techniques for extracting information from unstructured documents by generating structured documents that are analyzed by natural language processing engines and other analytical services to extract relevant information. As described in more detail below, the unstructured documents may be converted into structured documents that maintain contextual information associated with textual content within the unstructured documents. The contextual information, along with an identification of the relevant text, may be used by the natural language processing or other analytical engines, or both, to generate structured documents that link the extracted relevant text to the location of such text within the unstructured documents. Aspects of the present disclosure further provide techniques for rendering and navigating the unstructured documents using the structured documents and information extracted from the structured documents. The unstructured documents may be rendered alongside a navigation structure generated from the information extracted from the structured documents, and user manipulation of the navigation structure may enable navigation to the location of various elements within the unstructured documents. By doing so, a user may quickly navigate to the location of relevant information within the document and verify that information was correctly extracted from the unstructured documents.
[0013] 1 illustrates an exemplary networked computing environment in which information is extracted from unstructured documents using structured documents generated from the unstructured documents in accordance with an embodiment of the present disclosure. As shown, computing environment 100 includes a client device 120, an application server 130, a document analyzer 140, a training data store 150, and an ontology data store 160.
[0014] Client device 120 typically represents a computing device on which a user can access services running on application server 130 to view and navigate unstructured documents and can access services running on document analyzer 140 to initiate extraction of information from unstructured documents. As shown in the figure, the client device includes software application 122.
[0015] The software application 122 typically provides an interface through which a user can invoke information extraction operations in the document analyzer 140 and view unstructured documents via services running on the application server 130. In some embodiments, the software application 122 may be a web browser or other application that allows a user to interact with applications hosted remotely on the application server 130. Typically, to invoke information extraction operations in the document analyzer 140, the software application 122 may send commands to the document analyzer 140 to initiate a process to generate a structured document from an unstructured document and to extract information from the structured document. Typically, the command may include information identifying the document to be analyzed or the location where the document to be analyzed is stored, and the document may be an unstructured document (e.g., an image-based format document) from which information is to be extracted.
[0016] The software application 122 may further provide an interface through which the unstructured document and a navigation pane generated from the analysis of the unstructured document may be displayed. The content of the navigation pane may include one or more links to the location of the extracted information in the unstructured document, and when a user selects a link in the navigation pane, the portion of the unstructured document at the location associated with the selected link may be displayed in the first pane. The software application 122 may provide a mechanism through which a user may modify the extracted information displayed in the navigation pane and identify additional relevant information to be included in the navigation pane. Given the extracted information and the location of the extracted information, the modified information and the identified additional information may be added to a structured document (e.g., stored in the document data store 150), and the updated structured document may be used in future displays of the unstructured source document from which the information was extracted.
[0017] The application server 130 typically includes a document viewer 132. The document viewer 132 typically provides an interface through which a user can view and navigate an unstructured document. The interface may include a first pane in which the unstructured document is rendered and a second pane in which a navigation structure is rendered. The first pane may integrate a document viewer that scrolls to a specified portion of the unstructured document in response to a command to navigate to a specified portion of the document. The command to navigate to a specified portion of the document may include, for example, a command to navigate to a specified anchor (e.g., a uniform resource locator (URL) that includes an anchor tag) in the unstructured document, a command to navigate to a particular page or section of the unstructured document, etc.
[0018] The navigation structure depicted in the second pane may be, for example, a navigation tree or other hierarchical structure, where the navigation structure links to specified portions of the unstructured document that contain information associated with the links displayed in the navigation tree. As described in more detail below, each link in the navigation structure may be associated with a bounding box that indicates a location in the unstructured document where the information associated with the link can be found. The bounding box may be determined, for example, by working backwards from one or more intermediate documents generated from the unstructured document, toward the unstructured document itself. For example, the bounding box may be generated by first identifying a set of characters in a plain text intermediate document that was used by the document analyzer 140 to extract information from the unstructured document. The set of characters may be defined, for example, as an offset from a starting character of a text element (e.g., a paragraph, a cell in a table, etc.) in the intermediate document, and the identified offset information in one intermediate document may be linked to location information in other intermediate documents generated from the unstructured document. This location information may include, for example, names of cells in a table, named anchors within the document, absolute or relative location information on a page of the unstructured document (e.g., absolute pixel locations given an assumed resolution of the page in the document, or relative pixel locations as a percentage of the height and width of the page in the document), etc. Based on the location information in the structured document and the one or more intermediate documents, a location of the extracted information may be identified within the unstructured document, and one or more visual effects may be rendered with respect to the location of the extracted information. For example, upon selection of a link in the navigation structure, content associated with the link may be highlighted using a color specific to the content associated with the link. In another example, a colored box may be rendered around the content associated with the link.
[0019] In some embodiments, the navigation structure may provide a mechanism for the user to correct the content extracted from the unstructured document. The text extracted from the document may be inaccurate because text recognition systems (e.g., optical character recognition systems) do not always accurately convert text in an image into the correct text content for various reasons (e.g., image quality, document resolution, etc.). Thus, in a verification process, the user may be allowed to examine the document at the location to which the extracted information is linked, and if the extracted information is inaccurate, the user may manually overwrite the machine-extracted information with accurate data. The accurate data may be written to the structured document storing the data extracted from the unstructured document for future use.
[0020] In some embodiments, document analyzer 140 may not extract all of the relevant data from the unstructured document. For example, document analyzer 140 may time out before analyzing the entirety of the unstructured document, or during analysis, the content of the document may simply be overlooked. While the user is viewing the unstructured document, the user may select relevant content within the unstructured document. Document viewer 132 may parse the selected portion of the unstructured document to identify the selected textual content and the location of the selected portion. Based on the identified portion of the unstructured document, document viewer 132 may add information to the structured document such that one or more structured text elements contain one or more references to specified information deemed relevant by the user of document viewer 132. In some embodiments, document viewer 132 may search one or more intermediate documents, identify other location pointers within the intermediate documents to associate with the identified portion of the unstructured document and the text extracted therefrom, and write the identified location pointers to the structured document along with identification information of the intermediate documents associated with the identified location pointers.
[0021] The document analyzer 140 typically receives unstructured documents for analysis and uses a natural language processing engine and a domain-specific ontology (e.g., ontologies 1621-1622). n The document analyzer 140 extracts information from the unstructured document into the structured document using one or more of the domain-specific ontologies. The domain-specific ontologies may be defined a priori based on the type of document to be analyzed by the document analyzer 140. If the document analyzer 140 is configured to extract information from documents in a particular domain (e.g., healthcare plan summary documents, retirement plan documents, etc.), the domain-specific ontologies may be hard-coded into the document analyzer 140. If the document analyzer 140 is a general-purpose document analyzer that can be used to analyze unstructured documents from a variety of domains, the document analyzer 140 may determine the domain-specific ontology to use based on user input (e.g., an explicit indication of the type of document provided to the document analyzer 140) or based on natural language analysis of the unstructured document to identify the domain of the document based on key words in the document (e.g., words like "hospitalization" or "medical" may be associated with the healthcare document domain, while words like "retirement" or "minimum distribution" may be associated with the retirement document domain). As shown, the document analyzer 140 includes a document converter 142 and a structured document generator 144 .
[0022] The document converter 142 generates one or more intermediate documents from the unstructured document received for analysis. To generate the one or more intermediate documents from the unstructured document, the document converter 142 may first convert the unstructured document into a plain text file using an optical character recognition engine or other text recognition engine. The optical character recognition engine may be integral with the document converter 142 or may be integral with a remote service that returns a plain text file in response to a request to generate a plain text file from an input document. The plain text file may be the first of multiple intermediate documents generated by the document converter 142 for use in extracting information from the unstructured document. The output of the optical character recognition engine may be a plain text file that includes one or more annotations that indicate where text content is located within the unstructured document. For example, the optical character recognition engine may generate a plain text file that includes one or more text elements, and each text element in the plain text file may include information identifying where each text element is located within the unstructured document. In some embodiments, the information identifying a location in the unstructured document associated with the text element may be a unique identifier generated by an optical character recognition engine, for example, based on formatting information in the unstructured document. New location indicators may be added based on predefined rules for generating location markers, such as where typography or typographic formatting indicates section breaks, where page breaks exist in the document, etc.
[0023] A plain text file resulting from processing an unstructured document with an optical character recognition engine may be a Hypertext Markup Language (HTML) representation of the unstructured document. Each text element in the HTML representation may include location information associated with the location of the text element in the unstructured document. This location information may be, for example, an area in the coordinate system of the unstructured document from which the text extracted from the unstructured document was extracted. In general, the plain text file may include structured text that includes text content from tables or other organizational structures parsed from the unstructured document. The structured text, formatted in Java(R)script Object Notation (JSON) format, may maintain the tabular format of the text extracted from the table by storing the table column headers, row headers, and cell contents. The location information in the HTML representation may be represented by a start offset and an end offset of the extracted text in the HTML string. For example, given an HTML string represented as an array of i characters with indices 0 to (i-1), the string may be represented by a start index and an end index in the array as a positive offset from the starting character in the string, or a negative offset from the i-1th character in the string.
[0024] The plain text file may be further processed into the first intermediate document to add one or more text elements to the intermediate document in a structured format. These text elements may include, for example, text elements not already included in a table in the plain text document. The text may be concatenated into a series of flat text elements, and each text element may be associated with a position in an array of text positions relative to its content in the plain text file. Typically, the position information may include the coordinates of a bounding box in the unstructured document and the character offset of the string in the plain text file.
[0025] The first intermediate document may be processed to reorganize the tabular content stored in a structured text configuration. Typically, the structured text may be reorganized to add row headers, section headers, and other information headers included in the document that indicate information about the content contained in that portion of the document.
[0026] Finally, the first intermediate document may be processed into a second intermediate document that adds flat text elements to the body cells of the structured text composition containing the table content. The text of the section headers, row headers, column headers, and body cells may be concatenated into a series of flat text elements. For each flat text element generated by concatenating the section headers, row headers, column headers, and body cell text into a flat text element, position data may be stored. As previously described, the position information may include the coordinates of a bounding box in the unstructured document and the character offset of the string in the plain text file.
[0027] The structured document generator 144 uses the intermediate document generated by the document converter 142 to generate a structured document that includes the relevant information in the unstructured document. Typically, to identify relevant information in the unstructured document, the structured document generator 144 can process the intermediate document generated by the document converter 142 using a natural language processing engine trained on a corpus of domain-specific information. For example, a structured document generator 144 used to extract information from a healthcare plan summary document can use a natural language processing engine trained to recognize relevant information from other healthcare plan descriptions, a structured document generator 144 used to extract information from a retirement plan document can use a natural language processing engine trained to recognize relevant information from other retirement plan documents, and so on. The natural language processing engine can be associated with a domain-specific ontology that identifies relationships between certain domain-specific concepts in the document. For example, an ontology for healthcare plan summary descriptions can include relationships between healthcare-specific configurations, such as in-network and out-of-network coverage, deductibles, copayments, and the like.
[0028] During the extraction process, the structured document generator 144 may record position information associated with flat text elements in the intermediate document and save the extracted text and associated position information in the structured document. In some embodiments, the structured document may include a bounding box associated with the origin of the text in the unstructured document. After the structured document generator 144 generates the structured document according to the domain-specific ontology, the structured document generator 144 may commit the structured document to the document data store 150. In some embodiments, the structured document generator 144 may generate the structured document as a parsable text file (e.g., in JSON format). In some embodiments, the structured document generator 144 may generate the structured document as one or more database records that, when combined, form a structured document that identifies the relevant information extracted from the unstructured document and the position information associated with the identified relevant information (e.g., character offsets and bounding boxes of text in a graphical rendering of the unstructured document).
[0029] The document data store 150 generally represents a data repository in which unstructured documents, structured documents containing information extracted from unstructured documents, and other textual content used to train the natural language processing engine may be stored. The document data store 150 may include multiple separate repositories in which unstructured and structured documents are stored. These repositories may be for documents of different domains, such as, for example, technical requirement documents, medical diagnosis documents, and other classes of documents to be analyzed using the document analyzer 140. The document data store 150 may be structured as a relational or non-relational database, a flat file store, a hierarchical file store, or any other storage system in which documents used to train relevance and knowledge graph models are stored.
[0030] The ontology data store 160 generally represents a data repository in which domain-specific ontologies may be stored for use by the document analyzer 140 in extracting relevant data from unstructured documents. As shown, the ontology data store 160 includes multiple ontologies 1621-1622. n Each ontology 162 may correspond to a particular domain of documents analyzed by document analyzer 140. For example, one ontology 162 may be associated with health care plan summary documents, another ontology 162 may be associated with retirement plan summary documents, and so on.
[0031] 2 illustrates example operations 200 performed by a system to extract information from an unstructured document using natural language processing techniques that maintain location information associated with the extracted information. As noted, maintaining location information associated with the extracted information may be used in rendering the unstructured document with a navigation structure that allows a user to jump to a location within the unstructured document where the extracted information is stored.
[0032] As shown, operation 200 begins at block 210 where the system receives a source document from which information is to be extracted. The source document may be, for example, a document in a file format used by a word processing program (e.g., an OpenDocument Format file, an Office Open XML file, etc.), an image-based file format (e.g., a Portable Document Format file), etc. Typically, the source document may be associated with a particular domain of knowledge that indicates the type of information (or document type) that may be associated with the document.
[0033] At block 220, the system converts the source document into a text document file. To convert the source document into a text document file, a first intermediate file may be generated by passing the unstructured source document to an optical character recognition engine. The optical character recognition engine may generate a plain text file that includes the text content extracted from the source document and a structure that defines the characteristics of the tabular data contained in the source document. The first intermediate file may include position information (e.g., vertical / horizontal position information using the coordinate system of the source document) that the system can use to identify the location of the relevant text in the source document. One or more other intermediate files may be generated to convert the unstructured text content into structured text content that can be analyzed by a natural language processing engine to extract information from the text document. The resulting file may be a structured text document representing the unstructured source document including position information for each text element contained in the source document.
[0034] At block 230, the system extracts information from the text document files based on a domain-specific natural language processing engine and a domain-specific ontology. The domain-specific natural language processing engine may be trained to recognize relevance information in a particular type of document (e.g., healthcare plan summary descriptions, retirement plan summary descriptions, industrial specifications, etc.), and the domain-specific ontology may specify relationships between various words or concepts within a particular type of document.
[0035] At block 240, the system links the extracted information to location information within the source document. As noted, the extracted information may be associated with a particular set of characters contained in a text document file and / or one or more intermediate documents generated from the source document. A set of characters identified by a character offset or character index within a document may be linked to coordinate location information within the unstructured document via location information (e.g., table cell identifiers) contained in one or more intermediate documents.
[0036] At block 250, the system generates a structured document containing the extracted information and the linked location information. As noted, a structured document may be a document written according to a hierarchical schema in which information extracted from a document is stored. In some embodiments, the structured document may be stored in a data repository as a series of database records that, when aggregated, may constitute a single structured document that is used to generate a navigational structure for the document.
[0037] FIG. 3 illustrates example operations 300 that may be performed to render an unstructured document and a navigation structure generated from a structured document that contains information extracted from the unstructured document.
[0038] The operations 300 begin at block 310, where the system receives an unstructured document and an associated structured document that contains information extracted from the unstructured document and location information associated with the extracted information. The structured document may be formatted using a hierarchical schema that can be converted into a hierarchical navigation structure. Typically, the hierarchical schema may be structured such that the nesting of elements within other elements indicates parent-child relationships between various data extracted from the structured document.
[0039] At block 320, the system renders the unstructured document in a first pane of a graphical user interface. The first pane may include a document viewer that supports documents in a range of document file formats commonly used for unstructured documents. For example, the document viewer may be an HTML rendering engine used to display documents in HTML file formats, a PDF rendering engine, etc. In some cases, the document viewer may allow a user to select information within the document but may not include editing features that allow a user to modify the content of the unstructured document.
[0040] At block 330, the system generates a graphical representation of the structured document in a second pane of the graphical user interface. The graphical representation of the structured document may be, for example, a navigation tree or other hierarchical structure that can be used to represent hierarchical data within the structured document (e.g., as nested lists of data). As noted, the structured document may contain the extracted information in a hierarchy of text elements within the structured document, with different levels of information representing parent-child relationships between higher and lower level information.
[0041] At block 340, the system links each element in the graphical representation of the structured document to the unstructured document based on location information included in the structured document. By linking elements in the graphical representation of the structured document to the unstructured document, the system can enable navigation of the unstructured document based on manipulation of elements in the graphical representation of the structured document. As noted, the location information can be determined based on location information included in one or more intermediate documents generated from the structured document, and the location information can be character offsets relative to strings of characters in the intermediate documents. These character offsets can be mapped back to coordinate location information in the unstructured document. Thus, each element in the graphical representation of the structured document can be linked to coordinates in the unstructured document where extracted text is located, and user selection of an element in the graphical representation of the structured document can cause the document viewer to render the unstructured document at the coordinate location associated with the selected element.
[0042] 4 illustrates a graphical user interface displaying an unstructured document and a navigation pane for the unstructured document populated based on a structured document containing information extracted from the unstructured document. As shown, the graphical user interface 400 includes a first pane 410 and a second pane 420.
[0043] The first pane 410 typically includes a document viewer that allows a user to view an unstructured document, and the second pane 420 typically includes a hierarchical viewer in which a structured document is rendered. Each element from the structured document displayed in the hierarchical viewer in the second pane 420 may be user selectable. If the user selects an element 422 in the second pane 420 that corresponds to an element extracted from the unstructured document rendered in the first pane 410, the system may display additional information about the selected element in an information viewer 424 in the second pane 420. To do so, the system may retrieve information from the structured document associated with the selected element. The retrieved information may include, for example, location information associated with the selected element and other relevant information associated with the selected element. For example, as shown, the selected element 422 is associated with information on page 11 of the unstructured document displayed in the first pane 410 and references the "Individual Lifetime Benefit Limit" property of the plan described in the unstructured document, which has a value of "$2,500." This information may be displayed in the information viewer 424 in a read-only format or in an editable format based on user selections in the second pane 420.
[0044] The system may further display in the first pane 410 the portion of the unstructured document in which the selected element resides. In some embodiments, the system may further render one or more visual effects with respect to the portion of the document in which the selected element resides. For example, a bounding box 412 may be drawn around the portion of the document in which the selected element 422 resides in the first pane 410, text contained in the portion of the document in which the selected element resides is highlighted, etc. These visual effects may be rendered based on position information associated with the selected element 422 in a structured document generated from the unstructured document displayed in the first pane 410. As noted, this position information may include the offset of the text in one or more intermediate documents and coordinates within the unstructured document displayed in the first pane 410.
[0045] The second pane 420 also typically includes one or more selectable icons or buttons that allow the user to edit or delete existing information extracted from the unstructured document to the structured document, or to add information from the unstructured document to the structured document, or both. For example, each item extracted from the unstructured document that is included in the structured document may be associated with an edit button 426 and a delete button 428. If the user selects the edit button 426 associated with an item in the structured document, the system may populate the information viewer 424 with information associated with the item in the structured document and make one or more user interface elements in the information viewer 424 editable. If the user indicates that editing is complete, the system may write the edited information back to the structured document. If the user selects the delete button 428 associated with the item in the structured document, the system may remove the item from the structured document and redraw the hierarchical structure displayed in the second pane 420.
[0046] To add a value, a user can initiate the process by operating an add value button 430 displayed in the second pane 420. After initiating the process, a user can select a portion of the unstructured document in the first pane 410. The system can identify a portion of the structured document to add information to based on the location of the selected portion of the unstructured document. At the identified portion of the structured document, the system can add one or more text elements representing the selected portion of the unstructured document and location information within the unstructured document such that the selected portion of the document can be displayed in the first pane 410 when rendered in the second pane 420.
[0047] 5 illustrates a graphical user interface displaying an unstructured document and a navigation pane for the unstructured document. Similar to FIG. 4, the graphical user interface 500 includes a first pane 410 and a second pane 420.
[0048] As shown, extracted text 528 may be selected by the user for analysis by clicking a button provided in second pane 420 to display expanded information (e.g., location and relationship information) about extracted text 528. In some embodiments, selecting extracted text 528 may cause the system to render one or more visual effects for elements in the structured document that correspond to the parent element of the selected extracted text 528. In this example, extracted text 528 may be an immediate child of extracted text 526, which may be an immediate child of extracted text 524, which may be an immediate child of extracted text 522. Extracted text 522 may represent the final parent text object in the hierarchical navigation structure of extracted text 528. In this example, the user selected an amount related to an emergency department visit in the unstructured document rendered in first pane 410, then related to the out-of-pocket amount for which the user is responsible, and finally related to information regarding a "low deductible PPO."
[0049] Based on the user selection of extracted text 528, the system can look up location information within the structured document generated from the unstructured document rendered in the first pane 410 and identify the location of each of the extracted texts 522, 524, 526, and 528 within the unstructured document. As noted, the identified location of each of the extracted texts can include the offset of the text within the string and the coordinate location of the extracted text within the unstructured document. Based on the identified location of the extracted text, the system can render one or more visual effects around the associated text within the unstructured document rendered in the first pane 410 and corresponding visual effects around the extracted text in the second pane 420. For example, a different colored bounding box can be rendered in the first pane 410 and the second pane 420 for each piece of extracted text associated with the user selected extracted text (i.e., associated with extracted text 528). As shown, bounding box 512 may correspond to extracted text 522, bounding box 514 may correspond to extracted text 524, bounding box 516 may correspond to extracted text 526, and bounding box 518 may correspond to extracted text 528. To enable quick recognition of the extracted text elements in the second pane 420 and the corresponding associated text in the unstructured document rendered in the first pane 410, the system may render visual effects that are the same or similar to the bounding boxes in the first pane 410 and the associated associated text in the second pane 420. In this example, bounding box 512 and a box drawn around associated extracted text 522 may include a border of a first color, bounding box 512 and a box drawn around associated extracted text 524 may include a border of a second color, bounding box 516 and a box drawn around associated extracted text 526 may include a border of a third color, and bounding box 518 and a box drawn around associated extracted text 528 may include a border of a fourth color.In other examples, for each of the extracted text 522, 524, 526, and 528, a different visual effect may be rendered in the unstructured document displayed in the first pane 410. For example, based on the coordinates of the bounding boxes described above, a different color highlighting may be applied to the associated text in the unstructured document displayed in the first pane 410, the color of the associated text in the unstructured document displayed in the first pane 410 is changed, etc.
[0050] 6 illustrates an exemplary system 600 that uses natural language processing techniques and ontologies associated with types of unstructured documents to extract information from unstructured documents into structured documents that can be used to navigate the unstructured documents, according to an embodiment. As shown in the figure, system 600 includes, but is not limited to, a central processing unit 602, one or more I / O device interfaces 604 that may enable connection of various I / O devices 614 (e.g., keyboards, displays, mouse devices, pen input, etc.) to system 600, a network interface 606, memory 608, storage 610, and interconnects 612.
[0051] CPU 602 may retrieve and execute programming instructions stored in memory 608. Similarly, CPU 602 may retrieve and store applications residing in memory 608. Interconnect 612 transmits programming instructions and application data between CPU 602, I / O device interface 604, network interface 606, memory 608, and storage 610. CPU 602 is included to represent a single CPU, multiple CPUs, a single CPU containing multiple processing cores, and the like. Additionally, memory 608 is included to represent random access memory. Additionally, storage 610 may be a disk drive, solid state drive, phase change memory device, and the like. Although storage 610 is shown as a single unit, it may be a combination of fixed and / or removable storage devices, such as a fixed disk drive, solid state drive, removable memory card or optical storage, network attached storage (NAS), or a storage area network (SAN).
[0052] As shown, memory 608 includes a document converter 620, a structured document generator 630, and a document viewer 640. The document converter 620 typically receives a document for analysis and generates one or more intermediate plain text documents that the structured document generator 630 can use to generate a structured document. The received document may be, for example, an image-based file format (e.g., portable document format) or other document file format that is rendered on a screen. The document converter 620 may use an optical character recognition engine to convert the unstructured document into a plain text document that may serve as the first of multiple intermediate documents used to generate the structured document. Text content within a table may be formatted in the first intermediate document in a manner that maintains information about each cell in the table, and subsequent intermediate documents may add structure around text elements in the unstructured document that are not included in a table in the unstructured document. Each text element may include position information, such as character offsets and bounding box coordinates within the unstructured document, which may be used by the document viewer 640 in rendering the unstructured document and applying visual effects to portions of the unstructured document.
[0053] The structured document generator 630 typically uses intermediate documents to generate structured documents that contain relevant information extracted from unstructured documents. To generate structured documents, the structured document generator 630 uses a domain-specific natural language processing engine and a domain-specific ontology that are trained to recognize words or phrases that are particularly important to a class of documents. The structured documents generated by the structured document generator 630 may be hierarchical documents that the document viewer 640 can use in rendering navigational structures that maintain the hierarchies and relationships between the data extracted from the documents.
[0054] The document viewer 640 typically provides a graphical user interface in which an unstructured document is rendered in a first pane of the graphical user interface and a navigation structure generated from a structured document containing information extracted from the unstructured document is displayed in a second pane. Typically, when a user interacts with an element in the navigation structure, the document viewer 640 may jump to the portion of the unstructured document containing the selected text element and apply one or more visual effects to the portion of the unstructured document containing the selected text element. The document viewer 640 may further permit a user to edit and remove existing content from the structured document associated with the unstructured document displayed in the first pane, as well as add information from the unstructured document to the structured document.
[0055] Storage 610 includes document data store 650 and ontology data store 660 as shown. Document data store 650 typically serves as a repository where structured documents containing information extracted from one or more unstructured documents are stored. Ontology data store 660 typically contains a priori defined relationships between data in documents of various domains processed by system 600. Document data store 650 and ontology data store 660 may be relational databases, non-relational databases, flat file stores, or any other data repository where encrypted data is stored and retrieved.
[0056] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein.
[0057] In the following, reference is made to the embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or advantages over the prior art, whether a particular advantage is achieved by a particular embodiment is not a limitation of the scope of the disclosure. Thus, the following aspects, features, embodiments, and advantages are merely illustrative and should not be considered as elements or limitations of the appended claims unless expressly set forth in the claims. Similarly, references to "the present invention" should not be construed as a generalization of the subject matter of any invention disclosed herein, and should not be considered as elements or limitations of the appended claims, unless expressly set forth in the claims.
[0058] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system."
[0059] The present invention may be a system, method, and / or computer program product, which may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.
[0060] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridges in a groove on which instructions are recorded, and any suitable combinations of these. As used herein, computer-readable storage media should not itself be construed as being ephemeral signals such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.
[0061] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing device / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing device / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage on a computer readable storage medium within each computing device / processing device.
[0062] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, C++, and traditional procedural programming languages such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the invention, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions.
[0063] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, are implemented by computer readable program instructions.
[0064] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, where the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0065] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process and cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or in reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flowchart illustrations, as well as combinations of blocks included in the block diagrams and / or flowchart illustrations, are implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.
[0067] While the forgoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, the scope of the invention being determined by the following claims.
Claims
1. A method for extracting information from an unstructured document by computer information processing, said method comprising: Receiving a source document from which information is to be extracted; converting the source document into a text document file; extracting information from the text document files based on a domain-specific natural language processing engine and a domain-specific ontology; linking the extracted information to location information within the source document; generating a structured document containing the extracted information and the linked location information.
2. the source document includes a plurality of text elements; converting the source document to a text document file includes generating position indicators for a plurality of text elements within the source document; and generating a first intermediate text document including content of each of the plurality of text elements and an associated position indicator of each of the plurality of text elements.
3. The method of claim 2 , wherein the generated first intermediate text document comprises a structured document formatted according to a predefined schema.
4. the plurality of text elements comprises cells in a table; 3. The method of claim 2, wherein the first intermediate text document replaces the table with a structured representation, with each cell in the table being represented by a disjoint set of elements in the structured representation and associated with a position indicator in the source document.
5. 5. The method of claim 4, further comprising generating a second intermediate text document from the first intermediate text document by concatenating text content not included in the table into one or more flat text elements, each of the one or more flat text elements including an array of locations representing a location of each flat text element within the source document.
6. The method of claim 4 , wherein one or more flat text elements comprise a structured representation of a non-tabular element in the source document.
7. 5. The method of claim 4, further comprising concatenating table headers and table cell content into a flat text element, the flat text element including a bounding box defining a location of the table cell in the source document and a start offset and an end offset within the flat text element.
8. Extracting information from the source documents identifying relevant text information within the text document file and position offset information associated with the relevant text information; and generating a bounding box within the source document based on the position offset information.
9. A system for executing the method according to any one of claims 1 to 8 by computer hardware.
10. A computer program product for causing a computer to execute the method according to any one of claims 1 to 8.
11. 11. A computer readable storage medium having the computer program of claim 10 stored thereon.
Citation Information
Patent Citations
Document processor and document processing method
JP1999250041A
Link embedding method
JP2013037645A
Machine translation method for PDF file
US20090030671A1
Method and system for maximum-informativeness information extraction using a domain-specific ontology
US20110113069A1
Automated capture of technical documents for electronic review and distribution
US5963966A
