Visual structure of documents in question answering

Abstract

A question answering system that receives a natural language question includes a database that provides a basis for the answer, and a structured query generator that constructs a structured query from the question and uses it to retrieve an answer to the question from the database.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is an international application claiming the benefit of U.S. Provisional Patent Application No. 63 / 388,046, filed July 11, 2022, and U.S. Provisional Patent Application No. 63 / 423,527, filed November 8, 2022, the contents of both of which are incorporated herein by reference in their entireties.

[0002] The present invention relates to question answering ("QA") systems, and more particularly to natural language question answering systems. [Background technology]

[0003] A question-answering system accepts written or spoken questions. A "natural language" question-answering system is one that accepts and answers questions posed in a colloquial form that can be used when speaking to others and responding using similar natural language answers.

[0004] Natural language question-answering systems can thus create the impression of interacting with a human being. Partly for this reason and generally due to their ease of use, such systems are often used by organizations to answer questions within limited knowledge domains.

[0005] To answer a question, a question-answering system draws on information contained in a "knowledge base," which is pre-compiled from many documents containing information relevant to a particular knowledge domain.

[0006] Documents are the kind found in print, such as books, manuals, papers, specification sheets, etc., and / or on a computer screen, such as a web page. In some cases, these documents are scanned images of physical documents stored in computer files. In some cases, these documents are "digitally generated," i.e., they arise without the need to scan a physical document. Some documents are hybrid documents, having both scanned and digitally generated content. In all of these cases, the documents can be rendered in a visible form. Question-answering systems utilize the text and images found in such documents.

[0007] Documents such as those described above can be created in many different ways. It is not surprising, therefore, that they are received in any of a variety of formats, including portable document formats, text files, and proprietary formats associated with various applications. To enable a question-answering system to access the information in these documents, the process of building a knowledge base involves certain preprocessing steps that include, among other things, making the information in those documents available in a common format. For example, the question-answering system may use the text of the documents, which may be directly accessible in digital form or may be obtained using optical character recognition (OCR) of an image of the document.

[0008] Such document pre-processing is often referred to as "ingestion." The ingestion process involves certain processing steps that facilitate locating the information needed to answer a question. Details about ingestion itself, and the structure and operation of question-answering systems of the type contemplated herein, are set forth in WO 2021 / 263138, published December 30, 2021, the contents of which are incorporated herein by reference. Summary of the Invention [Means for solving the problem]

[0009] Some information retrieval tasks, such as natural language-based question answering, may be performed on the textual content of a document. However, some tasks may utilize the visual structure of a document (e.g., the arrangement of parts of the document and / or their visual appearance, such as font size, style or color, or background color), which may not be apparent from the textual content of the document for one or more reasons. These uses of visual structure may include (1) extracting more information than might be available from the document text alone to be used in answering a question, (2) enhancing the presentation of information (e.g., answers) returned to a querying user, and / or (3) low-level processing, such as optical character recognition (OCR) of document images.

[0010] Some documents that are processed (e.g., "ingested") for subsequent information retrieval may contain semantic markup, such as tagged headings, subheadings, page headings, figure captions, and so on. However, in practice, such markup is rare, and even when a document does have markup, the markup may, for example, focus on the document's rendering rather than its content and not expose the semantic relationships between elements. One example of markup that can expose semantic structure is the Structured Standard Markup Language (SGML) document format, which may identify nested elements such as chapters, sections, and subsections. One example of markup that can affect the visual characteristics of the rendering, such as is commonly used for web pages, is the Hypertext Markup Language (HTML).

[0011] However, in many cases, tags may affect font size and alignment but may not expose semantic relationships between elements. Another format that can be considered markup is the PostScript format or its embedding within a Portable Document Format (PDF) document. The PostScript format can explicitly encode the positions and visual characteristics of characters and other elements when printed on a page (e.g., because they are typesetting machine instructions) without exposing semantic relationships between elements.

[0012] It should be noted that in the following discussion, when we refer to "visual structure," we mean that this includes encoding the structure of the page as it is rendered (e.g., on a screen or on a printed page), without the page needing to be rendered directly or even formed in pixel image form for automatic processing. That is, rendering can be simulated to infer the visual structure of a document. Furthermore, when we refer to "vision-based" processing of a document, we are referring to processing that may be performed by an automatic image processing system.

[0013] The first step in exploiting the visual structure in an input document is segmentation into distinct parts, which can be called "visual segments." This step can be followed by interpretation of the parts and linking different parts to aid in subsequent information retrieval. In some cases, segmentation is performed before OCR. In some cases, OCR may be performed before segmentation, or the document's characters are explicitly encoded in the document (e.g., as in PostScript format). If segmentation is performed before OCR, segmentation can improve the OCR process by, for example, allowing OCR of one segment to be separated from OCR of another segment.

[0014] Documents of the type captured by question-answering systems typically contain text, and may also contain explicit figures or embedded tables and images, which are collectively referred to herein as "textual information," "input textual information" or "original textual information" when distinguished from textual information directly available in the document, and "derived textual information" or "OCR text" when identified during capture.

[0015] However, as introduced earlier, the total information contained in a document is greater than that provided by the textual information alone. In fact, a document contains structural information that refers to all the information not contained in the textual information (i.e., the textual information and structural information together form the information extracted from the document for question answering). This structural information is visible to a reader of the document when the document is rendered and is not available by simply accessing the textual information alone.

[0016] The systems and methods described herein extract and provide this structural information to question answering systems, which improves the accuracy and comprehensiveness of question answers.

[0017] Structural information includes that derived from the visual structure of a document. This visual structure comes from observing the arrangement of textual content within a document. This visual structure interacts synergistically with the textual content to add semantic information (e.g., context and relationships) that determines the meaning of the text. This addition of semantic context can result, for example, from providing spatial context to the textual content.

[0018] One method for identifying structural information in documents is visual document segmentation. Such segmentation can identify semantic units such as headings, paragraphs, lists, images, tables, and their captions, and can identify relationships between units based on spatial relationships in the visual representation. For example, a caption can be associated with an image by identifying both locations and associating them by proximity. The image can then be searched for using the caption in natural language. Similarly, the content of a separate table often lacks important context that informs what the table content represents. The visual system can identify the table's caption and associate it by proximity to more reliably search the table using a search query. Furthermore, important context for retrieving an object can reside in a document's paragraphs, which can often be made explicit in text (e.g., "As shown in the figure..."). Identifying captions for figures and tables can then serve as queries against a set of documents, broadening the context for retrieving related tables and images.

[0019] Vision-based document understanding processes can detect / identify semantic objects (images, tables) with little if any linguistic context associated with them by proximity-detecting captions and using the language of the captions to search external document evidence, further increasing the linguistic context of the image / table for search query retrieval.

[0020] In a first example of processing document images to identify structure, the first step involves, for a particular page image, identifying segments of the image where information is presented, preferably associating a semantic label with each segment, such as "heading" or "caption." In some examples, these segments are optionally assumed to be rectangular regions, which may be referred to as "bounding boxes" defined by the coordinates of their upper left and lower right corners. One approach to detecting segments is rule-based, whereby a human designer may create a rule such as "A wide, short section within the top x% of a page image is a 'page header.'" Another approach is to use machine learning techniques that use a training corpus of segmented page images to train a segment detector (e.g., to estimate values for configurable parameters of the segment detector). For example, the segment detector may utilize a convolutional neural network that outputs indicators of the bounding box corners of various labeled segments. The output of the segment detector is then used to identify segments. Segments may be disjoint (i.e., non-overlapping) or optionally some segments may have nested segments, for example a "graph" segment may have an "axis labels" nested segment within it, or may have lists identified within lists, or may have paragraphs within a table.

[0021] When processing document images, segmentation can be used to improve character recognition accuracy. One approach to improving character recognition involves background estimation. For example, segments with dark backgrounds can be "lightened" before character recognition. More generally, the image content of a segment can be used to form a mapping from a grayscale or color space to a standardized scale based on what is present in the segment. For example, the standardized scale can be grayscale with a target cumulative intensity distribution. For color images, color content can enter into the mapping specification; for example, if there is white text on a blue background, white can be mapped to black and blue can be mapped to white, thereby standardizing the image. A variety of techniques can be used to specify the mapping, for example, by specifying the color and / or intensity distribution of points (e.g., pixels) within a segmented image and forming a mapping to a grayscale or black-and-white (e.g., binary) image. Preferably, an OCR system is trained or configured to process character images at such a standardized scale, thereby improving character recognition accuracy.

[0022] Other preprocessing steps that may be used include rescaling, binarization, denoising, dilation / erosion, and / or rotation / deskewing. The character recognition task may be approached as a sequence-decoding problem, with optimal decoding of the sequence being achieved using a recurrent network or a transformer. One approach is to train a recurrent network or a transformer to implicitly embed character and word language models in the configured parameters. The longer the coherent context (e.g., a caption or paragraph), the better such a model will be at predicting the next word, thereby reducing error rates in noisy visual text, for example. Another approach is to explicitly train a language model jointly with the underlying words and characters that compose them. Note that visual-based segment labels (e.g., "caption") may be used to inform the language model, for example, by biasing it toward the type of language likely to be found in such segments.

[0023] Segmentation can also improve character recognition by allowing different segments to be processed in different orientations. For example, a figure caption may have a landscape orientation, while the vertical axis label can be rotated 90 degrees (e.g., extending up and down instead of left and right). As another example, a large table or figure containing text may be landscape-oriented, while the corresponding caption is portrait-oriented; the recognized text, along with the estimated orientation, is at the visual segment level, benefiting from a longer period of consistent context. One approach to account for possible rotations is to repeat the OCR process for each segment, e.g., at rotations of 0, 90, 180, and 270 degrees. Another approach is to estimate orientation at the initial segmentation, e.g., by training a machine learning system to provide not only segment labels but also categorical or continuous text orientation estimates.

[0024] As introduced above, segmentation problems can exist even when characters are explicit in a document format, but the semantic units that form the segments are not. For example, an Adobe Portable Document Format (pdf) document may have a strict representation of where character glyphs should be placed, their size, rotation, etc., but the pdf document may not explicitly expose visual structure that would be immediately apparent to a human reader of a rendered version of the document (e.g., printed, displayed). One approach to segmenting such document formats is to generate an equivalent document image (e.g., as a pixel-based image in computer memory) and then form segments using rule-based or previously discussed machine learning techniques for segmentation purposes prior to character recognition. In this case, however, the character recognition stage may revert to an explicit document format rather than relying solely on the document image. Similarly, some document formats require explicit representations of characters and segments (e.g., HTML " " tags) may be explicitly encoded, but again such encoding may affect the visual presentation without necessarily exposing the semantic classes and relationships.

[0025] The original format of the input document dictates the specific steps taken to arrive at the captured (e.g., "curated") content. For example, in the case of HTML, a web page may first be rendered in a visual, headless browser, and style information is calculated for each element on the page. After the source HTML is augmented to include external style labels, the augmented file is downloaded and preprocessed to ensure visibility and preserve inline formatting. The HTML is then parsed into an element tree, which is recursively traversed to create a flattened list of high-level elements defined according to a combination of hard-coded rules and configurable CSS selectors. All container elements that do not themselves serve a semantic function are replaced with their content. Once the list of high-level elements is created, a final post-processing step is performed to further ensure HTML validity and reapply inline formatting. At this stage, any elements can be removed or down-weighted based on their tags.

[0026] In contrast, PDF files may undergo a visual processing step in which various structural elements such as headings, tables, and footnotes are identified based on their appearance. Any items tagged with undesirable tags such as "Table of Contents" may be downweighted or discarded.

[0027] The language model may be used to discover non-standard text, including foreign languages. Any text that does not score above a minimum threshold, which is the general English language model, is discarded or weighted low.

[0028] In addition to content curation, it can be advantageous to standardize documents to map variations such as PC vs. PC to a uniform format. To do this, define a standard form for various terms and the variants that map to it. Each time one of the non-standard variants is encountered, it is mapped to the standard term. The same standardization can be applied to queries as well.

[0029] Identifying the semantic class of a segment may be based on processing of the document image, on classification of the text within the segment (e.g., recognized and / or explicitly provided in the document format), or on a combination of such sources.

[0030] One example of a visual segment that may be easily recognizable by humans but is not explicit in the document (either as an image or encoded in the document format) is a legend regarding confidentiality, distribution rights restrictions, etc. (e.g., "Confidential," "Top Secret," etc.). Sometimes such legends are printed in normal text in a header or footer or on a title page, or they may be printed so that they are visible but not easily recognized automatically, for example, as a gray diagonal "watermark" printed across the page. Automatic visual segmentation and classification can be used to detect such legends, thereby allowing tagging of relevant information on the page where restrictions are desired. Such restrictions can then be observed during subsequent information retrieval.

[0031] Another example of a visual segment that may be readily recognizable by a human reader is a table of contents (or other structured section, such as an index). Visual indicia of such a segment may include a list-like format, leader lines, or dots connecting title words, page numbers, etc. It is generally undesirable to return answers to questions by referencing the table of contents rather than or in addition to the actual content referenced in the table of contents. Therefore, classification of such segments can be used to prevent retrieval of content from these segments (e.g., by eliminating the content of such segments for question-answering purposes). On the other hand, discovery of table of contents segments, when linked to content identified in the table of contents, can be useful to provide context away from nested explicit headings that may not be easily discoverable from the rest of the document itself.

[0032] Documents may also have images (e.g., photographs), and text may be presented within these images. For example, photographs of natural scenes may have road signs or other text presented in live viewing, which may be useful for searching. Similarly, synthetic images may contain text, for example, in the form of text overlaid on the image. By identifying image segments and performing text detection processes on the images, the text can be incorporated into the information available for searching from the document. A situation in which synthetic images may be frequent is in presentations (e.g., PowerPoint slides). Often, it is the text within those synthetic images that may provide the most relevant information available in a document.

[0033] Semantic relationships between segments, such as relationships between the content of segments on the same page or across pages, can aid in information retrieval and can be identified, for example, by spatial relationships. As a simple example, detecting and processing headings such as chapters or sections in a document provides the content of consecutive paragraph segments within that chapter or section. One way to encode such relationships is by linking segments; for example, a paragraph segment is linked to a section heading segment with an "intra-section" link. Such relationships can be used in ingestion to absorb context during a single input pass.

[0034] Another class of semantic relations concerns figures and / or tables. Segments of a figure or table may contain (possibly nested) captions. A figure may contain a graph with horizontal and vertical axis label segments or key segments. These segments may be linked in a semantic graph, for example with a nested "figure"-"graph"-"axis".

[0035] The semantic relationships of visual segments may extend based on the content of the segments. For example, a text segment (e.g., a paragraph) may contain the text "With reference to the graph shown in Figure 1...", and a caption segment may begin with the text "Figure 1. Graph...". These visual segments are linked by context.

[0036] The linkage of segments, such as a caption and text that references that caption, may not be explicit (e.g., "With reference to Figure 1" and "Figure 1"), and natural language techniques may be used to identify similarities between segments. For example, a paragraph that discusses a graph is likely to be similar to the caption of that graph. One technique for identifying such linkages or relationships is to use a question-answering mechanism, for example, that treats the caption as a question and finding the relevant paragraph as the answer.

[0037] Another type of visual segment that can be processed by visual segmentation is a table. For example, the presence of columnar text, headers above the columns or to the left of the rows, provides one way of organizing information. For example, a human reader scanning a document containing a table naturally scans for matrix labels and values at intersections. One way to link tabular information is to associate each cell in the table with the associated row and / or column headers (e.g., as a separate segment).

[0038] While relationships between segments can be specified by rules, e.g., linking captions with axis labels, etc., another option is to form an entire generative model of the document, including the locations of segments and characters within the segments, and then address the visual segmentation and character recognition problems as a combined recognition problem.

[0039] Related to processing the visual structure of a document for information retrieval is the problem of identifying the document's structure in order to present the retrieved information to a user. A document can exist in two formats: the original format of the document as rendered (e.g., printed), and the text format suitable for natural language processing tasks such as question answering. Neither of these formats is particularly suitable for presenting information to a user. For example, text forms used for information retrieval are generally normalized to be useful for automated processing, but this may obscure the nature of the original document. For example, if the answer is in a table, a presentation of the original table is likely to be preferable to a normalized text form of the table. On the other hand, presenting the original document has several drawbacks. One drawback is that different documents are available in a wide range of formats (e.g., PDF, HTML, scanned / OCR, etc.), and therefore, in practice, the presentation of information may need to be adapted for each format. Furthermore, even with adaptation for each document format, users may perceive differences between the rendering of parts of a scanned document and parts of a structured document, and the differences in presentation may be distracting when multiple top-ranked answers are presented to the user. Furthermore, if the output includes context such as section headings, paragraphs (e.g., with highlighting of answers within the paragraphs), referenced graphs or tables (e.g., with highlighting of answers presented within the graphs or tables), it may be desirable to construct an output that both reassembles (but is not necessarily an exact reproduction of) the original document and highlights the answers the user is looking for.

[0040] One solution for presenting information to users in response to information search queries is to use a third format for the document, in addition to its original format and the format used for natural language processing, which can be called a "search view." This third format conforms to a Document Object Model (DOM) that encodes both the content (i.e., words), specific formatting characteristics from the original document, and information derived from the visual segmentation described above, such as headings, captions, etc. Furthermore, visual processing that identifies things like headings, titles, captions, lists, etc., can be used to inject a hierarchy of information into the search view to enhance the information presentation value of the search view.

[0041] This particular form of the DOM utilizes a "simplified HTML" format that provides a way for a document's markup to be rendered from an original document that preserves its general format and structure but does not produce an exact replica of the original document when rendered for presentation to a user.

[0042] Nevertheless, the rendered presentation is preferably visually pleasing ("good looking") and well-organized. Maintaining a common representation across different original formats can also provide uniformity and consistency across multiple results presented simultaneously or in response to different queries.

[0043] Returning to document segmentation, segmentation can improve the quality of retrieved information. As a simple example, consider a document that has technical information about gas turbines and steam turbines in different sections, which may be long sections spanning many pages. A question such as "What is the recommended operating oil pressure for steam turbines?" should preferably find its answer within the steam turbine section. By incorporating the context of a section heading such as "Steam Turbines" along with a context such as "Oil pressure should be maintained between 40 and 80 psi," the correct answer can be retrieved, and answers intended for gas turbines may be filtered out or scored as less preferred.

[0044] One way to incorporate context, which may be based on relationships between segments formed in visual segmentation, is to form composite content that combines the context with the body of text. For example, for a paragraph of text or a limited (e.g., 200 word) section of a paragraph, a section heading context may be presented in the text. For example, in the example "Steam Turbine" section, the content to which a question may be mapped may be "Steam Turbine / / Hydraulics...", and an explicit context boundary " / / " may optionally be used. That is, the context is repeated in each section of text that may be considered for an answer. Preferably, the training material for the question answerer uses similarity-contextualized passages.

[0045] Such contextualization of answers can be extended to several situations. For example, cells in a table can be contextualized using the table caption and associated column and row labels. Text in a graph can be contextualized using axis labels. As another example, items in a visually identified list can be contextualized by the passage (e.g., sentence) that precedes the list.

[0046] Contextualizing passages into answers may use techniques other than enriching results using concatenation and matching against questions, for example. For example, the context may be processed separately from the passage in which the answer may be found, and both the question and the context may be used (e.g., concatenating the question and answer embeddings) to determine the presence of a relevant answer. Yet other, more complex models may be used, for example, using attention-based techniques in which words in potential answers are transformed based on the context to yield a contextualized representation of the passage.

[0047] Document structures can also be used in multimodal documents. For example, a video recording of a slide presentation can have both audio and visual content. The audio content can be automatically transcribed using a machine-based automatic speech recognizer. Frames of the video can be extracted as images and selected, for example, using keyframe detection techniques. The audio transcript can be synchronized with the extracted images so that it is clear what is being said while the images are being presented.

[0048] In such multimodal cases, images of video frames may be processed using the segmentation techniques described above to identify, for example, slide titles, bulleted lists, etc. Additionally, a series of images (e.g., a series of slides in a video) may be used to identify longer-range context. For example, major section headings of some slides may be presented in a different style (e.g., larger font), which may provide context used to contextualize subsequent slides.

[0049] In such multimodal cases, the transcription associated with a captured frame image may be processed in much the same way that a figure caption or text referencing a figure is printed and processed for that text. For example, if the image shows a graph or table, the transcription may provide context for retrieving the image in response to a question.

[0050] Text segments within an image may be used to contextualize the transcription. For example, a speaker in a presentation may omit obvious context from slides presented simultaneously with the speech. Thus, text retrieved from an image may provide helpful context for retrieving the transcription.

[0051] In the case of multimodal documents with images or video and audio, the results can be presented to the user in audio form, as a transcription, or both.

[0052] Some general aspects introduced above that may be used individually or in combination include the following: Vision-based document understanding systems used to detect / identify semantic objects with little, if any, relevant linguistic context (images, tables) through proximity-detection captioning. Use of the language of the caption to search supporting and external documents to further enhance the language context of the image / table for search query retrieval. A document processing method comprising: processing information representing the visual layout of one or more document units (e.g., printed pages, presentation slides) to identify document segments; and processing the document segments including identifying at least one relationship between two document segments. · Visual layout processing is performed during the ingestion of documents into a document repository for subsequent information retrieval. Processing information representing visual layout includes processing a pictorial representation (e.g., after OCR) or processing layout instructions within a document (e.g., processing instructions within an HTML or Acrobat document) that affect the layout of the document's rendering (e.g., print or display). Identifying document segments involves classifying the segments (e.g., according to semantic categories such as headings, captions, titles, paragraphs, lists, icons, etc.). Automatically identifying a table may include identifying cells in the table and preferably extracting structured data from the table using the identified cells (e.g., their row and column arrangement). Training an object localization / identification system (e.g., a neural network system including, e.g., a convolutional neural network trained to find bounding boxes) to find and / or classify visual segments (e.g., text regions) of documents. Identifying the relationships between segments and / or the content within a segment according to the classification of segments and / or the visual relationships between segments. · Use the identified relationships to add context to the content in the segment for use in automated question answering tasks. Segments may include figures and captions, and preferably relationships between segments may include related figures and their captions. Process segments separately to improve text extraction, e.g. rotate / deskew segments separately and / or apply image transformations to enhance characters for automatic character recognition.

[0053] In one aspect, the invention features a method that includes building a knowledge base for a natural language question-answering system. Building such a knowledge base includes ingesting documents that include first, second, and third visual segments, each of which includes semantic content, and preprocessing the documents to enable the question-answering system to access the semantic content in response to natural language questions posed by users of the natural language question-answering system. Preprocessing the documents includes extracting non-semantic information from the documents using the document's visual structure and incorporating the non-semantic information into the knowledge base. The document's visual structure includes the spatial distribution of the visual segments in the document when the document is in a form that visualizes the document.

[0054] Some practices involve using the visual structure by performing certain steps based, at least in part, on the positions of the visual segments relative to one another and determining whether a contextual relationship should exist between a pair of visual segments based, at least in part, on the positions of the segments.

[0055] Some of these practices include determining, at least in part, that a contextual relationship should exist between the first visual segment and the second visual segment based on the positions of the first and second visual segments relative to each other within the document; determining, at least in part, that a contextual relationship should not exist between the first visual segment and the third visual segment based on the positions of the first and third visual segments relative to each other within the document; and establishing a contextual relationship between the first visual segment and the second visual segment, but not establishing a contextual relationship between the first visual segment and the third visual segment.

[0056] In another practice, using the visual structure includes determining, based at least in part on a distance between the first visual segment and the second visual segment, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on a distance between the first visual segment and the third visual segment, that a contextual relationship should not exist between the first visual segment and the third visual segment; and establishing the contextual relationship between the first visual segment and the second visual segment.

[0057] Yet other practices include using semantic information in addition to non-semantic information to determine whether a semantic relationship should exist between a pair of visual segments.

[0058] Some of these practices include a further step of determining, based at least in part on a reference in the first visual segment to the second visual segment, that a contextual relationship should exist between the first visual segment and the second visual segment; a further step of determining, based at least in part on a lack of a reference in the first visual segment to the third visual segment, that a contextual relationship should not exist between the first visual segment and the third visual segment; and a further step of establishing a contextual relationship between the first visual segment and the second visual segment.

[0059] In yet another practice, extracting structural information from a document using visual structures includes classifying visual segments into different classes based on visual appearance differences of the visual segments; determining, based at least in part on the classes of the first and second visual segments, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on the classes of the first and third visual segments, that a contextual relationship should not exist between the first visual segment and the second visual segment; and establishing a contextual relationship between the first visual segment and the second visual segment.

[0060] In yet another practice, extracting structural information from a document using a visual structure includes determining, based at least in part on the positions of the first and second visual segments relative to each other within the document and the semantic content of the first and second visual segments, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, at least in part on the positions of the first and third visual segments relative to each other within the document and the semantic content of the first and second visual segments, that a contextual relationship should not exist between the first visual segment and the third visual segment; and establishing the contextual relationship between the first visual segment and the second visual segment.

[0061] Further implementations of the present invention include those in which a contextual relationship is established between a first visual segment and a second visual segment based on one being a figure and the other being a caption for that figure, based on the two being in the same position, i.e., having a common location, and based on one being a figure and the other being text overlaid on the figure.

[0062] A wide variety of documents are contemplated as carrying semantic content, including documents that include instructions for rendering the document, where the visual structure of the document is determined based on those instructions, documents in portable document formats such as PDF files, where the visual structure of the document is determined based on rendering instructions expressed in the portable document format, and documents in the form of HTML files, where the visual structure of the document is determined based on tags within the HTML file.

[0063] Another practice includes determining that a first one of the visual segments includes text arranged in a matrix, determining that a second one of the visual segments includes text at an intersection of one of the rows and one of the columns, and establishing a contextual relationship between the first visual segment and the second visual segment based at least in part on the determination.

[0064] In some practices, extracting structural information from a document using visual structure includes tagging at least one of the visual segments as belonging to each class in a set of classes, the set including lists, paragraphs, headings, and images.

[0065] Practices also include those in which one or more of the visual segments includes audio or video. Among these, there are practices in which a first visual segment includes audio and a second visual segment includes a transcript of the concept, where a contextual relationship is established between the two visual segments. Among these practices, there are also practices in which a first visual segment includes audio and a second visual segment includes video, where a contextual relationship is established between the first and second segments to synchronize the audio and video.

[0066] In another aspect, the invention features a natural language question-answering system for providing answers to questions from a user. The natural language question-answering system includes an ingestor for ingesting documents, each of which, when visualized, has a visual structure in which visual segments, each containing semantic content, are spatially distributed throughout the document, a visual parser for extracting structural information from the documents based at least in part on the visual structure, and a knowledge base that incorporates semantic content from the documents and the structural information.

[0067] Embodiments include those in which the visual parser includes a segmentation circuit configured to identify visual segments and classify the visual segments into classes, an interpretation circuit that extracts semantic information from the visual segments, and a linkage circuit that establishes a contextual relationship between a pair of selected visual segments based on inputs from the segmentation circuit and the interpretation circuit.

[0068] These and other features of the present invention will become apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0069] [Figure 1] 1 illustrates a system for extracting structural information from documents. [Figure 2] An example of using structural information to establish contextual relationships as shown in FIG. [Figure 3] 1 illustrates various classes of visual segments within a document of the type shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0070] 1 shows a question-answering system 10 that includes an ingestor 12 for ingesting documents 14. The documents 14 are used to build a knowledge base 16. This knowledge base 16 is then referenced to formulate answers 18 to questions 20 asked by users 22. Both the questions 20 and the answers 18 can be spoken or written.

[0071] A typical document 14 defines a two-dimensional space with regions containing semantic content. In many cases, these regions are disjoint. In some cases, however, for example, in images with embedded text, these regions overlap.

[0072] It is useful to divide semantic content into "visual segments" 24 and categorize these visual segments 24 into different types or classes. In addition to the visual segments 24, the figure shows specific first and second visual segments 26, 28. Reference will be made herein to these first and second visual segments 26, 28 where necessary for explanation. In other cases, the expression "visual segments 24" will be used.

[0073] In most cases, visual segments 24 contain static content such as text and images. However, there are some documents 14 in which visual segments 24 contain dynamic content. Examples of dynamic content include audio and video.

[0074] The visual segments 24 contribute to the segmentation of information in the document 14. This semantic information arises because each visual segment 24 has a specific meaning, which is formed in the mind of a typical human examining the visual segments 24.

[0075] For ease of explanation, it is useful to consider the existence of a mapping from the visual segments 24 to a space that contains all possible meanings. Such a space is referred to herein as a "semantic space."

[0076] For each visual segment 26, 28, there exists a "context-free mapping" into semantic space. A "context-free mapping" is one that exists when the visual segments 26, 28 are viewed in isolation.

[0077] Adding context can confound the context-free mapping. In the case of document 14, some context arises from the document's visual structure, i.e., the distribution of visual segments 24 in document 14. This visual structure becomes "structural information" that also contributes to the document's information content. The structural information is analyzed, and ultimately, to exploit the structural information arising from this visual structure, question-answering system 10 includes visual parser 30.

[0078] A first visual segment 26 is said to "inform" a second visual segment 28 if the presence of the first visual segment disrupts the context-free semantic mapping of the second visual segment.

[0079] As is apparent, the total amount of information in a document 14 is often greater than the sum of the information in its visual segments 16. The total information of a document includes structural information that arises from the interaction of those visual segments 16 through context based on the visual structure of the document.

[0080] The visual parser 30 extracts this structural information 32 from the document 14 and uses it to establish context between the visual segments 24 and provide results to the knowledge base 16. This structural information 32 can then be used to improve the accuracy of the question answering system and the comprehensiveness with which it responds to the question 20.

[0081] Visual parser 30 evaluates the extent of the contextual influence of first visual segment 26 relative to second visual segment 28 as a result of the relative positions of the first and second visual segments in document 14. Where appropriate, visual parser 30 applies operators, referred to herein as “contextualization operators,” to indicate the existence of this contest.

[0082] The contextualization operator is a binary operator whose operands are first and second visual segments 26, 28. Application of the contextualization operator creates a contextual relationship 34 between its operands, i.e., between the first visual segment 26 and the second visual segment 28. The question and answering system 10 uses this contextual relationship 34 when searching the knowledge base 16 for information related to the question 20.

[0083] Visual parser 30 includes a segmentation circuit 36 , an interpretation circuit 38 , and a linkage circuit 40 .

[0084] The segmentation circuitry 36 segments the semantic content into visual segments 24 and classifies the generated visual segments 24 into different types. In some cases, the actual content of the visual segments 24 also provides the basis for determining whether a contextual relationship 34 should be established.

[0085] Interpretation circuitry 38 extracts semantic information from visual segments 24 to assist in this determination.

[0086] The linkage circuitry 40 applies contextualization operators as needed to form contextual relationships 34 between the first visual segment 26 and the second visual segment 28, based at least in part on information provided by the segmentation circuitry 36 and the interpretation circuitry 38.

[0087] 2 shows an example of the usefulness of a contextual relationship 34 between a first visual segment 26 and a second visual segment 28. In this example, the first visual segment 26 contains text that refers to "commercial uses" but omits the word "barium." The second visual segment 28 is a heading that reads "barium." During the formation of the knowledge base 16, a contextual relationship 34 was established between them.

[0088] Without contextual relations 34, question answering system 10 receiving the question "What are the commercial uses of barium?" would likely ignore second visual segment 28. After all, second visual segment 28 mentions "barium" but provides no hint about its commercial uses. Because first visual segment 26 never mentions the word "barium," question answering system 10 may also ignore first visual segment 26. Thus, the generated answer 18 would be disappointing; relevant information in knowledge base 16 would never surface.

[0089] However, as a result of the visual parser's activity during the preparation of knowledge base 16, knowledge base 16 will have structural information 32 that identifies a contextual relationship 34 between first visual segment 26 and second visual segment 28. Thus, with this contextual relationship 34 in place, the same question 20 will lead question answering system 10 to second visual segment 28. To omit any reference to "commercial use," second visual segment 28 will be ignored, but instead will be confronted with structural information 32 that indicates its relationship to first visual segment 26. As a result, thanks to structural information 32, question answering system 10 will return a comprehensive answer 18 to user 22.

[0090] Thus, Figure 2 represents the result of a visual parser recognizing the inherent topology of document 14 and making available information indicative of that topology so that various parts of document 14 that may not appear to have any apparent relationship (other than being within the same document 14) can work together synergistically to more fully answer question 20.

[0091] The initial step in extracting structural information 32 from the visual structure of a document is to have segmentation circuitry 36 identify visual segments 24 and where within document 14 they are located.

[0092] Based on the distribution of semantic content, such as text and images, the segmentation circuitry 36 performs the segmentation procedure by parsing the document 14 into visually observable visual segments 24 and identifying boundaries between the various visual segments 24. In some embodiments, the segmentation circuitry 36 does this by using a parsing model that is trained to recognize salient features that correspond to different types of visual segments 24.

[0093] FIG. 3 shows a document 14 having different types of visual segments 24, all of which are marked with bounding rectangles.

[0094] From FIG. 3 it is clear that different types of visual segments 24 present different visual appearances.

[0095] For example, list 42 can be easily identified by observing the presence of bullets or numbers along the first column and the distinctive alignment of text along the second column to the right of the first column.

[0096] Other visually salient features, namely, confidentiality notice 44, footer 46, which is identifiable because it is repeated in the same location near the bottom border of each page, heading 48, image 50, which is easily identifiable because it has no text, label 52, in-image label 54, page number 56, which is easily identifiable because it is a number that typically increments with each page, paragraph 58, and table of contents 60, have associated with them other visual segments 24 shown in Figure 3. Other examples of visual segments 24 include titles, section headings, figures, tables, and captions.

[0097] After identifying the visual segments 24, the segmentation circuitry 36 classifies them into different types. The segmentation circuitry 36 then tags each visual segment 24 with an appropriate semantic tag. The utility of classifying the visual segments 24 arises because it has been observed that different types of visual segments 24 interact in different ways to create different kinds of contexts.

[0098] Thus, the segmentation circuitry 36 identifies different visual segments 24 and associates an appropriate semantic tag with each such visual segment 24. Examples of such tags include "heading" and "caption." In some examples, the visual segments 24 are assumed to be rectangular regions, each of which defines a "bounding box" that is itself defined by the coordinates of its upper left and lower right corners.

[0099] One method of detecting visual segments 24 is a rule-based method, in which a human designer creates rules for classifying visual segments 24 based on visually observable characteristics. An example of such a rule is "If a visual segment 24 is less than two line widths and occurs within the top 5% of each page of the document, then classify it as a 'page header'."

[0100] Because rule-based methods are time-consuming and labor-intensive, an alternative is to train the segmentation circuitry 36 using a training corpus of documents 14 in which visual segments 24 have been previously identified and classified. Machine learning methods can then be used to extract features relevant to the identification and classification of the visual segments 24. Among these is the use of a convolutional neural network in which the segmentation circuitry 36 outputs indicators of the bounding box corners of the various tagged visual segments 24.

[0101] In other embodiments, the segmentation circuitry 36 forms a generative model of the document 14, which includes the placement of visual segments 24 and the placement of characters within the visual segments 24. In such embodiments, segmenting the document 14 becomes a combined recognition problem in which the interpretation circuitry 38 performs character recognition and the segmentation circuitry 36 performs visual segmentation.

[0102] The output of the segmentation circuitry 36 identifies visual segments 24. In some cases, the visual segments 24 may be disjoint or may have additional visual segments 24 nested within them. For example, a visual segment 24 classified as a "graph" may coexist in the same document region as another visual segment 24 classified as an "axis label."

[0103] In a preferred embodiment, the segmentation circuitry 36 uses a perspective model that is trained to be independent of the orientation of the visual segments 24. Thus, landscape and portrait orientations result in the same visual segments 24. This is particularly useful in documents 14 that have visual segments 24 in both portrait and landscape modes. This may occur, for example, when a figure caption is in portrait orientation, while the captioned figure contains sparse text in landscape orientation.

[0104] To improve performance, it is sometimes useful to ignore certain types of visual segments 24 based on their potentially useful extent. Therefore, the segmentation circuit 36 also performs cleaning and normalization procedures to omit visual segments 24 such as page numbers 56, tables of contents 60, and footers 46 and headers.

[0105] The visual parser's interpretation circuitry 38 extracts semantic content from the visual segments 24. Such semantic content is often useful for identifying the nature of contextual relationships between visual segments 24. For example, a visual segment 24 that explicitly references an image 50 is expected to have some relationship to that image 50. A particularly useful tool for extracting semantic content is optical character recognition. Therefore, it is useful for the interpretation circuitry 38 to include an OCR circuitry 62 to perform this function.

[0106] Information from both the interpretation circuitry 38 and the segmentation circuitry 36 is provided to a linkage circuitry 40, which applies contextualization operators as necessary and provides the knowledge base 16 with the various contextual relationships 30 created for the document 14 in question.

[0107] A common feature of many documents 14 to be captured is the presence of an image 50. In some cases, text is present in an in-image label 54 embedded within the image 50. For example, a building photograph may include the building's name inscribed on the facade. This in-image label 54 is therefore useful in answering a question 20 about that building. Thus, the visual parser 30 ensures that a contextual relationship 34 exists between the visual segment 26 containing the in-image label 54 and the visual segment 28 containing the image 50.

[0108] A similar phenomenon occurs in connection with figures and tables within document 14; for example, a visual segment 24 tagged as "table" may include a separate caption that appears, at least visually, to be within and part of the table. However, it is possible and advantageous to separately tag such visual segments 24 as, for example, "captions." This also occurs with figures, where visual segments 24 can be separately tagged as "abscissas" or "ordinates." This low level of granularity in tagging visual segments 24 provides great flexibility in defining contextual relationships 30 between visual segments 24.

[0109] This flexibility leads to increased precision: for example, if in response to question 20 it turns out that an abscissa on the graph is relevant, but the rest of the graph is not, it becomes possible to pinpoint that abscissa without having to bring up the entire graph.

[0110] In other cases, image 50 is a composite image that happens to also contain text. An example is image 50 of FIG. 3. Such images 50 are often found in slide presentations. Often, it is actually the text in the in-image labels 54 that is most relevant. For example, within image 50, the non-text material is primarily decorative.

[0111] A conventional ORC circuit would ignore this text because it is actually part of the image and therefore not strictly speaking "text." In a preferred embodiment, the OCR circuit 62 of the interpretation circuit is a hybrid OCR circuit that performs optical character recognition on the text embedded within the image. For example, if image 50 includes a photograph with road signs, hybrid OCR circuit 62 will recognize the text on the road signs.

[0112] To further promote accuracy when decoding images of document 14 using interpretation circuitry 38, it is useful to take into account both the high-level language and underlying influences that may adversely affect the quality of image preprocessing. For example, defects in an image may not be global, so it is preferable for interpretation circuitry 38 to limit the steps taken to repair defects to only those regions requiring repair. Thus, when applying procedures such as image denoising, dilation / erosion, and binarization, interpretation circuitry 38 applies them only locally. This is particularly useful because noise and background tend to vary locally and binarization requires adapting thresholds depending on the nature of the region.

[0113] The ability to perform optical character recognition seamlessly across different fonts is particularly important because different types of visual segments 24 can often be distinguished by differences in font. For example, headers tend to be in bolder and larger fonts than body text. Because excessively bold or thin characters can adversely affect recognition accuracy, it is useful for the interpretation circuitry 38 to perform steps, such as dilation or erosion steps, to compensate for such difficulties.

[0114] Further performance improvements arise from taking text density into account. In some cases, the segmentation circuit 36 recognizes regions of sparse text, for example within figures and tables. To improve the interpretation circuit's ability to recognize characters, it is useful to segment regions of high text density, such as paragraphs 58 or lists 42 within a table or figure.

[0115] Although text regions within a diagram are often sparse in text, their visual relationship to the diagram, i.e., their actual presence within the diagram, gives them undue importance in answering a question involving the diagram, especially when such sparse text regions are identified as having a contextual relationship with another visual segment 24. Establishing a contextual relationship 30 between a sparse region and another visual segment 24 allows for retrieval of the text in the sparse region for use in answering a question 20.

[0116] Additionally, the text in the sparse region can be used to formulate new questions that can be used to query the knowledge base 16 to identify further information that may be relevant to responding to the user's original question 20.

[0117] The different semantic units 24 into which the document 14 is segmented can also be used to improve the accuracy with which the interpretation circuitry 38 can recognize characters.

[0118] For example, in some cases, document 14 has visual segments 24 with dark backgrounds that interfere with character recognition. Because visual segments 24 have been identified and the spatial regions they occupy in document 14 are known, it is possible to lighten the dark backgrounds before optical character recognition operations are performed.

[0119] More generally, the color of the text and the color of the background on which the text is overlaid have a dramatic effect on optical character recognition, as any human can experience firsthand by simply trying to read light gray text on a slightly darker gray background.

[0120] Once a visual segment 24 has been defined and its bounding box specified, it is possible to analyze the colors present in the bounding box and map those colors to colors tailored to the capabilities of the optical character recognition process. If, for some reason, the author of document 14 wants the text on a slightly dark blue background to be dark blue, then "dark blue" can be mapped to "black" and "slightly dark blue" can be mapped to "white," thereby making it easier for optical character recognition to be performed.

[0121] In general, there is a source color space and a target color space. The source color space is the set of colors within the visual segment 24 in question. The target color space is defined by whatever the optical character recognition system considers optimal for character recognition. The problem is therefore to construct an optimal mapping from the source color space to the target color space.

[0122] This can be done by identifying a distribution of pixel values in the source color space and mapping them to corresponding colors or intensities in the target color space. In some embodiments, the target color space is a binary color space, allowing only two colors. Among these are embodiments where the two colors are black and white. In such cases, the mapping is a many-to-one mapping.

[0123] The presence of visual segments 24 also allows for certain preprocessing steps to be performed to enhance the accuracy with which optical character recognition is performed. Examples of such preprocessing steps include rescaling, binarization, noise removal, dilation, erosion, rotation, and deskew.

[0124] The task of optical character recognition can be viewed essentially as a sequence-decoding problem. A useful way to decode a sequence is to use either a set of recurrent networks or a set of transformers. Another approach to such decoding is to use a learning model and train it with the underlying words and the characters that make up the words. In such a training procedure, the semantic tags obtained by the visual parser 30 during the classification step are particularly useful for biasing the language model based on the language most likely to be seen in the visual segments 24 of that class.

[0125] The presence of separate visual segments 24 also allows for improved character recognition by allowing different visual segments 24 to be processed in different orientations.

[0126] For example, it is not uncommon for a visual segment 24 containing a figure caption to have a landscape orientation, and another visual segment 24 containing text along the figure's vertical axis to be rotated 90 degrees relative to the figure caption.

[0127] In some implementations, visual parser 30 accommodates these differences in orientation by repeating the optical character recognition process using four rotations that are separated from each other by multiples of 90 degrees, and doing so for each visual segment 24. In other implementations, visual parser 30 performs an initial segmentation that estimates the orientation of each visual segment 24. This is performed by using a machine learning system that is trained not only to classify visual segments 24 but also to provide an estimate of their orientation.

[0128] At this point, it is useful to make a distinction between the document 14 itself and how the document 14 appears to a viewer, i.e., the rendered version of the document 14. When considering the world of physical documents 14, these are clearly the same. However, documents 14 of the kind considered herein need not be physical documents 14. As a result, the visual appearance of an actual document 14 can differ significantly from its rendered appearance.

[0129] An easily understood example is an HTML file. The actual document 14 contains a mixture of tags and content that is difficult for a human reader to grasp. Furthermore, when rendered by a web browser, the document 14 takes on a distinctly different appearance.

[0130] The segmentation circuit 36 described above performs a segmentation procedure on the document 14 that is useful for visual inspection. One example of such a document 14 is one that results from simply scanning a physical document.

[0131] However, many documents 14 to be captured lack this property. This is particularly common in the case of digitally generated documents 14, where the document 14 amounts to instructions on how to display the document 14 in a visual form that can be understood by a human. Examples of such documents 14 are HTML files, as well as portable document files and many proprietary file formats associated with various applications.

[0132] In many of the above examples, characters and their positions are explicitly identified. However, there is no explicit division into visual segments 24. A similar difficulty arises with certain tagged files, such as HTML files. In such files, tags specify how something looks; they do not specify how it relates to anything else in any semantic way.

[0133] In such cases, a useful way for segmentation circuitry 36 to perform the segmentation procedure is simply to generate an equivalent image from the document 14 and then perform the segmentation procedure on this generated image.

[0134] Optical character recognition can be omitted in this case because the document 14 explicitly has text within it, and therefore the interpretation circuitry 38 relies on the original document 14 rather than on its image.

[0135] The original format of the input document 14 determines the steps that are performed to convert that document 14 to the corresponding captured content. The most common such documents 14 are HTML and PDF files.

[0136] If document 14 is an HTML file, visual parser 30 begins the process by using a virtual headless browser to render a corresponding web page that corresponds to the HTML file. Visual parser 30 then calculates style information for each element on the web page so rendered.

[0137] In the next step, the visual parser 30 augments the document's source HTML to include external style labels. This augmentation step generates an augmented file, which is then downloaded and preprocessed. These steps are performed to ensure validity and preserve inline formatting.

[0138] The next step is the parsing step, which involves parsing the HTML into an element tree and then recursively walking that tree to create a flattened list of higher-level elements. The flattened list is defined according to a combination of hard-coded rules and configurable cascading style sheet selectors.

[0139] The next step is to replace certain container elements with their respective contents. This is done for container elements that do not serve a semantic function by themselves. Completing this step will produce a list of higher-level elements.

[0140] With this list of high-level elements created, a final post-processing step is performed to ensure HTML validity and re-apply any omitted inline formatting. During this step, it is also possible to remove selected elements or to pare down selected elements based on their tags.

[0141] A similar process for portable document files is somewhat simpler. In such cases, visual parser 30 classifies various semantic elements based on their visual appearance. Examples of such classes are headings, tables, and footnotes. Visual parser 30 also applies a semantic tag to each visual segment 24 to indicate its class.

[0142] Certain classes are deemed to have little value for answering the question. Visual segments 24 with semantic tags that identify them as belonging to such classes are weighted low or discarded. An example of such a class is "Table of Contents."

[0143] Certain embodiments of visual parser 30 use a language model to identify any irregularities in the text that are likely to interfere with its comprehension or to identify the use of foreign language, and because such text is not useful in answering the question, the visual parser discards it or weights it low.

[0144] The various captured documents 14, authored by different people, may not use the exact same language to refer to particular concepts, and these small variations may cause the question answering system 10 to miss certain text.

[0145] As an example, to refer to a personal computer, some documents 14 may say "PC" and others may say "PC." To enforce uniformity, it is useful for interpretation circuitry 38 to modify visual segments 24 so that all such references to a common concept use common terminology, i.e., "standard terminology." This process is referred to herein as "standardization" and is also applicable to input questions 20.

[0146] The process performed by visual parser 30 to classify visual segments 24 depends on one or more of two factors: the location of the segment in the visual structure of the document and its text, which is recognized by OCR or explicitly derived from the document.

[0147] In many cases, the content of a document 14 includes instructions for rendering the document 14. Thus, upon inspection of such a document 14, it is not immediately apparent what its visual representation will look like. As a result, visual segments 24 that are easily recognized in a document 14 as being rendered may be difficult to recognize when inspecting the actual content of the document 14.

[0148] An example of a visual segment 24 having the above properties is the confidentiality notice 44 shown in Figure 3. Such a notice limits distribution rights (e.g., "Confidential," "Top Secret," etc.).

[0149] Another visual segment 24 having the above properties is a light gray watermark printed diagonally across each page.

[0150] The above types of visual segments 24 are often highly visible in rendered documents, but are not easily recognized in the actual document 14 before rendering. The segmentation circuitry 36 automatically identifies and classifies such visual segments 24 and uses the generated information to label relevant portions of the document 14 with associated restrictions on distribution. This allows the question and answering system 10 to exclude such information from answers 18 if the user 22 does not need to know it.

[0151] Other examples of visual segments 24 with the above properties include structured sections such as tables of contents 60 and indexes. These visual segments 24 have distinctive visual indicia that are easily discernible upon visual inspection, such as list-like formats, leader lines or dots linking title words, and page numbers 56.

[0152] The semantic content of visual segments 24 classified as indexes or tables of contents is of limited value in formulating answers 18. Therefore, in a preferred embodiment, linkage circuitry 40 avoids visual segments 24 that are tagged as such or that link them, so as to prevent retrieval of content from such visual segments 24. This prevents answers 18 from being found with information from such visual segments 24.

[0153] On the other hand, visual segments 24 classified as a table of contents 60 can be useful when linked to content identified within the table of contents 60 to provide distant context, particularly due to explicit nested headings that can be easily discovered from the rest of the document 14 itself. Accordingly, linkage circuitry 40 maintains contextual relationships 30 for this purpose.

[0154] A useful property of a visual segment is its position relative to other visual segments. The relative positions of visual segments provide a basis for establishing contextual relationships 34 between the visual segments. For example, in many cases, if the segmentation circuitry 36 finds and identifies a first semantic unit as, for example, a chapter or section heading, and identifies a second semantic unit as the text immediately following the heading, the linkage circuitry 40 will link the two visual segments with the contextual relationship 34.

[0155] Another cue relied upon by the segmentation circuit 36 to identify visual segments 24 is the presence of text blocks arranged in visually distinct rows and columns, with headers in each column and in each row. Upon observing such visual characteristics, the segmentation circuit 36 classifies the set of all text blocks as a visual segment of type "table." For convenience in accessing the information, the segmentation circuit 36 also classifies the text blocks within each row or column of this table as separate visual segments of type "table entry." This results in document regions with an overlay of different visual segments.

[0156] In some cases, the contextual relationship 34 between the first visual segment 26 and the second visual segment 28 arises from a combination of visual structure and semantic content.

[0157] In one example, first visual segment 26 includes text of the form "With reference to the graph shown in Figure 1," and second visual segment 28 is tagged as "caption" and begins with the text "Figure 1." In such a case, interpretation circuitry 38 observes this apparent relationship and provides this information to linkage circuitry 40, which then establishes the appropriate contextual relationship 34 between first visual segment 26 and second visual segment 28.

[0158] In many cases, contextual relationships 34 based on semantic content may not be as clear-cut as those described above. To identify more nuanced contextual relationships, interpretation circuitry 38 relies on natural language techniques. In some cases, interpretation circuitry 38 looks for similarities between visual segments 30 that consider a particular graph and similarities between that consideration and the semantic content present in any visual segments 32 tagged as "caption."

[0159] Because question-answering systems 10 are already available, one particularly useful technique for identifying relationships based on semantic content is to construct questions 20 based on text from captions. The generated questions 20 are then presented to the question-answering system 10, which then identifies other visual segments 24 that may be relevant. This technique generates a recursive architecture in which the question-answering system 10 becomes a component of an interpretation circuit 38 that is itself a component of the question-answering system 10.

[0160] The above process of identifying visual segments 24 based on the visual structure of document 14 and establishing contextual relationships 30 between visual segments 24 improves the quality of information presented to user 2 in response to question 20.

[0161] For example, consider a document 14 having first technical information relating to steam turbines and second technical information relating to gas turbines. Assume that the first and second technical information are fragmented across long sections of the document.

[0162] In the above context, it is not uncommon to find a section on "Steam Turbines" and a section on "Gas Turbines" within this document 14, which have similar but contradictory statements, such as "The oil pressure should be maintained between 2 and 3 bars" and "The oil pressure should be maintained between 3 and 4 bars."

[0163] The original author was no doubt writing for an audience, not a question-and-answer system. As a result, the author deliberately did not include in each sentence the type of turbine to which they were referring. After all, it is possible to visually parse Document 14. In doing so, one could easily infer from the context which sentence about steam turbines to rely on.

[0164] The visual parser 30 enables the question-answering system 20 to mimic this unique human ability. The visual parser 30 does so by adding contextual relationships 30 between the visual segments 24 containing each sentence and the corresponding heading. The addition of these contextual relationships 30 makes it possible to resolve ambiguities and preferentially present only correct sentences to the user 22.

[0165] One way that linkage circuitry 40 implements the contextualization operator is to replace the content of second visual segment 28 with content that incorporates information that indicates a contextual relationship 34 with first visual segment 26. The generated content is referred to herein as "composite content."

[0166] In the example described herein regarding turbine oil pressure, the content in the second visual segment 28, "Oil pressure should be maintained between 2 bar and 3 bar," is replaced with the compound content, "Steam turbine / / Oil pressure should be maintained between 2 bar and 3 bar / / ." The prepended text, i.e., "Steam turbine," indicates a contextual relationship 34 in which the first visual segment 26 informs the meaning of the second visual segment 28.

[0167] The above composite content allows question and answering system 10 to recognize that all content between the double slashes should have the meaning conveyed by the indicated contextual relationship 34 to "Steam Turbine." In effect, linkage circuitry 32 is doing what the original author of document 14 might do in response to being told that the target audience cannot see the visual relationship between the text and the heading.

[0168] The linkage circuitry 40 similarly applies contextualization operators to add contextual relations 34 to other types of visual segments 24. For example, if the visual segment 24 is a table, the content of each cell in the table can be replaced with composite content that includes information about the row and column in which the cell is located and information taken from the table caption, if one is available. Similarly, for semantic content tagged as a graph, the linkage circuitry 40 replaces the content of the graph with composite content that includes both information in the original content and information from the graph's axis labels.

[0169] In yet another example, linkage circuitry 40 replaces content from visual segment 24 corresponding to elements in list 42 with composite content that includes the original content along with information from the sentence preceding list 42 .

[0170] In some embodiments, the linkage circuitry 40 adds contextual relationships 62 by using a process other than concatenating contextual information with existing content to create composite content.

[0171] Among these are embodiments in which the linkage circuitry 40 processes a context separate from the text relied upon as the basis for the answer 18. Among these are cases in which the linkage circuitry 40 concatenates embeddings of a question and a corresponding answer.

[0172] In other embodiments, linkage circuitry 40 relies on an attention-based approach in which words in potential answers 18 are transformed based on context to produce a representation of the content that incorporates contextual information.

[0173] So far, visual segments 24 have been static in nature, i.e., text and pictures. However, the principles described herein are applicable when visual segments 24 are dynamic. Examples of such dynamic content include audio and video content. A document 14 containing such dynamic segments is considered a "multimodal document."

[0174] For visual segments 24 that include audio, interpretation circuitry 38 performs a speech recognition process to transcribe the audio content.

[0175] For visual segments 24 that include video content, interpretation circuitry 38 extracts selected frames. For example, in many video files, there are certain frames called "keyframes" that define the beginning and end of particular scenes. To identify such keyframes, interpretation circuitry 38 performs conventional keyframe detection techniques.

[0176] For certain slide presentations, it is not uncommon for audio and video content to be combined. In such cases, it is useful for interpretation circuitry 38 to also synchronize the audio content from first visual segment 26, including the audio, and the video content in second visual segment 28 that corresponds to the audio in first visual segment 26. This is done by establishing a contextual relationship between portions of the audio content in first visual segment 26 and corresponding portions of the video content in second visual segment 28.

[0177] The slide presentation itself often includes a mixture of lists 42, paragraphs 58, and images 50 distributed in two-dimensional space. Thus, the slide presentation is in fact a species of document 14. In that case, there is nothing to stop the visual parser 30 from simply processing the slide presentation as a whole in the same way that it processes a more traditional document 14.

[0178] Thus, the visual parser 30 identifies slide titles, bulleted lists 42, and sequences of images 50 as visual segments that may have contextual relationships 30 with one another, as appropriate. The details of this implementation are the same as those discussed in connection with other types of documents 14.

[0179] As with visual segments 24 that include suitable non-text content, such as images 50, visual segments 24 that have dynamic content may include other visual segments 24 embedded therein. Thus, in some embodiments, the segmentation circuitry 36 identifies the title of a slide in a slide presentation and creates a visual segment 24 that corresponds to the title. Other features that the segmentation circuitry 36 identifies in a slide presentation include bulleted lists 42 within a slide.

[0180] The visual parser 30 identifies contextual relationships 30 between visual segments 24 that are separated by significant distances within a slide presentation. The visual parser 30 does so by observing such visual features within the slide presentation. One example of such a visual feature is a major section heading, which is often in a much larger font or a different font style. In such cases, when one or more slides are between two consecutive section headings, the linkage circuitry 40 creates a contextual relationship 30 between each slide and the immediately preceding heading.

[0181] Multimodal content within a document 14 can be considered to be visual segments 28, and an audio transcript from the multimodal content can be formed, which transcript is itself a visual segment 28 that can be treated much like a caption. In particular, a transcript is to audio what a caption is to an image 50. Thus, the linkage circuitry 40 also forms a contextual relationship 34 between audio content and its corresponding text transcript. Thus, in response to a question 20 that mentions a subject found in the transcript, the question-answering system 10 has a basis for retrieving related audio. Such audio can then be provided to the user 22 in its original audio form or as a transcript thereof.

[0182] As described above, given a slide presentation having both an audio transcript and text embedded in a slide, two contextual relationships 34 can be established: one between the slide and its embedded text, and the other between the same slide and the text transcript of the audio content that accompanies that slide. As a result, for each slide, there will be two visual segments 24 that have a contextual relationship 34 with that slide: one visual segment containing the transcript of the audio narration and one visual segment containing the text embedded in the slide. This provides a degree of redundancy that is useful in providing an answer to question 20.

[0183] For example, given question 20, which refers to the subject of the slide, it is entirely possible that the narration omitted any reference to specific text within the slide. This is not uncommon, as a good presenter will do more than simply read the slide.

[0184] Nevertheless, question and answering system 10 is able to match the text in that question 20 with the text in visual segment 26 that contains the text embedded in the slide. Because a contextual relationship 34 exists between that visual segment 26 and visual segment 28 that contains the slide, question and answering system 10 now has a basis for utilizing the slide itself to answer question 20.

[0185] Conversely, it is entirely possible that a presenter, while talking about a particular slide, utters words that are not on the slide at all. If those words match words in question 20, question and answering system 10 then matches question 20 with the audio transcript. Again, a contextual relationship 34 will exist between visual segment 28 containing the audio transcript and visual segment 26 containing the slide, so question and answering system 10 will again have a basis for using the slide itself to answer question 20.

[0186] Embodiments of the techniques detailed above may be implemented in software, with computer instructions stored on a non-transitory machine-readable medium. These instructions, when executed by one or more processors, perform the functions described above. The instructions may be at various levels, from machine-level instructions to instructions for configuring an artificial intelligence system.

[0187] While specific embodiments have been disclosed herein, this is done by way of example for illustrative purposes only and is not intended to limit the scope of the invention, which is defined by the appended claims. Any features of the disclosed embodiments described herein can be combined, rearranged, etc. with one another to create many more embodiments within the scope of the invention. Certain other aspects, advantages, and modifications are considered to be within the scope of the claims provided below. The presented claims represent at least some of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated.

Claims

1. 1. A method comprising: Building a knowledge base (16) for a natural language question answering system (10) Including, Building the knowledge base of the natural language question answering system includes: capturing a document (14) including first, second, and third visual segments, each including semantic content; preprocessing the documents to enable the question-answering system to access the semantic content in response to natural language questions (20) posed by users (22) of the natural language question-answering system; Including, Preprocessing the document includes: extracting structural information from the document using the visual structure of the document; incorporating said structural information into said knowledge base; Including, The method, wherein the visual structure of the document comprises a spatial distribution of the visual segments in the document when the document is in a form that visualizes the document.

2. extracting structural information from the document using the visual structure, determining, based at least in part on positions of the first and second visual segments relative to each other visual segment in the document, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on positions of the first and third visual segments relative to each other visual segment in the document, that a contextual relationship should not exist between the first visual segment and the third visual segment; establishing the contextual relationship between the first visual segment and the second visual segment; The method of claim 1 , comprising:

3. extracting structural information from the document using the visual structure, determining, based at least in part on a distance between the first visual segment and the second visual segment, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on a distance between the first visual segment and the third visual segment, that a contextual relationship should not exist between the first visual segment and the third visual segment; establishing the contextual relationship between the first visual segment and the second visual segment; The method of claim 1 , comprising:

4. The method of claim 1 , further comprising using semantic information in addition to structural information to determine whether a semantic relationship should exist between a pair of visual segments.

5. determining, based at least in part on a reference in the first visual segment to the second visual segment, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on the absence of a reference in the first visual segment to the third visual segment, that a contextual relationship should not exist between the first visual segment and the third visual segment; establishing the contextual relationship between the first visual segment and the second visual segment; The method of claim 1 further comprising:

6. extracting structural information from the document using the visual structure, classifying the visual segments into different classes based on visual appearance differences of the visual segments; determining, based at least in part on classes of the first and second visual segments, that a contextual relationship should exist between the first semantic segment and the second semantic segment; determining, based at least in part on classes of the first and third visual segments, that a contextual relationship should not exist between the first semantic segment and the second semantic segment; establishing the contextual relationship between the first visual segment and the second visual segment; The method of claim 1 , comprising:

7. extracting structural information from the document using the visual structure, determining, based at least in part on positions of the first and second visual segments relative to each other visual segment in the document and semantic content of the first and second visual segments, that a contextual relationship should exist between the first visual segment and the second visual segment; determining, based at least in part on positions of the first and third visual segments relative to each other visual segment in the document and semantic content of the first and second visual segments, that a contextual relationship should not exist between the first visual segment and the third visual segment; establishing the contextual relationship between the first visual segment and the second visual segment; The method of claim 1 , comprising:

8. 8. The method of claim 1, wherein extracting structural information from the document using the visual structure comprises establishing a contextual relationship between the first visual segment and the second visual segment based on determining that the first visual segment is a figure and the second visual segment is a caption for the figure.

9. 8. The method of claim 1, wherein extracting structural information from the document using the visual structure comprises establishing a contextual relationship between the first visual segment and the second visual segment based on determining that the first and second visual segments have a common location.

10. 8. The method of claim 1, wherein extracting structural information from the document using the visual structure comprises establishing a contextual relationship between the first visual segment and the second visual segment based on determining that the first visual segment is a diagram and the second visual segment is text overlaid on the diagram.

11. The method of any one of claims 1 to 7, wherein the document includes instructions for visualizing the document, the method further comprising identifying the visual structure based on the instructions.

12. 8. The method of claim 1, wherein the document is in a portable document format, the method further comprising determining the visual structure of the document based on rendering instructions expressed in the portable document format.

13. The method of any one of claims 1 to 7, wherein the document is an HTML file, the method further comprising identifying the visual structure of the document based on tags in the HTML file.

14. 8. The method of claim 1, further comprising: determining that a first one of the visual segments includes text arranged in a matrix; determining that a second one of the visual segments includes text at an intersection of one of the rows and one of the columns; and establishing a contextual relationship between the first visual segment and the second visual segment based at least in part on the determining.

15. 8. The method of claim 1, wherein extracting structural information from the document using the visual structure comprises tagging at least one of the visual segments as belonging to a respective class in a set of classes, the set comprising lists, paragraphs, headings, and images.

16. 8. The method of claim 1, wherein the first visual segment includes audio and the second visual segment includes a transcript of the audio, the method further comprising establishing a contextual relationship between the first visual segment and the second visual segment.

17. 8. The method of claim 1, wherein the first visual segment includes audio and the second visual segment includes video, and the method further comprises establishing a contextual relationship between the first visual segment and the second visual segment to synchronize the audio and the video.

18. A natural language question-answering system for providing answers to questions from users, the natural language question-answering system comprising: an ingestor for ingesting documents, each of the documents having a visual structure, when visualized, in which visual segments each containing semantic content are spatially distributed throughout the document; a visual parser for extracting structural information from the document based at least in part on the visual structure; a knowledge base incorporating semantic content from the documents and the structural information; Natural language question answering systems, including:

19. The visual parser a segmentation circuit configured to identify the visual segments and classify the visual segments into classes; an interpretation circuit for extracting semantic information from the visual segments; a linkage circuit that establishes a contextual relationship between a pair of selected visual segments based on inputs from the segmentation circuit and the interpretation circuit; 20. The natural language question and answering system of claim 18, comprising:

20. A non-transitory machine-readable medium having stored thereon instructions which, when executed by a data processing system, cause said system to perform all the steps of any one of claims 1 to 17.

Images