Method and system for extracting relevant information from images and computer program product

Abstract

The present invention relates to a method and system for extracting relevant information from images and computer program product. Various methods use SQL based data extraction to extract relevant information from images. These are rule based methods which generate SQL queries from NL and require human intervention if any new English sentence is to be processed. This becomes more difficult for non technical users. A system and method for extracting relevant information from images using a conversational interface and database queries is provided. The system eliminates the effect of noise, identifies the type of document and detects various entities of the chart. Further, a schema is designed which makes the abstraction of entities detected by the deep vision model and the relationships between the entities easy to understand. Then, relevant information and fields can be extracted from the document by writing SQL queries on top of the relationship table. A natural language based interface is added so that non technical users who specify queries in natural language can easily get the information.

Description

[0001] Cross-reference and priority of related applications

[0002] This patent application claims priority to Indian patent application 201821045427, filed on November 30, 2018. Technical Field

[0003] The embodiments described herein generally relate to the field of extracting data from images. More specifically, but not particularly, the present invention provides an end-to-end system and method for extracting relevant information from document images using a dialog interface and database queries. Background Technology

[0004] Recent advances in computer vision, utilizing state-of-the-art neural networks, have boosted the accuracy of Optical Character Recognition (OCR). However, simply extracting characters / text is often insufficient, as documents also possess visual structures that OCR cannot capture. Extracting information from tables, charts, footnotes, boxes, and headings, and retrieving corresponding structured representations of documents, remains a challenge and is applied in numerous real-world use cases.

[0005] While OCR accuracy has significantly improved due to advancements in deep learning, this alone is insufficient for effectively extracting information from scanned documents. Most documents possess rich visual structures in the form of tables, text boxes, blocks, charts, and graphs. The visual relationships between different entities present in an image are crucial for understanding and reasoning about the information present before extraction. Furthermore, the often integrated nature of domain knowledge or business rules can frequently enhance extraction performance and enable the verification and correction of extracted data.

[0006] Extracting text from images has been an active area of ​​research for decades. Thanks to these advances, interpreting documents with relatively simple text layouts and high-quality scans is now quite straightforward. However, retrieving semantically accurate information can be very challenging when dealing with documents that follow several different templates with diverse visual layouts. Much work has been done to address this problem.

[0007] The research group has developed an information extraction system in which a document instance is matched against a set of pre-stored models defining document categories. The extracted document text is compared with a pre-compiled list of keywords and their spelling variations. The results are then matched against each document in a word model and its logical relation database. Another approach requires users to build a conceptual model of the document, which is then used to match, interpret, and extract content from the document. This work places more emphasis on the classification and organization of documents than on the extraction of key fields. These existing techniques construct document representations based on attribute relationship graphs to capture the relationships between entities in an image; however, their systems rely on considerable expertise from the user to create suitable template document models.

[0008] Current methods for information extraction require programming at the pixel coordinate level and are not abstract enough for rapid deployment by novice users. Furthermore, most existing methods generate SQL queries from NL using hand-created rules. If any new English sentences are to be processed, manual intervention is required, such as indexing new data and updating dictionaries, metadata, etc. Summary of the Invention

[0009] The following is a simplified overview of some embodiments of this disclosure to provide a basic understanding of the embodiments. This overview is not a comprehensive overview of the embodiments. It is not intended to identify key / deterministic elements of the embodiments or to depict the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description presented below.

[0010] In view of the foregoing, embodiments of this document provide a system for extracting relevant information from an image. The system includes an input module, a memory, and a processor. The input module provides an image as input. The processor further includes a noise removal module, a document identifier, an optical character recognition (OCR) engine, a pattern design module, a storage module, a dialog interface, a conversion module, and a database query module. The noise removal module performs a denoising method on the input image to remove noise. The document identifier identifies the type of document, wherein the type is printed text or handwritten text. The OCR engine performs optical character recognition (OCR) on the image according to the identified document type to detect a set of text entities. The pattern design module designs a pattern to establish relationships between the set of text entities. The storage module stores the detected set of text entities and the pattern in a relational database. The dialog interface extracts relevant information by providing a natural language query from the user. The conversion module converts the natural language query into an SQL query. The database query module provides an SQL query to the relational database to extract relevant information from the relational database.

[0011] On the other hand, the embodiments of this paper provide a method for extracting relevant information from an image. Initially, an image is obtained as the input image. In the next step, a denoising method is performed on the input image to remove noise. Further, a document identifier is run to identify the document type, where the type is printed text or handwritten text. In the next step, optical character recognition (OCR) is performed on the image according to the identified document type to detect a set of text entities. A pattern is then designed to establish relationships between these text entities. The detected set of text entities and the pattern are then stored in a relational database. In the next step, a natural language query is provided by a user using a dialog interface to extract relevant information. The natural language query is then converted into an SQL query. Finally, the SQL query is provided to the relational database to extract relevant information from it.

[0012] Those skilled in the art will understand that any block diagram herein represents a conceptual diagram of an illustrative system embodying the principles of the subject matter. Similarly, it will be understood that any flowchart, diagram, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a computer-readable medium and executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

[0013] In another embodiment, one or more non-transitory machine-readable information storage media are provided, including one or more instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform a method for designing one or more unified architectural models for building digital products and digital services. The method includes processor-implemented steps: obtaining an image as an input image; performing a denoising method on the input image to remove noise from the input image; running a document identifier to identify the type of document, wherein the type is printed text or handwritten text; performing optical character recognition (OCR) on the image according to the identified document type to detect a set of text entities; designing a pattern to establish relationships between the set of text entities; storing the detected set of text entities and the pattern in a relational database; providing a natural language query by a user through a dialog interface to extract relevant information; converting the natural language query into an SQL query; and providing the SQL query to the relational database to extract relevant information from the relational database.

[0014] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory, and not intended to limit the scope of the claimed invention. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this disclosure, illustrate exemplary embodiments and, together with the specification, serve to explain the principles disclosed.

[0016] Figure 1 A block diagram of a system for extracting relevant information from an image using a dialog interface and database queries, according to an embodiment of the present disclosure, is shown.

[0017] Figure 2 The workflow of a system for writing rules for queries according to an embodiment of the present disclosure is shown;

[0018] Figure 3 An architecture of a cGAN for denoising images according to an embodiment of the present disclosure is shown;

[0019] Figure 4 A convolutional neural network with a recursive encoder-decoder model according to an embodiment of the present disclosure is shown;

[0020] Figures 5A to 5B This is a flowchart illustrating the steps involved in extracting relevant information from an image using a dialog interface and database queries according to embodiments of the present disclosure;

[0021] Figure 6 An example of a depth reader mode according to an embodiment of this disclosure is shown; and

[0022] Figure 7 An example of a sample library document for information extraction according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Exemplary embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numerals of the reference numerals identify the figure in which the reference numeral first appears. The same reference numerals are used throughout the drawings to denote the same or similar parts, wherever convenient. While examples and features of the disclosed principles have been described herein, modifications, alterations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered exemplary only, while the true scope and spirit are indicated by the appended claims.

[0024] Now refer to the attached diagram, and more specifically to... Figures 1 to 7 Throughout the accompanying drawings, similar reference characters always denote corresponding features, illustrating preferred embodiments, which are described in the following section on exemplary systems and / or methods.

[0025] According to embodiments of this disclosure, in Figure 1The block diagram illustrates a system 100 for extracting relevant information from images using a dialog interface and database queries. System 100 utilizes the spatial relationships between different visual entities present in the image, transforming these spatial relationships into a relational table that can be queried via SQL or natural language queries. It achieves this by leveraging state-of-the-art deep learning models for visual entity extraction and taking advantage of recent advances in deep learning for NLP and program synthesis.

[0026] System 100 uses a framework also known as a "deep reader framework." The workflow of the deep reader framework is as follows: Figure 2 As shown, this framework facilitates information extraction from document images by recognizing pattern-based entities from the document and maintaining a meta-relation model across different entities in the document image. The pattern model makes the abstractions of entities detected by the deep vision model and the relationships between them easily understandable. System 100 runs on vision algorithms applied to recognize combinations of handwritten and printed text, eliminate noise effects, identify document types, and detect objects / rows / boxes in the graph. System 100 further maps the extracted entities to rich relational schemas to capture all relevant relationships between entities (words, text boxes, lines, etc.) detected in the document. Relevant information and fields can then be extracted from the document by writing SQL queries at the top of the relational tables. A natural language-based interface is also added at the top of the relational schemas, allowing non-technical users to easily access information by specifying queries in natural language.

[0027] According to embodiments of this disclosure, system 100 further includes an input module 102, a memory 104, and a processor 106, such as Figure 1 The block diagram is shown. Processor 106 communicates with memory 104. Processor 106 also includes multiple modules. The multiple modules access the set of algorithms stored in memory 104 to perform specific functions. Processor 106 also includes a noise removal module 108, a document identifier 110, an optical character recognition (OCR) engine 112, a pattern design module 114, a storage module 116, a dialog interface 118, a conversion module 120, and a database query module 122.

[0028] According to embodiments of this disclosure, input module 102 is configured to provide an image as an input image to system 100. The image can be captured from a document or any other source. Input module 102 may include various software and hardware interfaces, such as a web interface, a graphical user interface, etc., and can facilitate various communications within a wide range of network N / W and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, or satellite).

[0029] like Figure 2As shown in the workflow, system 100 processes documents in several stages. According to an embodiment of this disclosure, system 100 includes a noise removal module 108. The noise removal module 108 is configured to perform a denoising method on an input image to remove noise from the input image. Typically, an original document image that may be blurry, noisy, or faded is provided as input to system 100. The noise removal module 108 is used to clean the image before data extraction.

[0030] Image quality degradation can occur due to various reasons, such as camera shake, improper focus, imaging noise, coffee stains, wrinkles, low resolution, poor lighting, or reflections. These types of problems significantly impact the performance of many computer vision algorithms, such as text detection, OCR, and localization. The goal of this paper is to reconstruct high-quality images directly from noisy input while preserving the highly structured data within the image. Text document images differ significantly from natural scene images because text documents contain more detailed information and are therefore more sensitive to noise.

[0031] A denoising method is performed using a Generative Adversarial Network (GAN). System 100 uses a Conditional Generative Adversarial Network (cGAN), where both the generator and discriminator are conditional on additional information y. In this disclosure, the variable y is represented by a class label (i.e., a clean image). By feeding y as an additional input layer, only the discriminator is conditionally processed. The cGAN network is trained using the following min-max objective function:

[0032]

[0033] Among them, P r It is the data distribution, and P g It is by z = P(z) and the model distribution defined by z, one of the samples comes from a noisy image dataset.

[0034] Figure 3 The architecture of a cGAN for image denoising is shown. The figure illustrates the generator and discriminator models. The generator network consists of a combination of convolutional and deconvolutional layers (indistinguishable in the figure). Each block is followed by batch normalization, and the activation used in the convolutional layers is PReLU, while ReLU activation is used for the deconvolutional layers. The stride is set to 1 for all layers. Zero-padding is used where appropriate to keep the dimension of each feature map the same as the input dimension. Symmetric skip connections are used for efficient network training and better convergence performance, as shown in the generator network.

[0035] According to embodiments of this disclosure, processor 106 further includes a document identifier 110. The document identifier 110 is configured to identify the type of an input image. The type of the input image can be printed text or handwritten text. A deep Siamese network is used for document recognition. The basis of the Siamese network includes a conventional convolutional neural network (CNN).

[0036] Based on the identifier 110, if the output is handwritten text, further processing is required for the scanned document used to digitize the handwriting. In this embodiment, system 100 uses a handwritten text recognition (HTR) engine 124, which employs a convolutional feature extractor followed by a recursive encoder-decoder model that maps visual features to a set of characters present in the image. Figure 4 A general overview of the model is provided.

[0037] The model comprises two steps: visual feature extraction and sequence learning. It employs a 7-layer fully convolutional network that extracts dense feature maps from images of text sequences, words, or lines. These feature maps are then processed by specially designed map2seq layers to generate a sequence of feature vectors, each encoding the local properties of individual characters. The complete sequence captures their interdependencies, thus providing a comprehensive visual representation of the text present in the image.

[0038] The visual feature sequence is then fed into an encoder RNN that generates a background vector representing the key points of the visual feature sequence. The encoder RNN consists of bidirectional LSTM units that capture correlations along the forward and backward directions. A second decoder LSTM uses this background vector to output a character sequence, where each character is conditioned on the background vector and the previous output. Layer normalization is introduced to stabilize the RNN activations, and dropout is added to regularize the model. The encoder-decoder model also uses an attention mechanism to further enhance decoding capabilities for long sequences.

[0039] The model is trained using focus loss, an improvement over standard classification cross-entropy loss, as it addresses the inherent class imbalance problem in text recognition. During inference, a beam search algorithm is employed, which searches for the best possible output sequence over a set of hypotheses by maximizing the joint distribution over all elements of the sequence. The pre-trained model is applied to scanned images of standard documents. The evaluation metrics are word error rate (WER) and character error rate (CER). The model achieves a WER of 14.2% and a CER of 8.7%.

[0040] According to embodiments of this disclosure, processor 106 further includes an optical character recognition (OCR) engine 112. OCR engine 112 is configured to perform optical character recognition (OCR) on the input image based on the type of the document being identified to detect a set of text entities. Documents have many entities, such as words, lines, text blocks and text boxes, checkboxes, arrows, OMR circles, logos, tables, and graphics. The information of interest may relate to one or more of these entities, making it necessary to detect all entities present in the document.

[0041] OCR engine 112 extracts information based on the text type. For page lines, horizontal clustering is performed based on the Euclidean distance between connected component clusters. The page lines are then localized, and each page line patch is sent via a standard OCR engine (e.g., Tesseract, Google Vision, or Abby FineReader) to obtain the text corresponding to that line. Note that sending smaller patches results in higher accuracy from the OCR engine.

[0042] Similarly, a text block is a set of lines that begin at approximately the same x-coordinate and whose vertical distance does not exceed twice the height of the line. Lines located within a specific text block or box are also identified as block or box lines, respectively. For boxes, the image is first eroded, then thresholded and inverted. Afterward, the area of ​​each connected component is compared to the area of ​​its bounding box. If the area of ​​a connected component is within a percentage of the bounding box area, the connected component is considered a box.

[0043] According to embodiments of this disclosure, processor 106 further includes a pattern design module 114 and a storage module 116. Pattern design module 114 is configured to design patterns to establish relationships between the set of text entities. The designed pattern is a universal pattern applicable to all documents. The database pattern remains unchanged. The pattern can be used for any type of document. Furthermore, storage module 116 stores the detected set of text entities and the pattern in relational database 126. The pattern is designed to facilitate information extraction. The set of text entities is associated with the spatial coordinates of the entities, and this information conveys the positions of adjacent text entities. This information is then used to infer different logical and spatial relationships.

[0044] Once the relational schema is populated with data from the documents, it can now be queried just like a regular database schema using SQL. According to an embodiment of this disclosure, processor 106 includes a database query module 122. Database query module 122 is configured to provide SQL queries to relational database 126 to extract relevant information from relational database 126.

[0045] According to embodiments of this disclosure, system 100 can also be accessed by users who are not proficient in SQL. Processor 106 also includes a dialog interface 118 and a conversion module 120. The dialog interface 118 is configured to extract relevant information from natural language queries provided by the user. Any user can provide a natural language query, regardless of their knowledge of SQL. Conversion module 120 is configured to convert the natural language query into an SQL query. A sequence-to-sequence model is used to transform the natural language query into SQL. A deep reader builds upon this work, providing the user with the dialog interface 118 that internally maps natural language statements to SQL queries and extracts relevant information.

[0046] While sometimes a simple SQL query is sufficient to retrieve the required information from a relational database, many times complex queries, or a series of simple queries, must be executed in a pipeline to obtain the correct information. These complex queries can be broken down into multiple simple queries, intermediate results stored, and built upon. These intermediate results are displayed visually so that users can further interpret and write better queries. To achieve this, the intermediate results are saved and fed as a data table into subsequent queries. In this way, a series of queries are executed, resulting in a workflow that can be saved in a single operation and applied to other similar documents to obtain the desired information. This will be explained in more detail later in this disclosure with the aid of examples.

[0047] According to an embodiment of this disclosure, system 100 also includes an intent identifier 128. The intent identifier 128 is configured to identify intents by classifying natural language queries. When a user interacts with system 100 via natural language, the user can request various different information. The intent identifier 128 is necessary for classifying NL queries and determining the intent. Given NL statements are categorized into three types: simple retrieval queries, workflow creation or saving, and bookkeeping. Once the intent is known, the NL query is passed to its corresponding model for further processing.

[0048] According to embodiments of this disclosure, system 100 is also configured to create workflows. Simple queries will only retrieve information easily obtainable from relational database 126. Complex queries are typically required to extract relevant information from images. Designing complex queries for every possible use case would expand the SQL template space and inhibit query reuse. However, complex queries can be broken down into multiple simple queries, allowing for the storage of intermediate results and building upon them. Different combinations of simple queries executed sequentially will yield complex entities. By default, the output of intermediate results is stored in a temporary table that can be further queried.

[0049] According to embodiments of this disclosure, system 100 can also maintain bookkeeping. A series of meaningful NL queries will retrieve the required information. This workflow can be applied to new documents with similar templates. Therefore, storing workflows to retrieve specific entities is of great value. A simple framework using queues has been established to store the most recent set of commands sequentially. Once the user has retrieved meaningful information, the workflow is saved. Simple NL queries such as "clear workflow," "save workflow," and "apply workflow to the document" are used for bookkeeping.

[0050] In operation, Figures 5A to 5B The diagram illustrates a flowchart 200 for a method to extract relevant information from an image. Initially, in step 202, an image is obtained as the input image. This input image can be obtained from any scanned document. In the next step 204, denoising is performed on the input image to remove noise. Various reasons can cause the image to become noisy. In the next step 206, a document identifier 110 is run to identify the document type, which is either printed text or handwritten text. If it is handwritten text, the input image is further processed using an HTR engine 124.

[0051] In the next step 208, optical character recognition (OCR) is performed on the image according to the type of the identified document to detect the set of text entities. This set of text entities may include one or more of the following: words, lines, text blocks or text boxes, checkboxes, arrows, OMR circles, logos, tables, figures, etc. In the next step 210, a pattern is designed to establish relationships between the set of text entities. This pattern is designed to facilitate information extraction. The set of text entities is associated with the spatial coordinates of the entities, and this information conveys the positions of adjacent text entities. This information is then used to infer different logical and spatial relationships. Furthermore, in step 212, the detected set of text entities and patterns are stored in relational database 126.

[0052] In the next step 214, the user provides a natural language query using the dialog interface 118 to extract relevant information. In step 216, the natural language query is converted into an SQL query. Finally, in step 218, the SQL query is provided to the relational database 126 to extract relevant information from the relational database 126. It should be understood that, in another embodiment, the SQL query can be provided directly to the relational database.

[0053] According to the embodiments of this disclosure, system 100 can also be explained using auxiliary examples.

[0054] Figure 6This diagram illustrates an example of a schema populated in a relational database after relevant relationships have been extracted from the original image. The main entities in the schema include words, lines, text blocks, boxes, and tables. Relationships between and within entities are indicated by the direction of the arrows. As the schema can become richer over time, this diagram highlights the entities currently useful for scanning the document image.

[0055] The line entities identified from the visual components include information about the line text, the individual words within the line, and the coordinates of the line and words. Using the coordinates of the words, the deep reader extracts the words to the left / right / above / below of other words and preserves those words in the pattern. Similarly, the deep reader uses the raw visual data to preserve the words and lines appearing in the pattern as separate tables. Figure 7 As shown, the word "Sripur" and its associated coordinates will be detected by the visual component. The word on the left is "Jilashara," the word on the right is "null," the word above is "comprehensive," and the word below is "Bangladesh." In this way, the deep reader maintains the relative spatial position of each word / line / block, etc. The identified text block entities include attributes in a similar manner. The line ID, word ID, word text, line text, and coordinates of each word and line in the text block, as well as the text of each line identified in the block, are populated in the database. Additional relationships, such as lines below / above words in the text block, are also maintained. For example, the relationship of lines below words can be used to extract "ticket recipient" from a document. The line below the word "ticket recipient" in the text block is "ABCD Private Limited."

[0056] It has been observed that most documents in the real world, such as bank receipts and invoices, share certain common patterns, and these patterns can be used for extraction. For example, colon-separated values ​​define key-value pairs. This background knowledge has been used to directly extract key-value pair relationships. Figure 7 As shown, "SWIFT:XABCLUS33" is an instance of a key-value pair. Using the key SWIFT, the value of that key can be directly retrieved by querying the relation table.

[0057] Intent identifiers can be categorized using the following example. Once the intent identifier categorizes a given NL query as a simple query, the NL query is sent to the model for further processing. A typical SQL query structure is as follows:

[0058] SELECT $SELECT COL FROM $TABLE

[0059] WHERE $COND COL $OP $COND VAL

[0060] Therefore, to map an NL statement to an SQL query, perform the following steps:

[0061] Entity Recognition: As shown above, $COND_VAL is a document-specific entity, and it must be extracted from the NL sentence. This corresponds to the Named Entity Recognition (NER) problem, and here a Conditional Random Field (CRF) is used to solve this extraction problem. Each word in the NL query is labeled as either an entity or a non-entity, and the CRF is trained on this encoding. Once $COND_VAL is obtained, the CRF is used to replace $COND_VAL with standard words from the dictionary. For example, in the sentence "Please give me the word to the right of SWIFT", "SWIFT" will be recognized as an entity by the CRF and will be replaced with "Please give me..."<COND VAL> The words on the right. This will help subsequent models in the pipeline process NL queries.

[0062] Template Mapping: A template-based approach is used to generate SQL queries and formulate them as a slot-filling problem. All simple NL queries are mapped to a template in the template pool. Furthermore, this is formulated as a classification problem, and the modified NL queries are classified using a deep sequence model. Below are some sample SQL templates used:

[0063] SELECT * FROM TABLE

[0064] WHERE id=(SELECT id FROM TABLE WHERE string="VALUE")

[0065] SELECT*FROM TABLE WHERE primary str="VALUE"

[0066] SELECT SUBSTR(line,pos(VALUE),)FROM TEMP

[0067] SELECT SUBSTR(line,pos(VALUE1),pos(VALUE2)-pos(VALUE1))FROM

[0068] TEMP

[0069] Table Mapping: Once the correct template is identified, slots for TABLE and VALUE need to be populated. VALUE is easily obtained from the entity recognition model. NL queries have many linguistic variants that can be mapped to the relevant table. For example, the sentences "Give me the word to the right of SWIFT" and "Give me the word next to SWIFT immediately" will map to the same table "rightof". This mapping is accomplished by classifying these variants using a trained LSTM model.

[0070] The written description outlines the subject matter of this document to enable those skilled in the art to make and use embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that may occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they include equivalent elements that are not substantially different from the literal language of the claims.

[0071] The embodiments disclosed herein address the difficulty faced by non-technical users in efficiently extracting data from images using a system. This disclosure provides a method and system for extracting relevant information from images using a dialog interface and database queries.

[0072] It should be understood that the scope of protection extends beyond the computer-readable means containing messages to include programs; such computer-readable storage means contains program code means for implementing one or more steps of the method when the program is run on a server or mobile device or any suitable programmable means. The hardware means can be any type of programmable means, including, for example, any type of computer, such as a server or personal computer, or any combination thereof. The means may also include means that are, for example, hardware means (e.g., application-specific integrated circuits (ASICs)), field-programmable gate arrays (FPGAs), or a combination of hardware and software means (e.g., ASICs and FPGAs), or at least one memory in which at least one microprocessor and a software module reside. Therefore, the means may include both hardware and software means. The method embodiments described herein can be implemented in both hardware and software. The means may also include software means. Alternatively, the embodiments may be implemented on different hardware means (e.g., using multiple CPUs).

[0073] The embodiments described herein may include hardware and software elements. Software implementations include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this specification, a computer-usable or computer-readable medium may be any device, including any device that can contain, store, communicate, propagate, or transmit a program used by or in connection with an instruction execution system, device, or apparatus.

[0074] The steps shown are described to explain the exemplary embodiments illustrated, and it should be expected that ongoing technological developments will change the way particular functions are performed. The examples presented herein are for illustrative purposes and not for limitation. Furthermore, for ease of description, the boundaries of functional building blocks have been arbitrarily defined herein. Alternative boundaries can be defined as long as the specified functions and their relationships are properly performed. Based on the teachings contained herein, alternatives (including equivalents, extensions, variations, deviations, etc., described herein) will be apparent to those skilled in the art. These alternatives fall within the scope and spirit of the disclosed embodiments. Moreover, the words “comprising,” “having,” “including,” and “containing,” and other similar forms are equivalent in meaning and are open-ended, as one or more items following any of these words do not imply an exhaustive list of that item or the plurality of items, or that they are limited to only the listed one or more items. It must also be noted that, unless otherwise expressly indicated, the singular forms “a,” “an,” and “the” as used herein and in the appended claims also include plural references.

[0075] Furthermore, one or more computer-readable storage media can be used to implement embodiments consistent with this disclosure. A computer-readable storage medium refers to any type of physical memory on which processor-readable information or data can be stored. Therefore, a computer-readable storage medium can store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, i.e., it is non-transient. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disk drives, CD-ROMs, DVDs, flash drives, magnetic disks, and any other known physical storage media.

[0076] This disclosure and examples are to be considered exemplary only, and the true scope and spirit of the disclosed embodiments are indicated by the appended claims.

Claims

1. A method (200) for extracting relevant information from an image, the method comprising processor-implemented steps of: obtaining an image as an input image (202); performing a denoising method on the input image to remove noise from the input image (204), wherein, the denoising method is performed by utilizing a conditional generative adversarial network (cGAN); running a document identifier to identify a type of a document, wherein the type is printed text or handwritten text (206); performing optical character recognition (OCR) on the image according to the identified type of the document to detect a set of text entities (208); designing a schema to establish relationships among the set of text entities (210), wherein the set of text entities are associated with spatial coordinates of the set of text entities and the association indicates locations of adjacent text entities; storing the detected set of text entities and the schema in a relational database (212); providing a natural language query by a user using a conversational interface to extract relevant information (214), wherein the extraction of the information is facilitated by identifying pattern-based entities from one or more documents and by maintaining a meta-relationship model among different entities in a document image; converting the natural language query into a SQL query (216); and providing the SQL query to the relational database to extract the relevant information from the relational database (218).

2. The method of claim 1, further comprising the step of providing the SQL query directly to the relational database after storing the detected set of text entities and the schema in the relational database.

3. The method of claim 1, further comprising the step of processing handwritten text using a recurrent encoder-decoder model for mapping visual features to a set of characters present in the input image.

4. The method of claim 1, further comprising the step of classifying a natural language query and its intent.

5. The method of claim 1, further comprising the step of storing a workflow of a set of SQL queries for extracting the relevant information.

6. The method of claim 1, wherein, running the document identifier using a twin network composed of a convolutional neural network (CNN).

7. The method of claim 1, wherein, the set of text entities is one or more of a word, a line, a block of text, a checkbox, an arrow, an OMR circle, a logo, a table, a graph, and a text box.

8. The method of claim 1, wherein, running the document identifier to determine a correct template before performing OCR.

9. The method of claim 1, wherein, the noise in the input image is due to one or more of camera shake, improper focusing, imaging noise, coffee stains, creases, low resolution, poor lighting, and reflections.

10. The method of claim 1, further comprising the step of breaking an initial SQL query into one or more queries that are simpler than the initial query.

11. A system (100) for extracting relevant information from an image, the system comprising: an input module (102) for providing the image as an input image; a memory (104); and a processor (106) in communication with the memory, the processor further comprising: a noise removal module (108) for performing a denoising method on the input image to remove noise from the input image, wherein the denoising method is performed by utilizing a conditional generative adversarial network (cGAN); a document identifier (110) for identifying a type of a document, wherein the type is printed text or handwritten text; an OCR engine (112) for performing optical character recognition (OCR) on the image according to the identified type of the document to detect a set of text entities; a schema design module (114) for designing a schema to establish relationships among the set of text entities, wherein the set of text entities are associated with spatial coordinates of the set of text entities and the association indicates locations of adjacent text entities; a storage module (116) for storing the detected set of text entities and the schema in a relational database; a conversational interface (118) for providing a natural language query by a user to extract relevant information, wherein the extraction of the information is facilitated by identifying pattern-based entities from one or more documents and by maintaining a meta-relationship model among different entities in a document image; a conversion module (120) for converting the natural language query into a SQL query; and a database query module (122) for providing the SQL query to the relational database to extract the relevant information from the relational database.

12. A computer program product comprising a non-transitory computer readable medium having embodied therein a computer readable program, wherein, When executed on a computing device, the computer readable program causes the computing device to: obtain an image as an input image (202); perform a denoising method on the input image to remove noise from the input image (204), wherein the denoising method is performed by utilizing a conditional generative adversarial network (cGAN); run a document identifier to identify a type of a document, wherein the type is printed text or handwritten text (206); perform optical character recognition (OCR) on the image according to the identified type of the document to detect a set of text entities (208); design a schema to establish relationships among the set of text entities (210), wherein the set of text entities are associated with spatial coordinates of the set of text entities and the association indicates locations of adjacent text entities; store the detected set of text entities and the schema in a relational database (212); provide a natural language query by a user to extract relevant information using a conversational interface (214), wherein the extraction of the information is facilitated by identifying pattern-based entities from one or more documents and by maintaining a meta-relationship model among different entities in a document image; convert the natural language query into a SQL query (216); and provide the SQL query to the relational database to extract the relevant information from the relational database (218).

13. The computer program product of claim 12, further comprising the step of providing the SQL query to the relational database directly after storing the detected set of text entities and the schema in the relational database.

Images