Automated key-value pair extraction

Abstract

A document to be analyzed and a set of key names to be extracted from the document are received. A set of strings contained within the document and a position of each string are identified. Further, a document graph is generated for the document. The document graph includes a set of nodes and a set of edges. Each node in the set of nodes corresponds to a string in the set of strings. Each edge in the set of edges interconnects two or more nodes. Further, a set of keys is identified based on the document graph and the received set of key names. Further, a set of values ​​is extracted from the document, and a set of key-value pairs is generated based on the identified set of keys and the extracted set of values.

Description

[Technical field]

[0001] The described embodiments relate generally to data analysis, and more particularly to extracting key-value pairs from documents. [Background technology]

[0002] The growth of data analytics platforms, such as big data analytics, has expanded data processing to tools for processing large amounts of data to extract information of business value. For example, small businesses may use third-party data analytics environments that utilize dedicated computing and human resources to aggregate, process, and analyze vast amounts of data from various sources, such as external data providers and internal data sources (e.g., files on a local computer). Processing such large data sets to extract useful quantitative and qualitative information for use in data analytics typically requires complex software tools implemented on powerful computing devices.

[0003] The data used by such data analytics platforms may come from a set of documents. Many documents contain associations between keys and values ​​(e.g., the "name" field on a tax form). These key-value pairs can be quite difficult for a computer to extract. For example, basic OCR extraction can lead to complex and tedious post-processing and text mining. Thus, there is a need for tools that can extract key-value pairs from documents by providing only the names of the keys that the user wants to extract. Summary of the Invention

[0004] These and other problems are addressed by a computer-implemented method, computer system, and computer-readable storage medium for extracting key-value pairs from a document. An embodiment of the computer-implemented method includes receiving a document to be analyzed and a first set of keys to be extracted from the document. The method further includes identifying a set of strings contained within the document and a location of each string. The method further includes generating a document graph. The document graph includes a set of nodes and a set of edges. Each node in the set of nodes corresponds to a string in the set of strings. Each edge in the set of edges connects two or more nodes to each other. The method further includes identifying a second set of keys based on the document graph and the received first set of keys. Additionally, the method includes extracting a set of values ​​from the document. Finally, the method includes generating a set of key-value pairs based on the identified second set of keys and the extracted set of values.

[0005] An embodiment of a computer system includes a computer processor for executing computer program instructions. The system also includes a non-transitory computer readable memory storing computer program instructions that, when executed, cause the computer processor to perform operations. The operations include receiving a document to be analyzed and a first set of keys to be extracted from the document. The operations further include identifying a set of strings contained within the document and a location of each string. The operations further include generating a document graph. The document graph includes a set of nodes and a set of edges. Each node in the set of nodes corresponds to a string in the set of strings. Each edge in the set of edges connects two or more nodes to each other. The operations further include identifying a second set of keys based on the document graph and the received first set of keys. Additionally, the operations include extracting a set of values ​​from the document. Finally, the operations include generating a set of key-value pairs based on the identified second set of keys and the extracted set of values.

[0006] An embodiment of a non-transitory computer readable memory stores executable computer program instructions. The instructions are executable to perform operations. The operations include receiving a document to be analyzed and a first set of keys to be extracted from the document. The operations further include identifying a set of strings contained within the document and a location of each string. The operations further include generating a document graph. The document graph includes a set of nodes and a set of edges. Each node in the set of nodes corresponds to a string in the set of strings. Each edge in the set of edges connects two or more nodes to each other. The operations further include identifying a second set of keys based on the document graph and the received first set of keys. Additionally, the operations include extracting a set of values ​​from the document. Finally, the operations include generating a set of key-value pairs based on the identified second set of keys and the extracted set of values. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a data analysis environment including a data analysis system in accordance with one or more embodiments. [Diagram 2] FIG. 2 is a block diagram illustrating a workflow for extracting key-value pairs from a document in accordance with one or more embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating a data analysis application in accordance with one or more embodiments. [Figure 4A] FIG. 2 illustrates a marked-up image of a document showing the output of an OCR module in accordance with one or more embodiments. [Figure 4B] FIG. 2 illustrates a marked-up image of a document showing a representation of a document graph in accordance with one or more embodiments. [Figure 4C] FIG. 2 illustrates a marked-up image of a document showing the output of a key identification module in accordance with one or more embodiments. [Figure 4D]FIG. 2 illustrates an image corresponding to horizontal and vertical lines generated by applying a kernel to an image of a document, in accordance with one or more embodiments. [Figure 4E] FIG. 2 illustrates a line-removed document generated by subtracting from an image of the document images that correspond to horizontal and vertical lines in the document, in accordance with one or more embodiments. [Figure 4F] 1 illustrates a marked-up image of a document showing a bounding box generated by a value identification module in accordance with one or more embodiments. [Figure 4G] FIG. 1 illustrates a marked-up image of a document showing bounding boxes with identified keys and values ​​in accordance with one or more embodiments. [Diagram 5] 1 is a flow diagram illustrating a process for extracting key-value pairs using an analysis workflow in accordance with one or more embodiments. [Figure 6] FIG. 2 is a high-level block diagram illustrating a functional view of a representative computer system that may be used as the data analysis system of FIG. 1 in accordance with one embodiment.

[0008] The drawings depict various embodiments for purposes of illustration only. Those skilled in the art will readily appreciate from the following discussion that alternative embodiments of the structures and methods shown herein may be utilized without departing from the principles of the embodiments described herein. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1 is a block diagram illustrating a data analysis environment 100 including a data analysis system 110 according to one embodiment. The environment 100 further includes multiple data sources 120 connected to the data analysis system 110 via a network 130. Although the illustrated environment 100 includes only one data analysis system 110 coupled to multiple data sources 120, an embodiment may have multiple data analysis systems and one data source. Additionally, although the data sources 120 are shown as being connected to the data analysis system 110 via a network, one or more data sources may be stored locally to the data analysis system. Alternatively, a user may provide data to the data analysis system directly (either via the network 130 or via a local interface of the data analysis system).

[0010] The data analysis system 110 is a computer-based system utilized to process large amounts of data. In some embodiments, data is collected, aggregated, or otherwise accessed from multiple data sources 120 over a network 130. The data analysis system 110 can implement scalable software tools and hardware resources utilized for data processing tasks from a wide variety of data sources. A data processing task (also referred to as a task or data analysis task) is a production operation for processing data. Data processing tasks include, for example, accessing data, preparing data, blending data, packaging data, analyzing data, other types of operations on data, or any combination thereof. The data analysis system 110 can perform multiple tasks in parallel. When the data analysis system 110 has available computer resources (e.g., memory), it can choose a task to perform from multiple tasks that are ready to be performed.

[0011] 1 embodiment, data analysis system 110 includes memory 140 and data analysis application 150. Memory 140 stores data used in data analysis operations and provides access to data storage. In some embodiments, memory 140 includes random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), other forms of computer memory, or some combination thereof. In some embodiments, memory 140 includes primary memory 140, which provides relatively fast access to a relatively small data storage area, and secondary memory 150, which provides relatively slower access to a relatively large data storage area.

[0012] The data analysis application 150 is a software application that supports the execution of data analysis tasks using analysis workflows (also referred to as "workflows"), such as the workflow illustrated in FIG. 2. A workflow is a sequence of data analysis tools (also referred to as "tools") or data analysis modules (also referred to as "modules") through which a data analysis task progresses from initiation (e.g., input of a data file) to completion (e.g., output of the results of the task). Each tool in the sequence performs a particular processing operation or data analysis function, such as input, output, preparation, combination, predictive, spatial, exploration, parsing, and transformation. Execution of a workflow involves the operation of the tools in the workflow and produces result items that represent the results of the task. A result item, in one example, is a chart that provides a visual representation of the results of the task.

[0013] The data analysis application 150 provides an environment that facilitates the creation of a workflow. For example, the data analysis application 150 provides a visual workflow environment that is supported by a GUI of the data analysis application 150. The visual workflow environment enables a set of drag-and-drop tools that eliminate the need to provide software code to design a workflow and eliminate the need to identify complex mathematical formulas implemented by the workflow. In another embodiment, the workflow 200 is created and described in a document, such as an Extensible Markup Language (XML) document.

[0014] The environment may also facilitate the creation of logically controlled workflows, such as the workflow shown in FIG. 2. A logically controlled workflow is a workflow that includes a sequence of tools integrated with one or more logical controllers. The logical controllers have logic functions and are adapted to determine whether or not to trigger (and / or not) a tool or module based on an input. The data analysis application 150 uses the sequence of tools to process the data file and uses the logical controllers to control the control operation of the tools in the sequence during the processing of the data file. In some embodiments, the logical controllers are associated with at least two tools in the workflow, a first tool and a second tool that follows the first tool in the sequence. The data analysis application 150 applies a logging function to the first tool to log events that occur during the performance of the data analysis function of the first tool in the processing of the data file. The log data generated by the first tool describes activities performed by the first tool and / or events detected by the first tool during the processing of the data file. The events may be, for example, errors detected during performance of the data analysis function of the tool, warnings generated during performance of the data analysis function of the tool, the number of records processed by the tool, the output of a metadata file, and completion of performance of the data analysis function of the tool, etc. The first tool also outputs analysis data, such as in a metadata file. The analysis data is generated by the first tool through performance of the data analysis function in processing the data file.

[0015] The data analysis application 150 uses the log data and / or analysis data generated by the first tool as input to the logical controller. For example, the logical controller can analyze the log data to detect events described in the log data and determine whether to trigger a second tool based on the detected events. In some embodiments, the logical controller is a data analysis tool having a logic function. The tool of the logical controller performs a data analysis function on the log to detect the events logged by the first tool. The logic function outputs a command based on the output of the tool of the logical controller. The second tool is adapted to receive the command and process the data according to the command.

[0016] For example, in response to detecting the successful completion of the data analysis function of the first tool, the logical controller triggers the second tool to operate. In some embodiments, the logical controller receives a metadata file from the first tool and transmits the metadata file to the second tool in response to the decision to trigger the second tool. Upon receiving the metadata file from the logical controller, the second tool processes the metadata file. The logical controller may also selectively execute other tools in the workflow based on the log generated by the first tool. If there are tools following the second tool, they also operate to perform their data analysis functions and finish processing the data file. Result items are generated that represent the results of the processing of the data file by the sequence of tools. The data analysis application 150 provides the result items for display to the user.

[0017] In another example, the logical controller does not trigger the operation of the second tool in response to detecting an error that occurred during the operation of the first tool. In some embodiments, in response to deciding not to trigger the operation of the second tool, the logical controller triggers the operation of a third tool. The third tool may not be in the sequence. The third tool is adapted to generate an auxiliary result item. In one example, the auxiliary result item is a message specifying an event that occurred during the operation of the first tool that caused the logical controller to decide not to trigger the data analysis function of the second tool. The third tool may generate this message based on information of the event from the logical controller and send this message to a client device associated with the user. In some embodiments, the data analysis application 150 receives an action from the user to address the event. The data analysis application 150 may resolve the event based on the user's action. In response to resolving the user's action, the data analysis application 150 may instruct the logical controller to trigger the second tool to finish processing the data file.

[0018] These examples are provided for illustrative purposes. The logical controller may detect other types of logged events. The logical controller may also control the execution of other tools. For example, in addition to deciding to trigger the second tool, the logical controller may decide to trigger (or not) one or more other tools. The one or more other tools may be connected to the second tool, for example, in the same order as they are arranged in the sequence of tools. Similarly, in addition to deciding not to trigger the second tool, the logical controller may decide to trigger (or not) one or more other tools. Furthermore, the logical controller may receive as input logs generated by one or more other tools in addition to the first tool and use the input.

[0019] The logically controlled workflow generated by the data analysis application 150 is advantageous because it allows the user to monitor and control the data analysis process. By using such a logically controlled workflow, the data analysis process can be paused at the first tool after a negative event (i.e., an event that causes the logical controller to decide not to trigger the second tool). The negative event may cause the data generated by the first tool to be unsuitable for further processing. For example, the first tool may stop processing a data file when an error occurs, such that the data generated by the first tool is incomplete. Thus, the pause of the data analysis process prevents other tools from wasting computing resources to process unsuitable data. Also, the user may add a third tool to the workflow so that they are notified of the negative event through an auxiliary result item. In this way, the user has the opportunity to resolve the negative event and resume the data analysis process. Compared to conventional data analysis techniques, this logically controlled workflow facilitates the user's contribution to the data analysis process, saving computing resources that would otherwise be wasted on processing unsuitable data.

[0020] In some cases, the data analysis application 150 provides software that supports network-based or cloud-based access to data analysis tools and macros to multiple end users. For example, the data analysis application 150 supports the creation of workflows in the form of dashboards, web applications, mobile applications, etc. In another example, the data analysis application 150 allows users to share, view, and consume analysis, similar to a mobile application store or other cloud-based service. The analysis data, macros, and workflows can be packaged and executed as smaller, customizable analysis applications (i.e., apps) that can be accessed by other users of the data analysis system 110, for example. In some cases, access to published analysis apps can be managed by the data analysis system 110, i.e., access can be granted or revoked, thereby providing access control and security features. The data analysis application 150 can perform functions associated with analysis apps, such as creating, deploying, publishing, iterating, and updating.

[0021] Further, the data analysis application 150 can support functions performed at various stages related to data analysis, such as the ability to access, prepare, blend, analyze, and output analysis results. In some cases, the data analysis application 150 can access various data sources to retrieve raw data, for example, in a stream of data. The data stream collected by the data analysis application 150 can include raw data of multiple records, where the raw data is in various formats and structures. After receiving at least one data stream, the data analysis application 150 can perform operations that enable the records of the data stream to be used as input to a data analysis operation. Furthermore, analysis functions related to statistical, qualitative, or quantitative processing of records, such as predictive analysis (e.g., predictive modeling, clustering, data exploration, etc.), can be performed by the data analysis application 150.

[0022] As described above, the data sources 120 provide electronic data to the data analysis system 110. The data sources 120 may be computers, databases, network APIs (Application Programming Interfaces), or cloud storage systems. The data sources 120 may be computer systems that can retrieve data from other sources. The data sources 120 may be remote from the data analysis system 110 and provide data via the network 130. Additionally, some or all of the data sources 120 may be directly coupled to the data analysis system and provide data without passing it through the network 130. The data provided by the data sources 120 is typically organized into data records, each of which includes one or more values. For example, the data records provided by the data sources may include a series of comma-separated values. This data describes information relevant to the enterprises that use the data analysis system. For example, the data of the data sources 120 may describe content accessible on a website and / or computer-based interactions with social media applications (e.g., click tracking data).

[0023] Network 130 represents a communication path between data analysis system 110 and data sources 120. In one embodiment, network 130 is the Internet and uses standard communication technologies and / or protocols. Data exchanged over network 130 may be represented using technologies and / or formats such as HyperText Markup Language (HTML), Extensible Markup Language (XML), etc. In another embodiment, these entities may use custom and / or proprietary data communication technologies instead of or in addition to those described above.

[0024] 2 is a block diagram illustrating a workflow 200 implemented using data analysis system 110 to process data according to one embodiment. Workflow 200 is executed by a computing device of data analysis system 110. However, in other embodiments, workflow 200 is deployed on a separate computing device that may be communicatively connected to data analysis system 110 via a network (e.g., network 130).

[0025] 2 includes receiving a set of inputs 210 and generating and / or presenting a set of outputs 215. In some embodiments, the set of inputs is received (e.g., from a user of the data analysis system 110, by another process that interacts with the data analysis system 110, or from a storage device) using a set of input tools. Additionally, the set of outputs may be presented to a user of the data analysis system 110 via a browsing tool.

[0026] The data analysis tool 110 receives as input a document 220 and a set of user-provided key names 230. In some embodiments, the document is received in a specified document format, such as Portable Document Format (PDF). The data analysis tool 110 then extracts a set of key-value pairs 280 from the document 220 based on the set of user-provided key names 230.

[0027] In some embodiments, to extract the set of key-value pairs 280, the data analysis system 110 generates a document graph 240 by analyzing the received document 220. The document graph 240 may include a set of nodes and a set of edges. Each node in the set of nodes may correspond to a string identified in the document 220. In some embodiments, each node stores the string to which it corresponds and an associated location where the string is located within the document 220. Furthermore, each edge in the set of edges connects two or more nodes to each other.

[0028] Further, based on the generated document graph 240 and the set of user-provided key names 230, the data analysis system 110 identifies a set of keys 250 within the document. In some embodiments, the data analysis system 110 searches the document graph 240 to identify each of the user-provided key names 230 within the document graph 240. The data analysis tool may further extract the location of each key 250 within the document from the document graph 240.

[0029] The data analysis system 110 identifies a set of text blocks 260 in the document 220 and extracts a set of values ​​270 from the document based on the identified set of text blocks 260 and the identified set of keys 250. The data analysis system 110 then matches keys from the set of identified keys 250 with corresponding values ​​from the set of identified values ​​270 to generate a set of key-value pairs 280.

[0030] FIG. 3 is a block diagram illustrating a data analysis application 300 according to one or more embodiments. The data analysis application 300 is an embodiment of the data analysis application 150 of FIG. 1. In the embodiment of FIG. 3, the data analysis application 300 includes a data receiver module 310, an optical character recognition (OCR) module 320, a graph generation module 330, a key identification module 340, a value identification module 350, and a user interface module 360. Those skilled in the art will appreciate that other embodiments may have different and / or other components than those described herein, and that functionality may be distributed among those components in different ways. For example, some or all of the functionality of the data analysis application 300 may be implemented on a client device or a third-party server.

[0031] The data receiver module 310 receives data files from a data source, such as data source 120, and provides the data files to other modules in the data analysis application 300. In one embodiment, the data receiver module 310 acts as an interface between modules in the data analysis application. In another embodiment, there is no data receiver module 310, and the functionality of the data receiver module 310 is performed by each module of the data analysis application.

[0032] In some embodiments, the data receiver module 310 is configured to receive a document 220 to be analyzed and a set of parameters for analyzing the document. The set of parameters for analyzing the document may include a set of user-provided key names 230 for extracting key-value pairs 280 from the document 220. In some embodiments, the data receiver module 310 is configured to receive a document having a predefined format (e.g., PDF format). The data receiver module 310 may be configured to identify whether the document 220 has the correct format and may output an error if the document does not have the predefined format.

[0033] The OCR module 320 analyzes the document 220 and identifies character strings contained within the document 220. In some embodiments, the OCR module 320 further determines the location or coordinates of each of the identified character strings. The location or coordinates of the character string may correspond to a centroid of a bounding box around the character string within the document 220. In some embodiments, the OCR module 320 receives as input an image of the document (or one or more images corresponding to pages of the document). The OCR module 320 may pre-process the input image to increase the likelihood of correctly detecting character strings contained within the input image. For example, the OCR module may convert the image to a predetermined resolution (e.g., 300 dots per inch (dpi)), convert the image to grayscale, and perform adaptive Gaussian thresholding, dilation, erosion, median blur, and simple thresholding.

[0034] 4A illustrates a marked-up image of a document showing the output of an OCR module according to one or more embodiments. As shown in FIG. 4A, the OCR module identifies regions of the document that contain strings of characters and recognizes characters within the strings. In the example of FIG. 4A, each string of characters is surrounded by a box.

[0035] 3, the graph generation module 330 generates a document graph 240 for the document 220 from the output of the OCR module 320. Specifically, the document graph 240 generated by the graph generation module 330 based on the output of the OCR module 320 includes a set of nodes interconnected by a set of edges. In some embodiments, the graph generation module 330 adds a node for each string identified from the document by the OCR module 320. In some embodiments, each node stores the string to which it corresponds and the associated location where the string is located in the document.

[0036] In some embodiments, for each node in the document graph 240, the graph generation module 330 identifies a set of neighboring nodes that are connected to the node. The graph generation model 330 may also add edges to connect the node to a neighboring node if the neighboring node is within a threshold distance from the node. In some embodiments, the graph generation model 330 adds edges to connect the node to a neighboring node if the neighboring node is within a threshold distance from the node and if the node's location and the neighboring node's location are within an angular range of each other.

[0037] FIG. 4B illustrates a markup image of a document showing the output of a graph generation module according to one or more embodiments. In the example of FIG. 4B, each character string identified by the OCR module 320 in the image of the document shown in FIG. 4A is assigned to a node 421 in the document graph. Additionally, each node 421 is connected to one or more nodes by an edge 423. For example, node 421A corresponding to the character string "Work" is connected to node 421B corresponding to the character string "Request" 421B by an edge 423A. In response to determining that node 421B is within a threshold distance from node 421A and within a threshold angle from node 421A, the graph generation module 330 may connect node 421A to node 421B. Similarly, node 421A is connected to node 421D corresponding to the character string "Requesting" by an edge 423B. However, node 421A is not connected to node 421C corresponding to the character string "NumberEWR-001-1001". In the example of FIG. 4B, node 421A is not connected to node 421C because node 421C is not within the threshold angle of node 421A.

[0038] 3, the key identification module 340 identifies a set of keys 250 from the document 220 to form key-value pairs 280 that are extracted from the document. The key identification module 340 receives a set of user-provided key names 230 and traverses the document graph 240 to identify the user-provided key names 230 within the document 220.

[0039] In some embodiments, for each user-provided key name, the key identification module 340 determines whether the user-provided key name includes multiple words. If the user-provided key name includes multiple words, the key identification module divides the user-provided key name into a set of words (or a set of tokens). The key identification module 340 may then identify one or more chains of nodes corresponding to the set of words of the user-provided key name by searching the first word of the user-provided key name in the document graph 240 and traversing the document graph to identify subsequent words of the user-provided key name. That is, the key identification module 340 may identify a set of nodes that match the first word of the user-provided key name by comparing each node string identified by the OCR module 320 to the first word of the user-provided key name. For each identified node that matches the first word of the user-provided key name, the key identification module 340 determines whether a second node connected via an edge to the identified node matches the second word of the user-provided key name. If the identified node is not connected to a second node that matches the second word of the user-provided key name, the key identification module 340 determines that the identified node does not correspond to the user-provided key name. Alternatively, if the identified node has a second node that matches the second word of the user-provided key name, the key identification module 340 retains the identified node as a candidate key that corresponds to the user-provided key name. If the user-provided key name includes additional words, the key identification module 340 continues to traverse the document graph until all words of the user-provided key name are found. For example, if the user-provided key name includes a third word, the key identification module 340 determines whether a third node connected to the second node via an edge matches the third word of the user-provided key name, and so on.

[0040] In some embodiments, a chain of nodes (i.e., a set of nodes connected via a set of edges) matches a key name provided by a user, and the key identification module 340 merges the nodes in this chain of nodes into one node. The key identification module 340 may also determine a new location for the merged node. The key identification module 340 may then replace the node in the chain of nodes with the merged node.

[0041] In some embodiments, the key identification module 340 uses fuzzy matching to determine whether a word matches a node in the document graph. For example, the key identification may compare the word to a string corresponding to the node and determine a Levenshtein ratio (LR) based on the comparison. If the LR exceeds a threshold (e.g., 90%), the key identification module 340 determines that the word matches the string corresponding to the node. Alternatively, the key identification module 340 selects a top predefined number of nodes (e.g., the top 10 nodes) that have the highest LR value for the word under consideration.

[0042] 4C illustrates a marked-up image of a document showing the output of a key identification module according to one or more embodiments. Specifically, FIG. 4C illustrates a set of keys identified by the key identification module. Each key identified by the key identification module is surrounded by a box.

[0043] 3, the value identification module 350 identifies a set of values ​​270 from the document 220 to form a set of key-value pairs 280 that are extracted from the document. The value identification module 350 identifies text blocks 260 (i.e., blocks of text) from the document 220 and matches the text blocks 260 with the keys 250 identified by the key identification module 340. The value identification module 350 then extracts values ​​from the text blocks 260.

[0044] In some embodiments, the value identification module 350 delines the document. For example, the value identification module 350 delines horizontal and vertical lines from the image of the document (or from the images corresponding to each page of the document). To delineate the lines from the document, a kernel is applied to the image of the document to generate an image corresponding to the horizontal and vertical lines. In some embodiments, the kernel is determined by the size of the document. Further, in some embodiments, a first kernel is applied to generate a first image corresponding to the horizontal lines, and a second kernel is applied to generate a second image corresponding to the vertical lines. The generated image (or images) corresponding to the horizontal and vertical lines are subtracted from the image of the document to generate a line-delineated image of the document.

[0045] Figure 4D shows an image corresponding to horizontal and vertical lines generated by applying a kernel to an image of a document, according to one or more embodiments. Figure 4E shows a line-removed document generated by subtracting from the image of the document an image corresponding to the horizontal and vertical lines in the document, according to one or more embodiments.

[0046] The line-removed image of the document is then processed to generate a set of bounding boxes. In some embodiments, a run-length smoothing algorithm (RLSA) is applied to the line-removed image to generate the bounding boxes. In some embodiments, the RLSA identifies a set of foreground pixels (e.g., non-blank or black pixels) and a set of background pixels (e.g., blank or white pixels). The RLSA then converts background pixels to foreground pixels based on the proximity of the background pixels to the foreground pixels. In some embodiments, the RLSA converts a background pixel to a foreground pixel if the background pixel is within a predefined number of pixels from the foreground pixel in both directions on the horizontal axis of the line-removed image.

[0047] In some embodiments, RLSA converts background pixels that have a number of neighboring background pixels less than a threshold limit. In some embodiments, RLSA represents background pixels of a document with 0 and foreground pixels of a document with 1. Then, for each 0 in the representation of the document, RLSA replaces the 0 with a 1 if the number of neighboring 0's is less than the threshold limit. For example, if the threshold limit is 3, a portion of a document having a representation of 1000111000001 is converted to a representation of 1111111000001. In some embodiments, RLSA is applied horizontally. In other embodiments, RLSA is applied both horizontally and vertically.

[0048] After performing RLSA, the value identification module 350 performs a contour detection algorithm to generate bounding boxes around blocks of associated text. The contour detection algorithm identifies bounding boxes for pixels classified as foreground pixels after RLSA is applied. In some embodiments, the value identification module performs a contour detection algorithm on the bounding boxes to merge adjacent boxes. For example, the value identification module 350 may convert all pixels enclosed by the bounding boxes as foreground pixels, and reapply the contour detection algorithm to find bounding boxes for pixels classified as foreground pixels after the first contour detection algorithm is applied.

[0049] 4F illustrates a markup image of a document showing bounding boxes generated by a value identification module according to one or more embodiments. Specifically, FIG. 4F illustrates a set of foreground pixels identified by RLSA and a set of bounding boxes, each of which encloses a chunk of the foreground pixels.

[0050] Using the bounding boxes, the value identification module 350 then identifies one or more values ​​for each key identified by the key identification module 340. In some embodiments, for each key identified by the key identification module 340, the value identification module 350 identifies a value by identifying a bounding box based on the position of the key and extracting the text enclosed by the identified bounding box.

[0051] In some embodiments, for a given key, the value identification module 350 identifies a bounding box that encloses the key. Alternatively, the value identification module 350 identifies the closest bounding box to the key. In some embodiments, the value identification module 350 traverses the document graph to identify the bounding box to associate with the key. In some embodiments, the value identification module 350 determines whether any given bounding box is the closest bounding box to multiple keys. If the value identification module 350 determines that a bounding box is the closest bounding box to multiple keys, the value identification module 350 divides the bounding box and associates each key with a portion of the bounding box.

[0052] Alternatively, in some embodiments, instead of processing each key to identify a bounding box associated with that key, the value identification module 350 processes each bounding box to identify whether to associate the bounding box with one or more keys. For example, for each bounding box, the value identification module 350 may identify the keys enclosed by the bounding box. In this embodiment, the value identification module 350 may consider three different scenarios. First, the bounding box may include one key and an associated value. Second, the bounding box may include multiple keys and associated values. Third, the bounding box may include a value and no keys.

[0053] If the value identification module 350 determines that the bounding box contains a key, the value identification module 350 extracts the value from the bounding box and generates a key-value pair based on the key and the extracted value. For example, the value may be extracted by removing the text of the key from the text enclosed by the bounding box.

[0054] Furthermore, if the value identification module 350 determines that the bounding box contains multiple keys, the value identification module 350 divides the bounding box based on the number of keys enclosed by the bounding box. In some embodiments, the value identification module 350 divides the bounding box by attempting to optimize the boundary between the keys and the values. For example, if the bounding box contains n keys stacked in a row, the value identification module 350 divides the bounding box horizontally into n smaller bounding boxes, each containing one key and one value. The value identification module 350 then applies the steps corresponding to the case where the bounding box contains one key and associated value to each of the bounding boxes obtained by dividing the bounding box containing multiple keys.

[0055] Additionally, if the value identification module 350 determines that the bounding box does not contain the key, the value identification module associates the bounding box with the key using the document graph.

[0056] FIG. 4G illustrates a markup image of a document showing bounding boxes with identified keys and values ​​according to one or more embodiments. In the example of FIG. 4G, the document includes a set of bounding boxes 471 that include one key and one value. For example, bounding box 471A includes the key "Type" and the value "Change". When processing bounding box 471A, the value identification module determines that only one key is enclosed by bounding box 471A and extracts the value from the bounding box to generate a key-value pair for the key located within the bounding box. For example, the value identification module extracts the value from the bounding box by removing the key from the string located within bounding box 471A. Additionally, in the example of FIG. 4G, the document includes a set of bounding boxes 473 that include values ​​but no keys. For example, bounding box 473A includes the value "CARBURETOR" but no key. Here, the value identification module identifies the key that is closest to the value and associates the identified key with the value enclosed by the bounding box. In this case, the value identification module identifies the key "Name" as being closest to bounding box 473A and creates a key-value pair by associating the key "Name" with the value "CARBURETOR".

[0057] 5 is a flow diagram illustrating a process for extracting key-value pairs using an analysis workflow according to one or more embodiments. In some embodiments, the process is performed by data analysis application 150, although some or all of the operations in the process may be performed by other entities in other embodiments. In some embodiments, the operations in the process may be performed in a different order and may include different and / or additional steps.

[0058] In some embodiments, the data analysis application 150 accesses (510) a workflow. The analysis workflow includes a sequence of tools. Each tool in the sequence is adapted to perform a data analysis function, such as input, output, preparation, combination, predictive, spatial, exploration, parsing, and transformation. The analysis workflow is a logically controlled workflow and further includes one or more logical controllers.

[0059] The data reception module 310 of the data analysis application 150 receives (510) a document to be analyzed. The document to be analyzed may be received as a data file (e.g., uploaded by a user of the data analysis application). The data file may be received in one of a set of supported document formats (e.g., PDF format). Additionally, the data reception module 310 of the data analysis application 150 receives (515) a set of user-provided key names to be extracted from the document. In some embodiments, the keys are received from a user of the data analysis application 150 via a graphical user interface provided by a user interface module 360 ​​of the data analysis application.

[0060] The data analysis application 150 provides a document to the OCR module 320. The OCR module 320 of the data analysis application 150 identifies (520) a set of strings (or text strings) within the document. Additionally, the OCR module 320 may identify a location of each identified string. In some embodiments, the OCR module extracts the set of strings from an image corresponding to the document. Additionally, the OCR module 320 may perform preprocessing on the document or the image of the document before extracting the set of strings from the document.

[0061] The data analysis application 150 may provide the output of the OCR module 320 to a graph generation module 330. The graph generation module 330 of the data analysis application 150 generates (525) a document graph for the document based on the set of strings extracted by the OCR module 320. The document graph may include a set of nodes (each corresponding to a string in the set of strings identified by the OCR module 320) and a set of edges (each edge connecting two or more nodes to each other).

[0062] The data analysis application 150 may then provide the document graph generated by the graph generation module 330 and the user-provided key names to a key identification module 340. The key identification module 340 of the data analysis application 150 identifies (530) a set of keys in the document. The key identification module 340 may match each user-provided key name with one or more nodes or node chains (i.e., a set of nodes connected via a set of edges) in the document graph.

[0063] The data analysis application 150 may then provide the document and the keys identified by the key identification module 340 to the value identification module 350. In some embodiments, the data analysis application 150 further provides the document graph to the value identification module 350. The value identification module 350 of the data analysis application 150 identifies (535) a set of blocks of text in the document. In particular, the value identification module 350 generates a set of bounding boxes, each of which encloses one or more strings of text in the document. To generate the bounding boxes, the value identification module 350 may remove lines from the document, apply RLSA, and apply a contour detection algorithm.

[0064] Further, the value identification module 350 of the data analysis application 150 extracts (540) a value for each key from the identified blocks of text. In some embodiments, to extract the value, the value identification module 350 processes each of the identified bounding boxes to determine whether one or more keys are located within each of those bounding boxes. If multiple keys are located within a bounding box, the value identification model 350 divides the bounding box such that only one key is located within each of the resulting bounding boxes. For each bounding box that contains only one key, the value identification module 350 extracts from the text enclosed by the bounding box a value associated with the key located within the bounding box. Further, if no key is located within the bounding box, the value identification module determines whether to associate the bounding box with a key.

[0065] Finally, a set of key-value pairs is generated (545) based on the keys identified by key identification module 340 and the values ​​identified by value identification module 350. In some embodiments, data analysis application 150 outputs the generated key-value pairs to a user of the data analysis application. Alternatively, data analysis application 150 may store the generated key-value pairs in a file or in a database connected to the data analysis application.

[0066] FIG. 6 is a high-level block diagram illustrating a functional view of a representative computer system 600 for use as the machine learning server 110 of FIG. 1 in accordance with one or more embodiments.

[0067] The illustrated computer system includes at least one processor 602 coupled to a chipset 604. The processor 602 may include multiple processor cores on the same die. The chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. The memory 606 and the graphics adapter 612 are coupled to the memory controller hub 620, and the display 618 is coupled to the graphics adapter 612. The storage device 608, the keyboard 610, the pointing device 614, and the network adapter 616 may be coupled to the I / O controller hub 622. In some other embodiments, the computer system 600 may have additional, fewer, or different components, and these components may be coupled differently. For example, an embodiment of the computer system 600 may not have a display and / or a keyboard. Additionally, the computer system 600 may be instantiated as a rack-mounted blade server or as a cloud server instance in some embodiments.

[0068] The memory 606 holds instructions and data used by the processor 602. In some embodiments, the memory 606 is a random access memory. The storage device 608 is a non-transitory computer-readable storage medium. The storage device 608 can be a HDD, SSD, or other type of non-transitory computer-readable storage medium. Data processed and analyzed by the machine learning server 110 can be stored in the memory 606 and / or the storage device 608.

[0069] Pointing device 614 may be a mouse, trackball, or other type of pointing device and is used in combination with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. In some embodiments, display 618 includes touch screen capabilities for receiving user inputs and selections. Network adapter 616 couples computer system 600 to network 160.

[0070] The computer system 600 is adapted to execute computer modules that provide the functionality described herein. As used herein, the term "module" refers to computer program instructions and other logic that provide a specified functionality. A module can be implemented as hardware, firmware, and / or software. A module can include one or more processes and / or can be provided by only a portion of a process. A module is typically stored in a storage device 608, loaded into memory 606, and executed by the processor 602.

[0071] The individual names of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not required or important, and mechanisms for implementing the described embodiments may have different names, formats, or protocols. Furthermore, the system may be implemented as a combination of hardware and software as described, or may be implemented entirely with hardware elements. Also, the particular division of functionality between various system components described herein is merely exemplary and not required, and functionality performed by one system component may be performed by multiple components rather than one, and functionality performed by multiple components may be performed by one component rather than multiple.

[0072] Some portions of the above description are presented in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described in functional or logical terms, will be understood as being implemented by computer programs. Further, it has proven convenient at times to refer to arrangements of these operations as modules or by functional names, without loss of generality.

[0073] Unless specifically stated otherwise, as is clear from the above discussion, throughout the description, discussion utilizing terms such as "processing," "computing," "calculating," "determining," or "displaying" are understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data that is represented as physical (electronic) quantities within the computer system's memory or registers, or other such information storage, transmission, or display device.

[0074] Certain embodiments described herein include process steps and instructions that are described in the form of algorithms. It should be noted that the process steps and instructions of these embodiments may be implemented as software, firmware, or hardware, and when implemented as software, may also be downloaded to and reside on and operated from various platforms used by real-time network operating systems.

[0075] Finally, the language used herein has been selected primarily for ease of reading and educational purposes, and may not have been selected to define or limit the subject matter of the invention. Thus, the disclosure of these embodiments is intended to be illustrative and not limiting.

Claims

1. A method implemented by at least one computing device, the method comprising: receiving a document to be analyzed; receiving a set of key names extracted from the document; identifying a plurality of character strings contained within the document and the location of each character string of the plurality of character strings; a set of nodes, each node of the set of nodes corresponding to a string of the plurality of strings; and a set of edges respectively connecting one or more nodes of the set of nodes, the set of edges including an edge connecting a first node of the set of nodes to a second node of the set of nodes in response to the second node being within a threshold distance and a threshold angle from the first node; generating a document graph, identifying a set of keys based on the document graph and the received set of key names; extracting a set of values ​​from the document; generating a set of key-value pairs based on the set of keys and the set of values; A method comprising:

2. The method of claim 1 , wherein identifying the plurality of character strings comprises performing optical character recognition (OCR) on the received document.

3. The method of claim 1, further comprising the step of displaying a user interface, wherein receiving the set of key names extracted from the document is performed based on input to the user interface.

4. The method described in claim 1, wherein receiving the document to be analyzed includes receiving an image of the document to be analyzed, and the method further includes a step of converting the image to a predetermined resolution.

5. identifying the set of keys based on the document graph and the received set of key names, identifying a first node from the set of nodes of the document graph based on a first token of a key name, wherein a string corresponding to the first node matches the first token of the key name; identifying a second node from the set of nodes of the document graph based on a second token of the key name, the second node being connected to the first node by an edge from the set of edges of the document graph, and a string corresponding to the second node matching the second token of the key name; 10. The method of claim 1, comprising:

6. The step of extracting a set of values ​​from the document comprises: removing lines from the document to generate a line-removed document; applying a run length smoothing algorithm (RLSA) to the line-smoothed document to identify a set of foreground pixels and a set of background pixels in the line-smoothed document; generating a set of bounding boxes based on the set of foreground pixels in the line-removed document; extracting the set of values ​​based on the generated set of bounding boxes; 10. The method of claim 1, comprising:

7. The step of extracting the set of values ​​based on the generated set of bounding boxes includes: for each bounding box in the set of bounding boxes, determining the number of keys in the set of keys that lie within the bounding box; responsive to the number of keys being greater than one, dividing the bounding box to generate a set of divided bounding boxes, each divided bounding box having one key from the set of keys located within the divided bounding box; 7. The method of claim 6, comprising:

8. A non-transitory computer-readable memory storing executable computer program instructions, the instructions comprising: receiving a document to be analyzed; receiving a set of key names extracted from the document; identifying a plurality of character strings contained within the document and a location of each character string of the plurality of character strings; a set of nodes, each node of the set of nodes corresponding to a string of the plurality of strings; and a set of edges respectively connecting one or more nodes of the set of nodes, the set of edges including an edge connecting a first node of the set of nodes to a second node of the set of nodes in response to the second node being within a threshold distance and a threshold angle from the first node; generating a document graph, identifying a set of keys based on the document graph and the received set of key names; extracting a set of values ​​from the document; generating a set of key-value pairs based on the set of keys and the set of values; a non-transitory computer-readable memory executable to perform operations including:

9. The non-transitory computer-readable memory of claim 8 , wherein identifying the plurality of character strings includes performing optical character recognition (OCR) on the received document.

10. The non-transitory computer-readable memory of claim 8, wherein the operations further include displaying a user interface, and receiving the set of key names extracted from the document is performed based on input to the user interface.

11. identifying the set of keys based on the document graph and the received set of key names, identifying a first node from the set of nodes of the document graph based on a first token of a key name, wherein a string corresponding to the first node matches the first token of the key name; identifying a second node from the set of nodes of the document graph based on a second token of the key name, the second node being connected to the first node by an edge from the set of edges of the document graph, and a string corresponding to the second node matching the second token of the key name; 9. The non-transitory computer-readable memory of claim 8, comprising:

12. Extracting a set of values ​​from the document includes: removing lines from the document to generate a line-removed document; applying a run length smoothing algorithm (RLSA) to the line-smoothed document to identify a set of foreground pixels and a set of background pixels in the line-smoothed document; generating a set of bounding boxes based on the set of foreground pixels in the line-removed document; extracting the set of values ​​based on the generated set of bounding boxes; 9. The non-transitory computer-readable memory of claim 8, comprising:

13. Extracting the set of values ​​based on the set of generated bounding boxes includes: For each bounding box in the set of bounding boxes, determining the number of keys in the set of keys that lie within the bounding box; responsive to the number of keys being greater than one, dividing the bounding box to generate a set of divided bounding boxes, each divided bounding box having one key from the set of keys located within the divided bounding box; 13. The non-transitory computer-readable memory of claim 12, comprising:

14. a computer processor for executing computer program instructions; a non-transitory computer-readable memory storing computer program instructions executable by said computer processor to perform operations, said operations comprising: receiving a document to be analyzed; receiving a set of key names extracted from the document; identifying a plurality of character strings contained within the document and a location of each character string of the plurality of character strings; a set of nodes, each node of the set of nodes corresponding to a string of the plurality of strings; and a set of edges respectively connecting one or more nodes of the set of nodes, the set of edges including an edge connecting a first node of the set of nodes to a second node of the set of nodes in response to the second node being within a threshold distance and a threshold angle from the first node; generating a document graph, identifying a set of keys based on the document graph and the received set of key names; extracting a set of values ​​from the document; generating a set of key-value pairs based on the set of keys and the set of values; a non-transitory computer-readable memory including: A computer system comprising:

15. 15. The computer system of claim 14, wherein identifying the plurality of character strings includes performing optical character recognition (OCR) on the received document.

16. identifying the set of keys based on the document graph and the received set of key names, identifying a first node from the set of nodes of the document graph based on a first token of a key name, wherein a string corresponding to the first node matches the first token of the key name; identifying a second node from the set of nodes of the document graph based on a second token of the key name, the second node being connected to the first node by an edge from the set of edges of the document graph, and a string corresponding to the second node matching the second token of the key name; 15. The computer system of claim 14, comprising:

17. Extracting a set of values ​​from the document includes: removing lines from the document to generate a line-removed document; applying a run length smoothing algorithm (RLSA) to the line-smoothed document to identify a set of foreground pixels and a set of background pixels in the line-smoothed document; generating a set of bounding boxes based on the set of foreground pixels in the line-removed document; extracting the set of values ​​based on the generated set of bounding boxes; 15. The computer system of claim 14, comprising:

18. Extracting the set of values ​​based on the set of generated bounding boxes includes: For each bounding box in the set of bounding boxes, determining the number of keys in the set of keys that lie within the bounding box; responsive to the number of keys being greater than one, dividing the bounding box to generate a set of divided bounding boxes, each divided bounding box having one key from the set of keys located within the divided bounding box; 20. The computer system of claim 17, comprising:

19. The computer system of claim 17, wherein the operation further includes displaying a user interface, and receiving the set of key names extracted from the document is performed based on input to the user interface.

20. The computer system of claim 17, wherein receiving the document to be analyzed includes receiving an image of the document to be analyzed, and the operation further includes converting the image to a predetermined resolution.

Images