Image analysis-based document processing for inference of key-value pairs in non-fixed digital documents

Abstract

An online system extracts information from a non-fixed-format document. The online system receives an image of a form document and obtains a set of phrases and positions of the set of phrases on the form image. For at least one field, the online system determines a key score for the set of phrases. The online system identifies a set of candidate values ​​for the field from the identified set of phrases and identifies a set of neighbors for each candidate value from the identified set of phrases. The online system determines a neighbor score, wherein the neighbor scores of the candidate value and the corresponding neighbor are determined based on the key scores of the neighbors and the spatial relationship of the neighbors to the candidate value. The online system selects the candidate value and the corresponding neighbor as a value and a key for the field based on the neighbor scores.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. utility patent application No. 17 / 140,987, filed on January 4, 2021, the entire disclosure of which is incorporated herein by reference. Background Art

[0003] The present invention relates generally to processing digital documents, and more particularly to using image analysis of digital documents to infer key-value pairs in non-fixed digital documents.

[0004] Entities (e.g., organizations of various types) process many digital documents, which may include information related to the entity's operational processes. Information can be extracted from documents to perform or assist with one or more tasks of the operational processes. Among other types of documents, forms can structure information into a collection of fields, each with one or more key-value pairs. A field can represent a corresponding type of information to be extracted from the document. The key of a field can refer to the label called for the corresponding field on the form document and can vary, for example, depending on the naming convention used by the responsible entity for the form.

[0005] Since entities may frequently process a large number of documents, it is advantageous to automatically extract information from key-value pairs on form documents without a human operator. Analysis of digital documents typically includes receiving an image representation of the document, performing image analysis, optical character recognition, and using artificial intelligence techniques, for example, machine learning techniques, such as deep learning-based neural networks. These techniques train machine learning-based models, such as convolutional neural networks, and apply the trained neural networks to images representing new digital documents. These techniques typically work with prefixed field sets. However, while some types of form documents are standardized and have fixed key-value pair locations, many types of form documents are non-fixed because the type and format of the information varies depending on, for example, the entity that issues the form. This variability in non-fixed forms makes it difficult to automatically extract information. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram of a system environment including an online system according to one embodiment.

[0007] Figure 2 An example document and a set of phrases identified in an invoice document are shown, according to one embodiment.

[0008] Figure 3 According to one embodiment, a method for determining Figure 2 An example of a high-level procedure for key-value pairs in a sample invoice document.

[0009] Figure 4 is a block diagram of the architecture of an online system according to one embodiment.

[0010] Figure 5 A flow chart illustrating a method of determining key-value pairs in a form document according to one embodiment is shown.

[0011] Figure 6 is a diagram showing a method according to one embodiment of the present invention. Figure 1 A block diagram of the architecture of a typical computer system used in an environment.

[0012] These figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present invention described herein.

[0013] The figures use similar reference numbers to identify similar elements. A letter following a reference number, such as "110A," indicates that the text refers specifically to the element with that particular reference number. A reference number in the text without a following letter, such as "110," refers to any or all elements with that reference number in the figures (e.g., "client device 110" in the text refers to reference numbers "client device 110A" and / or "client device 110B" in the figures). DETAILED DESCRIPTION

[0014] Overview

[0015] The online system extracts information from digital documents. In one embodiment, the method employed by the online system allows for the extraction of information from non-fixed format documents that may have varying layouts and arrangements of information.

[0016] Specifically, the online system receives an image of a form document from a client device. The form document may include key-value pairs for a set of fields. In one embodiment, the online system also obtains a template indicating one or more fields to be extracted from the form image, wherein the fields may be associated with a set of candidate keys for the fields. The online system obtains a set of phrases and the positions of the phrases on the form image.

[0017] For at least one field, an online system determines a key score for a set of phrases, wherein the key score for a phrase indicates a likelihood that the phrase is a key for the field on the form image. The online system identifies a set of candidate values ​​for the field from the identified set of phrases, and identifies a set of neighbors for each candidate value from the identified set of phrases. The online system determines a neighbor score, wherein the neighbor scores for the candidate value and the corresponding neighbor are determined based on the key score of the neighbor and the spatial relationship of the neighbor to the candidate value. The online system selects the candidate value and the corresponding neighbor based on the neighbor score, sets the selected candidate value as the value of the field, and sets the selected neighbor as the key for the field. The disclosed technology can automatically process non-fixed forms, i.e., forms with non-fixed structures.

[0018] System environment

[0019] Figure 1 is a block diagram of a system environment 100 including an online system 130 according to one embodiment. Figure 1 The system environment 100 shown in FIG. 1 includes an online system 130, client devices 110A, 110B, and a network 120. In alternative configurations, different and / or additional components may be included in the system environment 100.

[0020] Online system 130 is a system that receives requests to process digital documents and provides information extracted from the digital documents to the requesting user. Entities such as businesses or government organizations process many documents, which may include information related to the entity's operational processes, such as financial transactions, onboarding new employees, or launching new products. Information can be extracted from documents to perform or assist with one or more tasks in the operational process. For example, a document may be an invoice for services provided to a business organization, and this invoice may be processed by the organization's employees to allow payment to the supplier. As another example, the document may be a mortgage application to a lending organization, and this mortgage application may be processed by an underwriter to determine whether the application can be approved.

[0021] As with other types of documents, forms may structure information into a collection of fields, each field having one or more key-value pairs. A field may represent a corresponding type of information to be extracted from the document. For example, a field for invoice number may represent an identifier unique to each invoice. The key of a field may refer to the label that the corresponding field is called on the form document and may vary, for example, depending on the naming convention used by the responsible entity for the form. For example, the key for invoice number for one supplier may be labeled "Invoice#", while the key for invoice number for another supplier may be labeled "InvoiceNumber". The value of a field may refer to the data value for that field on the form document and may follow the format used by the responsible entity for the form. For example, the value for invoice number may be "INV-023-US".

[0022] Typically, a human operator extracts information from a document and performs one or more tasks, such as entering the information into a record or making a payment based on the extracted information. Because entities may frequently process large volumes of documents, it is advantageous to automatically extract information from key-value pairs on form documents without a human operator so that processing can be more efficient. Some types of form documents are standardized and have fixed locations for key-value pairs on the document, which can enable a computerized system to automatically extract information because the locations of the keys and values ​​are known. For example, a driver's license for one state may include information for the same set of fields (e.g., name, address, eye color) and a uniform layout of key-value pairs for these fields across individuals residing in that state.

[0023] However, many types of form documents are fluid, as the type and format of information varies depending on, for example, the entity issuing the form. For example, a mortgage application from one lender might include different types of fields than a mortgage application from another lender, as each lender considers different types of information from applicants. As another example, an invoice from a plumbing supplier might have different key-value pairs for the invoice number and amount owed fields than an invoice from another supplier, both of which must be processed by the same company. Because the type and format of information on these forms is not standardized or fixed, this variation in fluid forms makes automated information extraction difficult.

[0024] Thus, in one embodiment, online system 130 provides a means for extracting information from non-fixed digital documents. Online system 130 receives a request from client device 110 to process a digital document and provides the information extracted from the digital document to the requesting user. In one embodiment, online system 130 is managed by the same entity as the requesting client device 110 and may be an internal system owned by that entity. In another embodiment, online system 130 is managed by a different entity than the requesting client device 110 and may be a separate system. For example, online system 130 may receive a request from client device 110 from an employee of a different company who needs to process a document.

[0025] Figure 2 An example invoice document and a set of phrases identified in the invoice document are shown, according to one embodiment. Figure 3 According to one embodiment, a method for determining Figure 2An example high-level process for processing key-value pairs in an example invoice document is provided below. Specifically, online system 130 receives a request from client device 110 that includes an image of a document. In one example, the requested document is a form document that includes information in the form of key-value pairs for a collection of fields. The image of the document may be received in the form of a computerized image file, such as a JPEG, GIF, PNG, EPIS, AI, PDF, RAW, TIFF, etc., but it should be understood that the document may be received in other formats.

[0026] like Figure 2 As shown in , this example shows a form image 200 for an invoice issued by the company "ABC Service Company." Specifically, image 200 may be a flexible form, as the type and format of the invoice information may vary depending on, for example, the company issuing the invoice. Image 200 includes key-value pairs representing a set of fields to be extracted by online system 130. Specifically, image 200 includes the key-value pairs {"Bill To:,"John Smith"} for the invoice recipient field, {"Invoice #,"US-001"} for the invoice number field, {"Invoice Date,"11 / 2 / 2019"} for the invoice date field, {"Quantity,"2"} for the item quantity field, {"Description,"Front Brake"} for the item description field, {"Unit Price,"100.00"} for the item unit price field, {"Amount,"200.00"} for the item price field, and {"Total Charge,"$200.00"} for the total invoice amount field. These key-value pairs are not yet known and will be extracted by online system 130.

[0027] The online system 130 obtains a set of phrases and positions of the phrases on the document. A phrase may include one or more words that are spatially located close to each other on the document. In one example, the set of phrases and positions may be identified as having a bounding box around each phrase on the document, and the position of the bounding box is determined as the position of the corresponding phrase. For example, Figure 2 As shown in , the online system 130 can identify a set of bounding boxes (indicated by dashed lines) for the phrase set "Bill To:", "John Smith", "Invoice #", "US-001", "Invoice Date", "11 / 2 / 2019", "Quantity", "Description", "Unit Price", "Amount", "2", "Front End Brake", "100.00", "200.00", "Total Charge:", and "$200.00". Each bounding box surrounds a corresponding phrase, and the position of the bounding box (e.g., the position of the center of the box) can be determined as the position of the corresponding phrase on the image 200 of the document.

[0028] For this request, the online system 130 may also obtain a form template, which includes a set of fields to be extracted from the form, and one or more candidate keys or known keys for each field. Specifically, when the label of the key is unknown, the candidate key of the corresponding field is a possible candidate phrase for the key of the field on the document. Figure 2 and Figure 3 In the example of , the online system 130 may obtain a form template that specifies one or more candidate keys {"invoice number", "invoice #", "invoice number"} for the invoice number field, which may be candidate keys that the key for the field is marked as in the form.

[0029] In one embodiment, the online system 130 may receive a customized form template from a requesting user, the customized form template specifying at least a portion of the form template, such as candidate keys for fields. Because the requesting user may be more familiar with the document, receiving the customized form template along with the document allows the online system 130 to access phrases that may be candidates for keys for the document. In another embodiment, the online system 130 may not receive a form template from the request or may receive a partially complete form template and internally determine the set of fields and one or more candidate keys. For example, the online system 130 may determine this information based on the type of form, such as invoices, mortgage applications, documents from different government agencies, etc., by storing templates for previously processed documents based on categories. The online system 130 may determine the appropriate category for the incoming form, retrieve a template from previously processed documents of the same category, and assign the retrieved template as the template for the incoming document.

[0030] For at least one field, the online system 130 determines a key score for a set of phrases, wherein the key score of a phrase indicates the likelihood that the phrase is a key for the field on the image. In one example, the key score of a phrase for a given field is determined based on a match between the phrase and a candidate key for the field, such that a phrase has a higher key score if the phrase is similar to the candidate key for the field. Figure 3 As shown in , for the field of invoice number, the online system 130 determines the key scores of the identified phrase set based on the candidate key of the field. Specifically, the phrase "invoice #" has the highest key score because it substantially or completely matches the candidate key of the field.

[0031] The online system 130 also identifies a set of candidate values ​​for the field from the set of phrases, and identifies a set of neighbors for each candidate value from the set of phrases. In one example, candidate values ​​are determined by first identifying the data type of the field value (e.g., cardinality, date, text string) and selecting only those phrases that match the data type. Figure 3As shown in , the online system 130 identifies the data type of the invoice number field as a text string or a cardinality, and selects a subset of phrases that include numbers or text, including "100.00", "US-001", "11 / 2 / 2019", and "2" as candidate values ​​for the field.

[0032] The online system 130 determines for each candidate value a neighbor that is spatially close to the candidate value on the document. Figure 2 In the example, the neighbor set of the candidate value "100.00" may include "unit price", "200.00", and "front brake" because their spatial locations on the document are close to the location of the candidate value. The online system 130 determines a neighbor score, wherein the neighbor score of the candidate value and the corresponding neighbor is determined based on the key score of the neighbor and the spatial relationship between the neighbor and the candidate value. Figure 3 As shown in , the online system 130 identifies neighbor scores for candidate value-neighbor pairs, and among these scores, the candidate value "US-001" and its neighbor "Invoice #" have the highest neighbor score of 0.92 because the neighbor "Invoice #" is associated with a high key score and is spatially close to the candidate value "US-001".

[0033] The online system 130 selects a candidate value and a corresponding neighbor based on the neighbor score, and sets the selected candidate value as the value of the field, and sets the selected neighbor as the key of the field. Figure 2 and Figure 3 In the example shown in FIG, the online system 130 selects the invoice number field with a key of "invoice#" and a value of "US-001" because the neighbor score for this pair is the highest among the other pairs for this field. The online system 130 can repeat this process for other fields specified in the document template to extract the remaining information in the form of key-value pairs.

[0034] return Figure 1 , the client device 110 is a computing device, such as a computer having an operating system (such as or ) smartphone, tablet, laptop, desktop computer, or any other type of network-enabled device. A typical client device 110 includes the hardware and software required to connect to the network 122 (e.g., via WiFi and / or 4G, 5G, or other wireless telecommunications standards). The client device 110 allows a user to submit a request to the online system 130 to extract information from a document. The client device 110 may include an operating system and various applications running on the operating system that enable the user to submit requests. For example, the client device 110 may include a browser application or a standalone application deployed by the online system 130 that allows users of the organization to interact with the online system 130 to submit requests.

[0035] The network 122 provides the communication infrastructure between the worker devices 110 and the process mining system 130. The network 122 is typically the Internet, but can be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network.

[0036] Online System

[0037] Figure 4 is a high-level block diagram illustrating a detailed view of an online system according to one embodiment. The online system 130 is composed of modules including a request management module 410, a template management module 415, an identification module 420, and a key-value identifier module 425. The online system 130 also includes a form and template data store 450 and a key-value data store 455. Some embodiments of the online system 130 may have different modules than those described here. Similarly, functionality may be distributed among the modules in a manner different from that described here.

[0038] Request management module 410 receives a request from client device 110 to process a digital document and provides the extracted information to the requesting user. Specifically, request management module 410 may receive a request including an image of a document and may apply pre-processing techniques to the image before extracting information from the document. For example, request management module 410 may perform cropping, image enhancement techniques, scaling, translation, or rotation of the document and provide the pre-processed image to recognition module 420. Request management module 410 may store the document in form and template data store 450.

[0039] In addition, the request may also include a customized document template from the requesting user, the request specifying one or more fields to be extracted from the document and any candidate keys or known keys for the fields. Request management module 410 may receive the customized document template and forward the template to template management module 415. Request management module 410 may also store any customized templates received in form and template data store 450.

[0040] In response to receiving information extracted from the requested document from a module of the online system 130, the request management module 410 provides the information to the requesting user. Specifically, the request management module 410 can receive the extracted information in the form of key-value pairs that can be provided to the user in an appropriate format. In one example, the request management module 410 provides the key-value pairs as text in a text file. In another example, the request management module 410 visually provides the key-value pairs by annotating the document with the location of the identified key-value pairs. The annotation can be in the form of a rectangular bounding box surrounding the key-value pair, or in the form of a segment outlining the actual text of the key-value pair in the document.

[0041] The template management module 415 creates and manages document templates. In one embodiment, the template management module 415 receives a custom document template attached with a request and may flag the custom template for any errors or incomplete information. For example, a template may include a field set, but may lack candidate keys for one or more fields of the template. In this case, the template management module 415 may generate candidate keys for the fields based on, for example, previous instances of documents processed by the online system 130. For example, the template management module 415 may generate candidate keys for the invoice number field based on keys previously identified by the online system 130 for invoice documents.

[0042] In another embodiment, the template management module 415 may determine that the requested document is not associated with any form template. In this case, the template management module 415 may create a template for the document based on, for example, previous instances of the document processed by the online system 130. For example, the template management module 415 may generate a form template for the document based on templates generated or received for documents of the same type (e.g., invoices, applications, government forms), documents originating from the same issuing entity, or documents of the same authorship. The template management module 415 stores the templates in the form and template data store 450, with each template associated with a corresponding requested document.

[0043] The recognition module 420 receives the document included in the request and identifies a set of phrases and the locations of the phrases on the document. A phrase can be defined as a grouping of one or more words that are placed close to each other in space on the document. In one embodiment, a group of one or more words is identified as a phrase on the document if the horizontal distance between the words is less than a predetermined threshold. The recognition module 420 can perform a text recognition method, such as optical character recognition (OCR) or the application of a machine learning model to identify words and word groups as phrases on the document. The recognition module 420 also associates each phrase with a location on the document. For example, the recognition module 420 can generate a bounding box that encloses the phrase and determine the spatial coordinates of the bounding box as the location of the phrase. The spatial coordinates can be defined as {x min ,y min , x max ,y max}, where x min is the leftmost horizontal coordinate of the bounding box, y min is the lowest vertical coordinate, x max is the rightmost horizontal coordinate, y max is the topmost vertical coordinate.

[0044] The key-value identifier module 425 receives a document and a set of phrases and templates for the document and extracts information in the form of key-value pairs. For at least one field specified in the template, the key-value identifier module 425 determines a key score for the set of phrases for the corresponding document. In one embodiment, the key-value identifier module 425 determines the key score for a phrase for a given field based on a string match between the phrase and the candidate keys for the field. In one example, the string match is a fuzzy match between the phrase and the candidate keys for the field, and the key-value identifier module 425 generates a match score between the phrase and the candidate keys for the field, which indicates the similarity between the two texts. The key score for a phrase is the maximum match score between the candidate keys.

[0045] The key-value identifier module 425 also identifies a set of candidate values ​​for the field from the set of phrases in the document. In one embodiment, the key-value identifier module 425 determines the data type of the field value and selects only those phrases that match the data type. In one example, the key-value identifier module 425 generates a set of categories and labels the field with one or more categories, including but not limited to a person's name, organization, location, cardinality, medical code, time expression (e.g., date or time), quantity, monetary value, percentage, etc. The key-value identifier module 425 applies a named entity recognizer (NER) model to determine whether a phrase belongs to a category that matches one or more categories of the field and can select only those phrases that match as candidate values ​​for the field.

[0046] The key-value identifier module 425 identifies a set of neighbors for each candidate value from a set of phrases that are spatially proximate to the candidate value on the document. In one embodiment, the neighboring phrases of a candidate value are phrases that have spatial locations within a predetermined distance from the location of the candidate value on the document. For example, the neighboring phrases of a candidate value may be phrases that have a bounding box that significantly overlaps the area of ​​the bounding box of the corresponding candidate value, for example, by more than 90%, 80%, or 70%.

[0047] The key-value identifier module 425 determines a neighbor score for a candidate value-neighbor pair, wherein the neighbor score of the candidate value and the corresponding neighbor is determined based on the key score of the neighbor and a spatial score indicating the spatial relationship of the neighbor to the candidate value. In one embodiment, the spatial score is given by a combination of a distance score and an angle score, for example, a weighted sum of the distance score and the angle score, and can be given by:

[0048] R (candidate i ,Neighbor j )

[0049] = distance (candidate i ,Neighbor j )+α·angle(candidate i ,Neighbor j)

[0050] in

[0051] Distance (candidate i ,Neighbor j )=Gauss(distance ij |μ=0,δ=z1)

[0052] Angle (candidate i ,Neighbor j )=Gauss(angle ij |μ=0,δ=z2)

[0053] Among them, the distance ij is the distance between the location of candidate i on the document (e.g., the center of the bounding box) and the location of neighbor j, and the angle ij is the angle between the candidate's position and the neighbor's position on the document, and α is a parameter.

[0054] Specifically, the key-value identifier module 425 determines the distance score as the distance between the location of the candidate value and the neighbor on the document (distance ij ), where in one example, the function is a Gaussian (or any other probability distribution) that receives the distance as input and is centered around mean 0 and standard deviation z1. Similarly, the key-value identifier module 425 determines the angle score as the angle (angle ) between the position of the candidate value and the position of the neighbor on the document. ij ), where in one example the function is a Gaussian (or any other probability distribution) that receives as input an angle and is centered around mean 0 and standard deviation z2. In one example, given an angle (angle ij ) is the minimum angular distance between the position of candidate i (e.g., the center of the bounding box) and the position of neighbor j relative to a set of anchor angles. For example, if key-value pairs may be arranged horizontally from left to right on the document (e.g., an anchor angle of 0°) or vertically up and down (e.g., an anchor angle of 90°), the set of anchor angles may be {0°, 90°}. Therefore, the spatial score increases for candidate-neighbor pairs that are close to each other and aligned with each other on the document.

[0055] In one example, the neighbor score ns(·) of the candidate value and the corresponding neighbor is given by:

[0056] ns(candidate i ,Neighbor j )=R(candidate i ,Neighbor j )×key score j

[0057] Among them, the key score jis the key score of neighbor j. In other words, the neighbor score ns(·) is given by the combination of the neighbor's key score and the spatial score between the candidate value and the neighbor. The key-value identifier module 425 can determine the final ranking score of the candidate value as the maximum of its neighbor scores.

[0058] The key-value identifier module 425 selects a candidate value and a corresponding neighbor based on the ranking score, and sets the selected candidate value as the value of the field and the selected neighbor as the key of the field. For example, the key-value identifier module 425 may select the candidate value-neighbor pair with the highest ranking score as the final key-value pair for the field. The key-value identifier module 425 repeats the process for other fields specified in the document template to extract the remaining information in the form of key-value pairs. The key-value identifier module 425 stores the extracted information in the key-value pair data store 455 and provides the extracted information to the request management module 410 so that the request management module 410 can provide the information to the requesting user.

[0059] Method for determining key-value pairs from digital documents

[0060] Figure 5 A flow chart of a method for determining key-value pairs in a form document according to one embodiment is shown. In one embodiment, various modules of the online system 130 perform Figure 5 In other embodiments, other entities may perform some or all of the steps of the process. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0061] The online system 130 receives 502 a form image from a client device and obtains a template indicating one or more fields to be extracted from the form image. At least one field is associated with a set of candidate keys for the field. The online system 130 obtains 504 a set of phrases from the form image and obtains the positions of the phrases on the document. For the at least one field, the online system 130 determines 506 a key score for the phrase set. The key score for the phrase may indicate the likelihood that the phrase is a key for the field on the document. The online system 130 identifies 508 a set of candidate values ​​for the field from the phrase set and identifies a set of neighbors for each candidate value from the phrase set.

[0062] The online system 130 determines 510 neighbor scores for the set of candidate values ​​and the set of neighbors. The neighbor scores for the candidate values ​​and corresponding neighbors can be determined based on the neighbor's key score and the neighbor's spatial relationship to the candidate value. The online system 130 selects 512 the candidate value and corresponding neighbor associated with a neighbor score above a threshold, sets the selected candidate value as the value of the field, and sets the selected neighbor as the key of the field. The online system 130 can repeat steps 506 through 512 for the remaining fields specified in the document template.

[0063] Computer architecture

[0064] Figure 6 is a diagram showing a method according to one embodiment of the present invention. Figure 1 6. FIG. 6 is a block diagram of the architecture of a typical computer system for use in an environment. At least one processor 602 is shown coupled to a chipset 604. Also coupled to chipset 604 are memory 606, storage device 608, keyboard 610, graphics adapter 612, pointing device 614, and network adapter 616. A display 618 is coupled to graphics adapter 612. In one embodiment, the functionality of chipset 604 is provided by memory controller hub 620 and I / O controller hub 622. In another embodiment, memory 606 is coupled directly to processor 602 rather than chipset 604.

[0065] Storage device 608 is a non-transitory computer-readable storage medium, such as a hard drive, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device. Memory 606 stores instructions and data used by processor 602. Pointing device 614 may be a mouse, trackball, or other type of pointing device and is used in conjunction with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. Network adapter 616 connects computer system 600 to a network.

[0066] As is known in the art, the computer 600 may have Figure 6 6. Furthermore, the computer 600 may lack certain components shown. For example, the computer system 600 acting as the online system 130 may lack the keyboard 610 and the pointing device 614. Furthermore, the storage device 608 may be local and / or remote from the computer 600 (e.g., embodied in a storage area network (SAN)).

[0067] Computer 600 is adapted to execute computer modules for providing the functionality described herein. As used herein, the term "module" refers to computer program instructions and other logic for providing a specified functionality. Modules may be implemented in hardware, software, and / or firmware. A module may include one or more processes and / or may be provided by only a portion of a process. Modules are typically stored on storage device 608, loaded into memory 606, and executed by processor 602.

[0068] Figure 1The type of computer system 600 used by the entity may vary depending on the implementation and the processing power used by the entity. For example, the client device 110 may be a mobile phone with limited processing power, a small display 618, and may lack a pointing device 614. In contrast, the online system 130 may include multiple blade servers working in concert to provide the functionality described herein.

[0069] Other considerations

[0070] The foregoing description of the embodiments of the present invention is presented for illustrative purposes; it is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0071] Some parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to effectively convey the essence of their work to other persons skilled in the art. Although these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without losing versatility, it is sometimes convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0072] Any steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0073] Embodiments of the present invention may also relate to apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to in the specification may comprise a single processor, or may be an architecture employing multiple processor designs to increase computing power.

[0074] Embodiments of the present invention may also relate to products produced by the computing processes described herein. Such products may include information produced by the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0075] Finally, the language used in the specification is selected primarily for readability and instructional purposes and may not be selected to describe or define the subject matter of the present invention. Therefore, the scope of the present invention is not intended to be limited by this detailed description, but rather by any claims issued based on this application. Therefore, the disclosure of the embodiments of the present invention is intended to illustrate, but not to limit, the scope of the invention as set forth in the following claims.

Claims

1. A computer-implemented method comprising: receiving a form image from a client device; obtaining a template indicating one or more fields to be extracted from the form image, wherein at least one field is associated with a set of candidate keys for said field; Obtain a set of phrases from the form image and obtain positions of the phrases; and For the at least one field: determining a key score for a phrase from the set of phrases, the key score for a phrase indicating a likelihood that the phrase is a key for the field on the form, identifying a set of candidate values ​​for the field from the set of phrases, identifying a set of neighbors for candidate values ​​from the set of phrases, determining neighbor scores for the set of candidate values ​​and the set of neighbors, the neighbor scores of candidate values ​​and corresponding neighbors being determined from the key scores of the neighbors and the spatial relationships of the neighbors to the candidate values, Select the candidate values ​​and corresponding neighbors associated with neighbor scores above a threshold, and The selected candidate value is set as the value of the field, and the selected neighbor is set as the key of the field. 2 . The computer-implemented method of claim 1 , wherein the key score for the phrase is determined by performing a string match between the phrase and the candidate key associated with the field to determine a similarity between the phrase and the candidate key. 3 . The computer-implemented method of claim 1 , wherein neighbors of a candidate value have locations on the form image that are within a threshold distance from a location of the candidate value. 4 . The computer-implemented method of claim 1 , wherein at least a portion of the information included in the template for the form image is received from the client device.

5. The computer-implemented method of claim 1 , wherein identifying the set of candidate values ​​further comprises: determining a data type of the at least one field by assigning the at least one field to at least one of a predetermined set of categories, and A subset of phrases in the phrase set that match the data type of the at least one field is selected as candidate values ​​for the at least one field.

6. A computer-implemented method according to claim 1, wherein the spatial relationship between the neighbor and the candidate value is determined based on a combination of the distance between the position of the candidate value and the position of the corresponding neighbor on the form image and the angle between the position of the candidate value and the position of the corresponding neighbor on the form image. 7 . The computer-implemented method of claim 1 , further comprising providing the key for the at least one field and the value for the at least one field to a user of the client device.

8. A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations comprising: receiving a form image from a client device; obtaining a template indicating one or more fields to be extracted from the form image, wherein at least one field is associated with a set of candidate keys for said field; Obtain a set of phrases from the form image and obtain positions of the phrases; and For the at least one field: determining a key score for a phrase from the set of phrases, the key score for a phrase indicating a likelihood that the phrase is a key for the field on the form, identifying a set of candidate values ​​for the field from the set of phrases, identifying a set of neighbors for candidate values ​​from the set of phrases, determining neighbor scores for the set of candidate values ​​and the set of neighbors, the neighbor scores of candidate values ​​and corresponding neighbors being determined from the key scores of the neighbors and the spatial relationships of the neighbors to the candidate values, Select the candidate values ​​and corresponding neighbors associated with neighbor scores above a threshold, and The selected candidate value is set as the value of the field, and the selected neighbor is set as the key of the field.

9. The non-transitory computer-readable storage medium of claim 8, wherein the key score for the phrase is determined by performing a string match between the phrase and the candidate key associated with the field to determine a similarity between the phrase and the candidate key. 10 . The non-transitory computer-readable storage medium of claim 8 , wherein a neighbor of a candidate value has a location on the form image that is within a threshold distance from a location of the candidate value. 11 . The non-transitory computer-readable storage medium of claim 8 , wherein at least a portion of the information included in the template of the form image is received from the client device.

12. The non-transitory computer-readable storage medium of claim 8, wherein identifying the set of candidate values ​​further comprises: determining a data type of the at least one field by assigning the at least one field to at least one of a predetermined set of categories, and A subset of phrases in the phrase set that match the data type of the at least one field is selected as candidate values ​​for the at least one field.

13. A non-temporary computer-readable storage medium according to claim 8, wherein the spatial relationship between the neighbor and the candidate value is determined based on a combination of the distance between the position of the candidate value on the form image and the position of the corresponding neighbor and the angle between the position of the candidate value on the form image and the position of the corresponding neighbor.

14. The non-transitory computer-readable storage medium of claim 8, the operations further comprising providing the key for the at least one field and the value for the at least one field to a user of the client device.

15. A system comprising: a processor for executing computer program instructions; and A non-transitory computer-readable storage medium storing computer program instructions executable to perform steps comprising: receiving a form image from a client device; obtaining a template indicating one or more fields to be extracted from the form image, wherein at least one field is associated with a set of candidate keys for the fields; Obtain a set of phrases from the form image and obtain positions of the phrases; and For the at least one field: determining a key score for a phrase from the set of phrases, the key score for a phrase indicating a likelihood that the phrase is a key for the field on the form, identifying a set of candidate values ​​for the field from the set of phrases, identifying a set of neighbors for candidate values ​​from the set of phrases, determining neighbor scores for the set of candidate values ​​and the set of neighbors, the neighbor scores of candidate values ​​and corresponding neighbors being determined from the key scores of the neighbors and the spatial relationships of the neighbors to the candidate values, Select the candidate values ​​and corresponding neighbors associated with neighbor scores above a threshold, and The selected candidate value is set as the value of the field, and the selected neighbor is set as the key of the field.

16. The system of claim 15, wherein the key score for the phrase is determined by performing a string match between the phrase and the candidate key associated with the field to determine a similarity between the phrase and the candidate key.

17. The system of claim 15, wherein a neighbor of a candidate value has a location on the form image that is within a threshold distance from a location of the candidate value.

18. The system of claim 15, wherein at least a portion of the information included in the template of the form image is received from the client device.

19. The system of claim 15, wherein identifying the set of candidate values ​​further comprises: determining a data type of the at least one field by assigning the at least one field to at least one of a predetermined set of categories, and A subset of phrases in the phrase set that match the data type of the at least one field is selected as candidate values ​​for the at least one field.

20. A system according to claim 15, wherein the spatial relationship between the neighbor and the candidate value is determined based on a combination of the distance between the position of the candidate value on the form image and the position of the corresponding neighbor and the angle between the position of the candidate value on the form image and the position of the corresponding neighbor.

Images