Data extraction system and method

Abstract

The present invention relates to a data extraction system and method for extracting data from documents, the data extraction system including a data extraction application, the data extraction application including a trained document classification model, and computer program code that, when executed by a processor of an electronic device, causes the processor to present a user interface on the electronic device that prompts a user to capture an image of a document, submit the captured image of the document to the document classification model, receive from the document classification model a document category value for the document, process the document category value to select a next data extraction operation to be performed by the data extraction application, and initiate the selected data extraction operation.

Description

[Technical Field]

[0001] The present invention relates generally to data extraction systems and methods, and more particularly to systems and methods for controlling electronic devices to extract data from official documents (including passports, etc.) for purposes such as identity verification and check-in automation. [Background technology]

[0002] Any discussion of documents, devices, acts, or knowledge in this specification is included for the purpose of explaining the context of the invention and is not an admission that any material formed part of the prior art or common general knowledge in the art prior to the priority date of the claims herein.

[0003] The travel industry is moving toward more automated and contactless procedures for customers to check in for selected modes of transportation, such as international flights. One approach to automated check-in involves customers utilizing a software application on their device to scan an official document (e.g., a passport) that would otherwise be manually checked by a check-in agent. This application extracts data from the official document and transmits the data to the airline's check-in system for verification and action. In a process known as "biometric enrollment," the data extraction application captures biometric information, typically in the form of a facial scan taken from the customer's photograph on the official document, in addition to extracting bibliographic data. To verify their identity, customers can submit an additional digital photograph through the data extraction application, and the check-in system compares the photograph to the recorded facial scan.

[0004] The biometric registration system allows customers to check in for international flights even before arriving at the airport, and then board the plane simply by having their photograph taken by the airport security system. In this regard, the airport security system can verify the customer's identity and check-in status by comparing the pre-boarding photograph with stored biometric information.

[0005] Current data extraction applications are somewhat cumbersome for customers to use and include a large number of on-screen instructions that customers may find frustrating and difficult to follow. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] "Machine Readable Travel Documents" specification (Doc 9303) Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention aims to provide a data extraction system and method that is more convenient for customers to use. According to the present invention, there is provided a data extraction system and method as defined in the claims. [Means for solving the problem]

[0008] According to one aspect of the present disclosure, a data extraction system is provided, the data extraction system including a data extraction application, the data extraction application including a trained document classification model; and computer program code that, when executed by a processor of an electronic device, causes the processor to present a user interface on the electronic device that prompts a user to capture an image of a document, submit the captured image of the document to the document classification model, receive a document category value for the document from the document classification model, process the document categories to select a next data extraction operation to be performed by the data extraction application, and initiate the selected data extraction operation.

[0009] The present disclosure provides a data extraction application that utilizes an on-device document classification machine learning model to classify official documents from captured images of the official documents and control the subsequent data extraction process based on the classification. This eliminates the need for travelers to manually enter the document type into the application (e.g., by selecting the document type from a list), and also improves the efficiency of the overall data extraction operation.

[0010] Additionally, performing document classification on the client side (using a local document classification model) improves processing time compared to solutions that involve sending captured document images to a backend server for processing. Client-side document classification also allows applications to perform necessary quality checks without sending the data to a backend server.

[0011] Preferably, the processor selects a next data extraction operation from a set of data extraction operations, the set including presenting a retry user interface instructing the user to capture another image of the document, performing optical character recognition on the document, presenting a manipulation user interface instructing the user to physically manipulate the document, and presenting an extraction success user interface informing the user that data extraction from the document was successful.

[0012] Typically, the successful extraction user interface presents the data extracted from the document on the electronic device.

[0013] The successful extraction user interface may include a confirmation user interface element that the user can operate to confirm that the data extracted from the document is correct.

[0014] Preferably, the operational user interface instructs the user to present another side of the document to the electronic device for capture. According to this embodiment, the computer program code may further cause the processor to perform an image capture routine on the other side of the document once captured by the electronic device. Alternatively, the computer program code may further cause the processor to perform optical character recognition on the other side of the document once captured by the electronic device.

[0015] In other embodiments, the operational user interface instructs the user to present another portion of the same side of the document to the electronic device for capture.

[0016] In some embodiments, the computer program code further causes the processor to input the features of the document classification model as heuristics into a heuristic filtering algorithm, and to run the heuristic filtering algorithm on the captured image before performing optical character recognition on the captured image.

[0017] The features of a document classification model typically include the presence of machine-readable zones and the structure of the machine-readable zones.

[0018] In a preferred embodiment, the data extraction application and the document classification model are contained in a web application that is executable by a web browser installed on the electronic device.

[0019] According to another aspect of the present disclosure, a method for extracting data from a document is provided, the method including: delivering a data extraction application and a trained document classification model to an electronic device; the data extraction application presenting a user interface on the electronic device that prompts a user to capture an image of the document; the data extraction application submitting the captured image of the document to the document classification model; the data extraction application receiving a document category value for the document from the document classification model; the data extraction application processing the document category value to select a next data extraction operation to be performed by the data extraction application; and the data extraction application initiating the selected data extraction operation.

[0020] In one embodiment, the data extraction application selects a next data extraction operation from a set of data extraction operations, the set including presenting a retry user interface that instructs the user to capture another image of the document, performing optical character recognition on the document, presenting a manipulation user interface that instructs the user to physically manipulate the document, and presenting an extraction success user interface that notifies the user that data extraction from the document was successful.

[0021] Preferably, the successful extraction user interface presents the data extracted from the document on the electronic device.

[0022] The successful extraction user interface may include a confirmation user interface element that the user can operate to confirm that the data extracted from the document is correct.

[0023] Optionally, the operational user interface prompts the user to present another side of the document to the electronic device for capture.

[0024] In other embodiments, the data extraction application performs an image capture routine on another side of the document once captured by the electronic device.

[0025] In other embodiments, the data extraction application performs optical character recognition on another side of the document once captured by the electronic device.

[0026] The operational user interface typically prompts the user to present different portions of the same side of the document to the electronic device for capture.

[0027] Preferably, the document extraction application inputs the features of the document classification model as heuristics into a heuristic filtering algorithm, and runs the heuristic filtering algorithm on the captured image before performing optical character recognition on the captured image.

[0028] The features of the document classification model may include the presence of machine-readable zones and the structure of the machine-readable zones.

[0029] Preferably, the data extraction application and the document classification model are contained in a web application executable by a web browser installed on the electronic device.

[0030] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic diagram of a computing environment in which aspects of the present invention may be implemented. [Figure 2] FIG. 2 is a block diagram illustrating modules of a data extraction application, according to one embodiment of the present invention. [Figure 3] 4 is a flowchart illustrating a data extraction and transmission process performed by a data extraction application according to one embodiment of the present invention. [Figure 4] FIG. 2 illustrates a user interface generated by a data extraction application, according to one embodiment of the present invention. [Figure 5] FIG. 10 illustrates a detailed verification user interface generated by a data extraction application, according to one embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating a card flip operation performed by a data extraction application, according to one embodiment of the present invention. [Figure 7] FIG. 10 illustrates a card flip user interface generated by a data extraction application, according to one embodiment of the present invention. [Figure 8] FIG. 10 illustrates the illumination correction and binarization operations performed by the data extraction application, according to one embodiment of the present invention. [Figure 9] FIG. 2 illustrates the CCL and clustering operations performed by a data extraction application according to one embodiment of the present invention. [Figure 10] FIG. 2 illustrates a heuristic filtering operation performed by a data extraction application, according to one embodiment of the present invention. [Figure 11] FIG. 2 illustrates an exemplary homography calculation performed by a data extraction application, according to one embodiment of the present invention. [Figure 12] FIG. 1 is a block diagram of a computer system suitable for implementing one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description. The exemplary embodiments described in the detailed description and illustrated in the drawings are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit and scope of the presented subject matter. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated by the present disclosure.

[0033] 1 illustrates a computing environment 100 in which aspects of the present invention may be implemented. Environment 100 is a network environment comprising an automated check-in server 102 in communication with a client system 104 via one or more communications networks 106. Aspects of the computer processing described below are performed by a server application 108 executing on automated check-in server 102 and a data extraction application 112 executing on client system 104.

[0034] Automated check-in server 102 further includes data storage 110 where data collected by data extraction application 112 and transmitted to auto check-in server 102 is stored. Data storage 110 is typically a storage medium such as a hard drive (or a collection of hard drives). A database management system (not shown) running on auto check-in server 102 implements a database on data storage 110 for storing and retrieving data.

[0035] The automatic check-in server 102 is shown as a single system. However, the automatic check-in server 102 can be a scalable server system with multiple nodes that can be delegated / de-delegated based on processing demands. Typically, the server system is a server computer that provides greater resources (e.g., processing, memory, network bandwidth, etc.) compared to the client system.

[0036] In the illustrated embodiment, data storage 110 is shown as part of the automatic check-in server. However, data storage 110 may also be a separate system in operable network communication with automatic check-in server 102. For example, data storage may be a network-attached storage device, an entirely separate storage system accessed via a database management system, or any other suitable data storage mechanism.

[0037] As described in further detail below, server application 108 performs various operations in response to commands received from (and initiated by) data extraction application 112. Thus, when executed by auto check-in server 102, server application 108 configures auto check-in server 102 to provide server-side functionality for data extraction operations. To provide this functionality, server application 108 comprises one or more appropriate application programs, libraries, or other software infrastructure.

[0038] If the data extraction application 112 is a web application executed by a web browser, the server application 108 is typically a web server, such as a server implemented in the node.js runtime environment, or interacts with it. If the data extraction application 112 is a native application of the client system 104, the server application 108 is typically an application server, or interacts with it. The automated check-in server system 102 may be provided with both a web server and an application server application, so that it can serve both web browsers and native client applications.

[0039] The automated check-in server 102 and the client systems 104 communicate data with each other directly or indirectly over one or more communications networks 106. The communications networks 106 may include a local area network (LAN), a public network (such as the Internet), or a combination of networks.

[0040] Although only one client system 104 is shown in environment 100, a typical environment will usually include many more client systems served by automated check-in server 102.

[0041] Client system 104 can be any type of computer system, including a desktop or laptop computer, but is more commonly a smartphone or tablet device with a built-in or connected camera. When executed by client system 104, data extraction application 112 provides client-side data extraction functionality and configures client system 104 to interact with automated check-in server 102.

[0042] As described above, the data extraction application 112 may be provided to the client system 104 as a web application executed by a common web browser application (e.g., Chrome, Edge, Safari, etc.) installed thereon. When provided as a web application, the data extraction application 112 accesses the server application 108 via an appropriate uniform resource locator (URL) and communicates with the server application 108 via common World Wide Web protocols (e.g., http, https, ftp, etc.) and application programming interfaces (APIs) (e.g., REST API, etc.). Alternatively, when the data extraction application 112 is a native application, it is typically programmed to communicate with the server application 108 using defined API calls.

[0043] A given client system 104 may have multiple client applications 112 installed, for example, both general web browser applications and dedicated program client applications.

[0044] As described below, a web application implementation may have certain advantages over a native application implementation, in particular, a web application implementation may be more easily integrated with the backends of various automated check-in systems offered by different airlines and airports.

[0045] As described below, the data extraction application 112 utilizes a document classification model 116 to perform data extraction operations. The document classification model 116 is a model that results from training a machine learning network, such as a convolutional neural network. The specific training performed to build the document classification model 116 is described below.

[0046] The server application 108 preferably provides the document classification model 116 to the client system 104 simultaneously with the code for the data extraction application 112. For example, when the data extraction application 112 is a web application, the server application 108 sends the document classification model 116 to the client system 104 along with the Javascript code for the data extraction application 112. Providing the document classification model 116 to the client system 104 allows the data extraction application 112 to perform inference locally, reducing processing time and improving reliability.

[0047] The data extraction application 112 includes several software modules, which are described below with reference to FIG.

[0048] The data extraction application 112 utilizes the document classification model 116 to perform inference on images of official documents. As described below, a customer operates the camera 120 of the client system 104 to capture images of official documents, typically by following instructions provided by the data extraction application 112 on the client system's display. The document classification model 116 is trained to recognize different types of official documents.

[0049] For example, the "Machine Readable Travel Documents" specification (Doc 9303), maintained by the International Civil Aviation Organization, promulgates standards for Machine Readable Travel Documents (MRTDs) to ensure global interoperability. MRTDs encode much of the relevant data in optical character recognition (OCR) format. The OCR-encoded information is located in an area of ​​the travel document known as the Machine Readable Zone (MRZ). According to Doc 9303, there are three standardized document types (TD1, TD2, TD3), defined with reference to the location of the MRZ within the document.

[0050] The document classification model 116 is trained to recognize documents that do not necessarily conform to the Doc9303 standard. For example, the document classification model 116 is trained on (and therefore can recognize) the MRZ of one line of a Chinese identity card.

[0051] TD1 is primarily used for identity documents. The MRZ is located on the back of the TD1 document, so data extraction requires capturing both the front and back of the document. Each issuing country can add optional content to the document, which is typically added to the back of the document adjacent to the MRZ.

[0052] The MRZ for a TD1 document spans three lines, each of 30 characters. Standardized data elements are contained in the MRZ along with one or more check digits (for data validation) and any optional information.

[0053] TD2 documents are also used for identification purposes and are larger than TD1 documents. The MRZ is located on the front of the TD2 document and contains two lines of 35 characters each. Like TD1 documents, the MRZ on TD2 documents contains standardized data elements, a check digit, and optional information.

[0054] TD3 documents are used in most travel passports issued by most issuing authorities. TD3 documents are typically in booklet format, but the document contains cards with information. TD3 documents are larger in size compared to TD1 documents and contain two MRZ lines of 44 characters each. Like TD1 and TD2 documents, the MRZ on TD3 documents contains standardized fields, a check digit, and optional information. TD3 documents also contain a photograph of the document owner located in the document's visual inspection zone (VIZ).

[0055] Using the training dataset and algorithms described below, the exemplary document classification model 116 can predict whether an image of a document falls into one of the following categories: - Front of TD1 document - Back of TD1 document - TD3 Document - Chinese ID card

[0056] The document classification model 116 outputs the results of its prediction operation as a data value of type document category. Exemplary document categories are TD1_Front, TD1_Back, TD3, and China. Those skilled in the art will appreciate that with proper training, the document classification model 116 can function to predict documents of other categories.

[0057] The document classification model 116, like the data extraction application 112, is distributed to the server application 108 (for eventual delivery to the client system 104) in the form of portable pre-compiled binary code such as WebAssembly (or WASM).

[0058] The data extraction application 112 further includes a preprocessing module 118. The preprocessing module 118 includes computer-executable code for preprocessing document images before performing OCR on the MRZ text. As described below, the preprocessing module 118 includes several submodules that enable the data extraction application 112 to locate MRZs within document images. In the illustrated embodiment, these submodules are: - Lighting Correction Module 120 - Connected Component Labeling (CCL) Module 122 - Fix module 124 - Binarization Module 126 - Heuristic-based filtering module 128

[0059] The data extraction application 112 further includes an information parser module 130. The information parser module 130 includes computer-executable code for performing OCR on the text of the MRZ identified by the preprocessing module 118. In addition to the OCR module 132, the information parser module 130 includes a regular expression (Regex) module 134 for processing the OCR text generated by the OCR module 130. As described below, the Regex module 134 utilizes document category values ​​when processing the OCR text.

[0060] FIG. 3 conceptually illustrates one embodiment of a computer-executable process 300 performed by data extraction application 112 to enable a traveler to scan an official document and transmit the extracted data to automated check-in server 102.

[0061] The process begins at step 302, when data extraction application 112 captures an image of an official document using the camera of client system 104. In the illustrated embodiment, data extraction application 112 presents a user interface 400 ( FIG. 4 ) on the display of client system 104 that includes textual and graphical instructions to the traveler for manipulating the camera to locate the official document within a frame 402 of the user interface. User interface 400 also includes a button 404 that the traveler operates to add the boarding pass to the data sent to automated check-in server 101.

[0062] If the data extraction application 112 detects that the traveler has photographed an official document, it proceeds to step 304 where the data extraction application 112 inputs the captured document image into the document classification model 116 .

[0063] In step 306, the document classification model 116 performs an initial inference operation on the captured image to determine whether the model recognizes the image as one of the official documents. If the document classification model 116 does not recognize the image as an official document, the process proceeds to step 307, where the data extraction application displays a retry user interface on the client system. The retry user interface may have a similar (or identical) appearance to user interface 400 by including text and graphical instructions to assist the traveler in capturing an image of the official document (step 302).

[0064] If the document classification model 116 recognizes the captured image as an image of an official document, the method proceeds to step 308. At step 308, the document classification model 116 performs inference on the image to classify the image into a document category and assign the document an appropriate document category value. The data extraction application 112 receives this document category value from the document classification model 116.

[0065] In step 310, the data extraction application 112 processes the document category values.

[0066] In step 312, the data extraction application 112 performs a determination whether the document category value indicates that the document is a TD1 document.

[0067] If the document is determined to be a TD1 document, the process proceeds to step 314, where the data extraction application 112 performs a card flip operation, which is described below.

[0068] If it is determined that the document is not a TD1 document, the process proceeds to step 316 where the data extraction application 112 performs a determination whether the document category value indicates that the document is a TD3 document.

[0069] If the data extraction application 112 determines that the document is not a TD3 document, the process ends.

[0070] If the data extraction application 112 determines that the document is a TD3 document, the process proceeds to step 318, where the data extraction application 112 performs an operation to capture the surface of the TD3 document. As described above, operation 318 involves the data extraction application 112 preprocessing the image to identify MRZs within the image. Operation 318 also involves the data extraction application 112 extracting a photo of the traveler from the document.

[0071] The process then proceeds to step 320, where the data extraction application 112 performs optical character recognition (OCR) on the MRZ to capture the text in an appropriate format (eg, ASCII text).

[0072] The method then proceeds to step 322. The method also proceeds to step 322 after the data extraction application 112 performs a card flip operation on the TD1 document.

[0073] In step 322, the data extraction application 112 displays a detailed verification user interface on the display of the client system 104. An exemplary detailed verification user interface 500 is shown in FIG. 5. The detailed verification user interface 500 lists the data that the data extraction application 112 extracted from the official document. In the illustrated embodiment, the extracted data includes the passenger's first name, last name, gender, nationality, date of birth, document number, expiration date, issuing authority, and document type. As described above, aside from the document type, data is extracted from the MRZ using OCR operations. The extracted data also includes a photo 502 of the traveler from the official document.

[0074] The details verification user interface 500 includes a submit details button 504 that the traveler operates to cause the data extraction application 112 to submit the extracted data to the automated check-in server 102 (if the extracted data is correct). The details verification user interface includes an edit details button 506 that the traveler operates to manually edit any incorrect extracted data.

[0075] After displaying the detailed verification user interface 500, the method proceeds to step 324, where the data extraction application displays a selfie capture user interface on the display of the client system 104. An example of a selfie capture user interface is shown in FIG. 4 in the form of a "take selfie button 404." Activating button 404 causes the data extraction application 112 to prompt the user to take a selfie (or select an existing photo) and send the selfie to the automated check-in server 102. As described above, the server application 108 uses the submitted selfie to verify identity by comparing the selfie with the official document photo 502.

[0076] An example of the flip card operation 314 is described with reference to Figure 6. As described above, the data extraction application 112 performs the flip card operation when the document category value indicates that the document is a TD1 document.

[0077] The process 314 begins at step 602, where the document classification model 116 performs inference on the document image to determine whether the image is from the front of the TD1 document. If the document classification model determines that the image is from the front of the document, the process proceeds to step 604, where the data extraction application 112 performs a front capture operation.

[0078] For a TD1 document, the surface capture operation 604 involves capturing an image of a traveler's photo within the document.

[0079] The process then proceeds to step 606, where the data extraction application 112 displays a card flip user interface on the display of the client system 104. An example of a card flip user interface 700 is shown in FIG. 7. Interface 700 includes text and graphic instructions instructing the traveler to flip the document over and photograph the back side. Similar to interface 400, interface 700 includes a frame 702 in which the traveler is instructed to locate the image of the ID document.

[0080] After the traveler photographs the back of the ID document, the process proceeds to step 608 where the data extraction application 112 performs OCR on the captured image. For TD1 documents, as previously mentioned, the MRZ is on the back of the document.

[0081] If the document classification model 116 determines that the image is from the back side of the TD1 document, the process proceeds to step 610 where the data extraction application 112 performs OCR on the captured image, similar to step 608 .

[0082] The process then proceeds to step 612 , where the data extraction application 112 displays a card flip user interface on the display of the client system 104 , similar to step 606 .

[0083] The process then proceeds to step 614 where the data extraction application 112 performs a surface capture operation similar to step 604 .

[0084] Specific data augmentation techniques were utilized to train the neural network into a document classification model that could be deployed as a web application and reliably classify official documents on the client side. In particular, the initial dataset of images of TD1-Front, TD1-Back, and TD3 documents was augmented using background randomization and scan randomization.

[0085] A training dataset of Chinese identity cards was generated by performing line expansion using interpolation from a set of TD3 sample documents, resulting in a set of training examples that have the visual characteristics of TD3 documents but with a single-line MRZ.

[0086] An expanded training set of TD3 documents trained the neural network to recognize photographs and documents with MRZs of three lines.

[0087] Similarly, an expanded training set of TD1-Back images trained the neural network to recognize two lines of MRZ.

[0088] An example of an illumination correction and binarization operation performed on an image prior to OCR is shown in Figure 8. The operation involves locating an ROI relative to the detected face.

[0089] An example of the CCL and clustering operations performed on an image prior to OCR is shown in Figure 9. In a preferred embodiment, these operations take advantage of the properties of MRZ symmetry.

[0090] An example of a heuristic filtering operation 950 performed on an image prior to OCR is shown in Figure 10. Heuristic filtering is shown for both a TD1 document and a TD3 document, both of which contain MRZs of OCRB fonts.

[0091] In the illustrated embodiment, the detected characters and their corresponding clusters are filtered by adjusting the following characteristics of the MRZ and its OCRB fonts: - characters per line - Pixels per character - Space per character - Distance between letters - Distance between MRZ lines - MRZ width and height

[0092] In the illustrated embodiment, the filtering algorithm is: - Sorting filtered clusters based on the number of characters in the cluster - Determining the median spacing between characters on the largest cluster - Filtering characters in all other clusters based on the calculated interval median - Compute the projected box of each cluster (potential MRZ row) - Place clusters in order (top, middle, or bottom row) based on their projected box position - Calculate the projected box of the entire MRZ region by taking the extreme values ​​of the four corners (top left, top right, bottom left, bottom right). This projection box can then be used for rectification using the homography matrix.

[0093] An example of a homography calculation 970 is shown in Figure 11. As will be appreciated by those skilled in the art, homography calculation takes advantage of the fact that any two images of the same planar surface in space are related by a homography.

[0094] FIG. 12 provides a block diagram of a computer processing system 1200 that can be configured to implement embodiments and / or features described herein. System 1200 is a general-purpose computer processing system. It will be appreciated that FIG. 12 does not show all functional or physical components of a computer processing system. For example, a power supply or power interface is not shown, although system 1200 may include a power supply and / or be configured to connect to a power supply. It will also be appreciated that the particular type of computer processing system will determine the appropriate hardware and architecture, and that alternative computer processing systems suitable for implementing features of the present disclosure may have components other than those shown.

[0095] Computer processing system 1200 includes at least one processing unit 1202. Processing unit 1202 may be a single computer processing device (e.g., a central processing unit, a graphics processing unit, or other computing device) or may include multiple computer processing devices. In some examples, all processing is performed by processing unit 1202, while in other examples, processing may be performed by a remote processing device that is accessible and usable by system 1200 (either in a shared or dedicated manner).

[0096] Via communication bus 1204, processing unit 1202 is in data communication with one or more machine-readable storage (memory) devices that store instructions and / or data for controlling the operation of processing system 1200. In this example, system 1200 includes system memory 1206 (e.g., BIOS), volatile memory 1208 (e.g., random access memory such as one or more DRAM modules), and non-volatile memory 1210 (e.g., one or more hard disks or solid-state drives).

[0097] System 1200 also includes one or more interfaces, generally indicated at 1212, through which system 1200 interfaces with various devices and / or networks. Generally speaking, the other devices may be integral to system 1200 or may be separate. If a device is separate from system 1200, the connection between the device and system 1200 may be via wired or wireless hardware and communication protocols, and may be a direct connection or an indirect connection (e.g., a network connection).

[0098] Wired connections to other devices / networks can be made via any suitable standard or proprietary hardware and connection protocol. For example, system 1200 can be configured to make wired connections to other devices / communications networks via one or more of USB, FireWire, eSATA, Thunderbolt, Ethernet, OS / 2, parallel, serial, HDMI, DVI, VGA, and SCSI. Other wired connections are also possible.

[0099] Wireless connections with other devices / networks can similarly be made via any suitable standard or proprietary hardware and communication protocol. For example, the system 1200 can be configured to wirelessly connect with other devices / communication networks using one or more of infrared, Bluetooth, Wi-Fi, Near Field Communication (NFC), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Long Term Evolution (LTE), Wideband Code Division Multiple Access (W-CDMA), and Code Division Multiple Access (CDMA). Other wireless connections are possible.

[0100] Generally speaking, depending on the particular system in question, the devices to which system 1200 connects, whether by wired or wireless means, include one or more input devices for allowing data to be input / received into system 1200 for processing by processing unit 1202, and one or more output devices for allowing system 1200 to output data. While exemplary devices are described below, it will be appreciated that not all computer processing systems will include all of the devices mentioned, and that devices in addition to and instead of those mentioned may well be used.

[0101] For example, system 1200 may include or be connected to one or more input devices through which information / data is input to (received by) system 1200. Such input devices may include a keyboard, mouse, trackpad, microphone, accelerometer, proximity sensor, GPS device, etc. System 1200 may also include or be connected to one or more output devices controlled by system 1200 to output information. Such output devices may include devices such as a CRT display, LCD display, LED display, plasma display, touchscreen display, speaker, vibration module, LED / other lights, etc. System 1200 may also include or be connected to devices that can function as both input and output devices, such as memory devices (hard drives, solid-state drives, disk drives, CompactFlash cards, SD cards, etc.) from which system 1200 can read and / or write data, and touchscreen displays that can both display (output) data and receive (input) touch signals.

[0102] System 1200 may also connect to one or more communication networks (e.g., the Internet, a local area network, a wide area network, a personal hotspot, etc.) to communicate data with and receive data from networked devices, which may themselves be other computer processing systems.

[0103] System 1200 may be any suitable computing system, such as, by way of non-limiting example, a server computing system, a desktop computer, a laptop computer, a netbook computer, a tablet computing device, a mobile / smartphone, a personal digital assistant, a personal media player, a set-top box, a game console, etc. [Note repetition of computing system description]

[0104] Typically, the system 1200 includes at least user input and output devices 1214 and a communication interface 1216 for communicating with a network, such as the network 106 of the environment 100 .

[0105] System 1200 stores or accesses computer applications (also called software or programs), i.e., computer-readable instructions and data that, when executed by processing unit 1202, configure system 1200 to receive, process, and output data. The instructions and data may be stored on a non-transitory machine-readable medium accessible to system 1200. For example, the instructions and data may be stored in non-transitory memory 1210. The instructions and data may be transmitted / received by system 1200 via data signals in a transmission channel enabled by (for example) a wired or wireless network connection.

[0106] Applications accessible to the system 1200 typically include operating system applications such as Microsoft Windows, Apple OSX, Apple IOS, Android, Unix, or Linux.

[0107] System 1200 also stores or has access to applications that, when executed by processing unit 1202, configure system 1200 to perform various computer-implemented processing operations described herein. For example, with reference to the environment of FIG. 1 above, client system 104 includes data extraction application 112 that configures client system 104 to perform the described client system operations. Similarly, automated check-in server 102 includes server application 108 that configures server system 102 to perform the described server system operations.

[0108] The flowcharts shown in the figures and described above define operations in a particular order to illustrate various features. In some cases, the operations described and illustrated may be performed in an order different from that illustrated / described, one or more operations may be combined into a single operation, a single operation may be divided into multiple separate operations, and / or functionality achieved by one or more of the operations described / illustrated may be achieved by one or more alternative operations. Still further, the functions / processing of a given flowchart operation may be performed by different systems or applications.

[0109] Variations and modifications can be made to the foregoing without departing from the spirit or scope of the present disclosure.

[0110] This specification describes various embodiments with respect to numerous specific details that may vary from implementation to implementation. No limitation, element, property, feature, advantage, or attribute not expressly recited in a claim should be considered an essential or essential feature. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0111] In the following claims and in the foregoing description of the invention, unless the context requires otherwise by express wording or necessary implication, the word "comprise" or variations such as "comprises" or "comprising" are used in the inclusive sense, i.e., to specify the presence of stated features, but do not exclude the presence or addition of further features in various embodiments of the invention. [Explanation of symbols]

[0112] 100 Computing Environments 102 Automatic check-in server 104 Client Systems 106 Communication Network 108 Server Applications 110 Data Storage 112 Client Applications, Data Extraction Applications 116 Document Classification Model 118 Pre-processing module 120 Camera 120 Lighting Correction Module 122 Connected Component Labeling (CCL) Module 124 Correction Module 126 Binarization Module 128 Heuristic-based Filtering Module 130 Information Parser Module 132 OCR modules 134 Regular Expressions (Regex) Module 300 Computer Executable Processes 314 Card flip operation 400 User Interface 402 frames 404 button 500 Detailed Verification User Interface 502 photos 504 Submit Details Button 506 Edit Details button 604 Surface Capture Operation 700 Card Reversal User Interface 702 frames 950 Heuristic Filtering Behavior 970 Homography Calculation 1200 Computer Processing System 1202 Processing Unit 1204 communication bus 1206 system memory 1208 Volatile Memory 1210 Non-volatile memory, non-transient memory 1214 User Input and Output Devices 1216 Communication Interface

Claims

1. 1. A data extraction system including a data extraction application, the data extraction application including a trained document classification model and computer program code that, when executed by a processor of an electronic device, causes the processor to: presenting a user interface on the electronic device that prompts a user to capture an image of a document; submitting the captured image of the document to the document classification model; receiving a document category value for the document from the document classification model; processing the document category values ​​to select a next data extraction operation to be performed by the data extraction application; initiating the selected data extraction operation; A data extraction system that allows

2. 2. The data extraction system of claim 1, wherein the processor selects the next data extraction operation from a set of data extraction operations, the set including presenting a retry user interface that instructs the user to capture another image of the document, performing optical character recognition on the document, presenting a manipulation user interface that instructs the user to physically manipulate the document, and presenting an extraction success user interface that notifies the user that data extraction from the document was successful.

3. The data extraction system of claim 2 , wherein the successful extraction user interface presents data extracted from the document on the electronic device.

4. The data extraction system of claim 2 , wherein the successful extraction user interface includes a confirmation user interface element operable by the user to confirm that the data extracted from the document is correct.

5. The data extraction system of claim 2 , wherein the operational user interface instructs the user to present another side of the document to the electronic device for capture.

6. The data extraction system of claim 5 , wherein the computer program code further causes the processor to perform an image capture routine on the other side of the document once captured by the electronic device.

7. The data extraction system of claim 5 , wherein the computer program code further causes the processor to perform optical character recognition on the other side of the document once captured by the electronic device.

8. The data extraction system of claim 2 , wherein the operation user interface instructs the user to present another portion of the same side of the document to the electronic device for capture.

9. The computer program code further causes the processor to: inputting features of the document classification model as heuristics into a heuristic filtering algorithm; performing the heuristic filtering algorithm on the captured image before performing optical character recognition on the captured image; The data extraction system according to any one of claims 1 to 8,

10. The data extraction system of claim 9 , wherein the features of the document classification model include the presence of machine-readable zones and the structure of the machine-readable zones.

11. 11. The data extraction system of claim 1, wherein the data extraction application and the document classification model are contained in a web application executable by a web browser installed on the electronic device.

12. 1. A method for extracting data from a document, comprising: delivering the data extraction application and the trained document classification model to the electronic device; the data extraction application presenting a user interface on the electronic device prompting a user to capture an image of a document; the data extraction application submitting the captured image of the document to the document classification model; the data extraction application receiving a document category value for the document from the document classification model; processing, by the data extraction application, the document category values ​​to select a next data extraction operation to be performed by the data extraction application; the data extraction application initiating the selected data extraction operation; and A method comprising:

13. 13. The method of claim 12, wherein the data extraction application selects the next data extraction operation from a set of data extraction operations, the set including presenting a retry user interface that instructs the user to capture another image of the document, performing optical character recognition on the document, presenting a manipulation user interface that instructs the user to physically manipulate the document, and presenting an extraction success user interface that notifies the user that data extraction from the document was successful.

14. The method of claim 13 , wherein the successful extraction user interface presents data extracted from the document on the electronic device.

15. The method of claim 13 , wherein the successful extraction user interface includes a confirmation user interface element operable by the user to confirm that the data extracted from the document is correct.

16. The method of claim 13 , wherein the operational user interface prompts the user to present another side of the document to the electronic device for capture.

17. The method of claim 16 , wherein the data extraction application performs an image capture routine on the other side of the document once captured by the electronic device.

18. The method of claim 16 , wherein the data extraction application performs optical character recognition on the other side of the document once captured by the electronic device.

19. The method of claim 13 , wherein the operational user interface instructs the user to present another portion of the same side of the document to the electronic device for capture.

20. The document extraction application inputting features of the document classification model as heuristics into a heuristic filtering algorithm; performing the heuristic filtering algorithm on the captured image before performing optical character recognition on the captured image; The method according to any one of claims 12 to 19, wherein

21. 21. The method of claim 20, wherein the features of the document classification model include the presence of machine-readable zones and the structure of the machine-readable zones.

22. 22. The method of claim 12, wherein the data extraction application and the document classification model are included in a web application executable by a web browser installed on the electronic device.

23. 23. A non-transitory machine-readable medium storing computer program code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 12 to 22.

Images