Methods and systems for performing data capture.

Abstract

A method (100) for performing data capture on an unlabeled document based on a labeled document including one or more elements each labeled for a field is provided. The method (100) includes determining (S110) one or more anchors having a pair of matching points between the labeled document and the unlabeled document, generating (S120) an entire kernel using the determined one or more anchors, and identifying (S130) one or more elements to be captured in the unlabeled document using the entire kernel in a kernel machine. The one or more elements to be captured correspond to one or more labeled elements included in the labeled document in a field.

Description

[Technical Field]

[0001] This disclosure relates to methods and systems for performing data capture. More specifically, this disclosure relates to systems and computer implementations for performing data capture on one or more unlabeled documents based on one or more labeled documents. [Background technology]

[0002] Various types of documents are widely used to collect and record information in many purposes and fields, including healthcare, commerce, education, and government. With the widespread use of computers and communication networks today, these documents are now typically created electronically and provided for digital generation and sharing. These documents usually contain data in a structured or semi-structured format. Structured documents may have embedded code that allows information to be arranged in a specific format. Semi-structured documents, such as invoices, bank statements, utility bills, and passports, do not adhere to the strict layout of structured documents and may not be limited to specific data fields like structured documents.

[0003] To use information from these electronic documents to perform desired operations (e.g., business-related operations), it is necessary to properly extract the information. However, at least for semi-structured electronic documents, these documents often lack a clearly defined layout, making it difficult to reliably analyze and extract the necessary information. [Overview of the project]

[0004] Data capture is the task of extracting relevant information from a document. The One-Shot Data Capture (OSDC) techniques described in this disclosure are designed to extract information from structured and semi-structured documents. The data captured may include values ​​in key-value pairs within a document, such as invoice numbers, total amounts, and / or values ​​in tables within the document. In the context of this disclosure, key-value pairs may be referred to as “field element pairs,” keys as “fields,” and values ​​as “elements.” In its simplest form, OSDC techniques work with pairs of labeled and unlabeled documents. In this case, the task is to extract from the unlabeled document the same type of information that is labeled in the labeled document.

[0005] The embodiments described herein relate to methods and systems for performing data capture on unlabeled documents based on labeled documents.

[0006] According to one aspect of the present disclosure, a computer implementation method is provided for performing data capture in an unlabeled document based on a labeled document containing one or more elements each labeled for a field. The computer implementation method includes determining one or more anchors having a pair of points that match between the labeled document and the unlabeled document; generating an entire kernel using the determined one or more anchors; and using the entire kernel in a kernel machine to identify one or more elements in the unlabeled document to be captured, wherein the one or more elements to be captured correspond by fields to the one or more labeled elements contained in the labeled document.

[0007] Another aspect of the present disclosure provides a system for performing data capture in an unlabeled document based on a labeled document containing one or more elements each labeled for a field. The system comprises a determination unit configured to determine one or more anchors having a pair of corresponding points that match between the labeled document and the unlabeled document; a generation unit configured to generate an entire kernel using the determined one or more anchors; and a identification unit configured to use the entire kernel in a kernel machine to identify one or more elements to be captured containing information to be captured in the unlabeled document, wherein the one or more elements to be captured correspond in fields to the one or more labeled elements contained in the labeled document.

[0008] According to another aspect of this disclosure, a computer-readable storage medium is provided. The storage medium includes instructions that, when executed by a computer, cause the computer to perform steps of the method described in this disclosure.

[0009] The characteristics of optional selection are described in the attached dependent claims.

[0010] These and other aspects of the Disclosure will be better recognized and understood when considered together with the following description and accompanying drawings. The following description illustrates various embodiments of the Disclosure and some specific details thereof, but is given by example rather than limitation. Many substitutions, modifications, additions or rearrangements are possible within the scope of the Disclosure, and the Disclosure includes all such substitutions, modifications, additions or rearrangements. [Brief explanation of the drawing]

[0011] Hereinafter, embodiments of the present disclosure will be described only as illustrative examples with reference to the attached drawings. [Figure 1] Figure 1 is a flowchart of a computer implementation method for performing data capture according to one embodiment of the present disclosure. [Figure 2]FIG. 2 is a block diagram of a system for performing data capture according to an embodiment of the present disclosure. [Figure 3A] FIG. 3A is a diagram showing exemplary labeled documents and exemplary unlabeled documents according to an embodiment of the present disclosure. [Figure 3B] FIG. 3B is a diagram showing exemplary labeled documents and exemplary unlabeled documents according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing an exemplary process for determining an anchor between the labeled documents and the unlabeled documents of FIGS. 3A and 3B. [Figure 5] FIG. 5 is a diagram showing an anchor determined from the exemplary process shown in FIG. 4. [Figure 6-1] FIG. 6 is a diagram showing exemplary labeled documents and exemplary unlabeled documents according to an embodiment of the present disclosure. [Figure 6-2] FIG. 6 is a diagram showing exemplary labeled documents and exemplary unlabeled documents according to an embodiment of the present disclosure. [Figure 7A] FIG. 7A is a diagram showing an anchor determined based on the labeled documents and the unlabeled documents of FIG. 6. [Figure 7B] FIG. 7B is a diagram showing another anchor determined based on the labeled documents and the unlabeled documents of FIG. 6. [Figure 8-1] FIG. 8 is a diagram showing an anchor determined from the documents of FIG. 6 and other unlabeled documents. [Figure 8-2] FIG. 8 is a diagram showing an anchor determined from the documents of FIG. 6 and other unlabeled documents. [Figure 8-3] FIG. 8 is a diagram showing an anchor determined from the documents of FIG. 6 and other unlabeled documents. [Figure 9] FIG. 9 is a diagram showing a topology for performing data capture according to an embodiment of the present disclosure.

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] The The following description provides numerous specific details to allow for a full understanding of the embodiments. However, it will be apparent to those skilled in the art that the use of these specific details is unnecessary for carrying out the embodiments. In other examples, well-known materials or methods are not described in detail in order to avoid obscuring the disclosure.

[0013] Figure 1 is a flowchart of a computer implementation method 100 for performing data capture according to one embodiment of the present disclosure. More specifically, the illustrated method 100 is for performing data capture in an unlabeled document based on a labeled document. A labeled document contains one or more elements, each labeled with respect to a field. Each element may be represented by a word, a string of words, a number, or a combination thereof. In some embodiments, a labeled document may further contain one or more fields indicated by one or more labels. The labeled document and / or unlabeled document may be a PDF document or an image. In some implementations, this method may be performed by a client computing device, a network, a server, or a combination thereof, for example, a client computer device 905, a network 910, and a server 920 as shown in Figure 9.

[0014] Method 100 begins in step S110, in which step S110 one or more anchors are determined. An anchor has a pair of points that match between a labeled document and an unlabeled document. In some implementations, for each new pair of documents (each pair containing an unlabeled document and a labeled document), one or more new sets of anchors are determined.

[0015] In some embodiments, the anchor determination in step S110 may include accepting user input to select a pair of points in an unlabeled document and a labeled document. Alternatively, in some embodiments, the anchor determination in step S110 may include the following steps: - Extract all words from labeled and unlabeled documents (e.g., using OCR). - Order the words contained in the labeled document into a linear text sequence to generate a first linear text sequence (optionally, remove any words corresponding to the labeled elements). - Order the words in the unlabeled document into a linear text sequence to generate a second linear text sequence. - Identify matches between the first linear text sequence and the second linear text sequence (if words corresponding to labeled elements have not been removed at this stage, these words are ignored in the match identification step). - Specify the matched word as the anchor.

[0016] In this context, a match includes the intrinsic similarity of two words and their alignment (i.e., similarity in terms of the linear order of each word in each sequence). In these embodiments, a match between a first text sequence and a second text sequence may exceed a predetermined threshold in terms of both intrinsic similarity (e.g., visual features) and alignment (i.e., how similar they are in terms of the linear order of each word in each text sequence). In other words, a match may be two words that have the same or similar appearance and whose linear order in each text sequence is the same or similar. As an example of visual similarity, the words "bam" and "barn" can be determined to exceed a predetermined visual similarity threshold based on their visual similarity (e.g., a threshold that takes into account the OCR (optical character recognition) errors between "m" and "rn"). As an example of alignment, the eighth word of the first text subsequence may have a higher degree of alignment with the eighth word of the second text subsequence than with the seventh word of that text subsequence. In some embodiments, the identification of a match (more specifically, the determination of whether two words exceed an alignment threshold) can be based on an alignment algorithm such as an algorithm similar to or equivalent to the Needleman-Wunsch algorithm.

[0017] Alternatively or additionally, in these embodiments, the ordering of words contained in labeled and unlabeled documents can be performed using a reading order algorithm (i.e., any algorithm that attempts to order words as a human would read them).

[0018] Alternatively or additionally, in these embodiments, each of the one or more anchors may be defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0019] In some embodiments, the determination of the anchor in step S110 may include identifying a visual feature present in both the labeled and unlabeled documents (e.g., using an image alignment algorithm) and designating the identified visual feature as the anchor. The image alignment algorithm may be based on at least one of the following: a keypoint detector (e.g., difference Gaussian, Harris, Good Features to Track (GFTT), a locally invariant descriptor (e.g., scale-invariant feature transformation (SIFT), fastened robust features (SURF), oriented fast and rotated brief (ORB)), and a keypoint matching process (e.g., random sample consensus (RANSAC)).

[0020] In some embodiments, the determination of anchors in step S110 may include placing a number of points around the boundary of an unlabeled document and the same number of points around the boundary of a labeled document, identifying matches between points in the labeled document and points in the unlabeled document, and designating the matched points as anchors.

[0021] In this embodiment, it is explained that one or more anchors are determined for a pair of documents (i.e., a labeled document and an unlabeled document), but it will be understood that the process for determining anchors described above can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). For example, an anchor may have multiple points that match (i.e., exceed a visual similarity threshold and alignment threshold), and the multiple points may be in different documents.

[0022] Furthermore, while this embodiment states that an anchor may include a single word (which may consist of numbers, contain numbers, or not) to match, in other embodiments, an anchor may include terms to match in addition to or instead of words. For example, the multiple words “Total amount” are treated as a single term rather than two separate words, and as a result, each anchor matching the term “Total amount” matches in a pair of documents.

[0023] Returning to Figure 1, in step S120, the entire kernel is generated using the one or more determined anchors. In some embodiments, generating the entire kernel in step S120 may include generating a convolution kernel for each of the one or more anchors determined in step S110, and generating the entire kernel by summing the one or more convolution kernels. The entire kernel may be a weighted sum of one or more convolution kernels.

[0024] Each convolution kernel can span all word pairs within and between labeled and unlabeled documents. In other embodiments where the number of documents on which data capture is performed exceeds two, each convolution kernel can span all word pairs within and across all two or more documents. Each convolution kernel can measure the similarity of words within and between labeled and unlabeled documents (or between all documents if more than two documents are involved), where similarity is a function of the distance from each point of the anchor to each word, and is the similarity of the relative position to each point of the anchor.

[0025] To explain the convolution kernel in more technical detail, see document d and word w d =w d,1 ,…,w d,nd Defined as a sequence of n. Here, n d This is the number of words in the document.

[0026] If the document is labeled, each word will have a label. d =l1,…,l nd A label may also be provided, and the label belongs to one of K categories (each category corresponds to a field).

number

[0027] Furthermore, each word may have a position on the page of the document. For example, the i'-th word w in the d'-th document. d,i’ The position is x d,i It is specified as follows.

[0028] An anchor a spanning two documents, d1 and d2, can be represented by a pair of locations a=(a1,a2) (one in each document), where a1 is the location within d1 and a2 is the location within d2. A labeled document is defined as d=d1, and an unlabeled document is defined as d=d2.

[0029] A pair of words (w d,i , w d’,j ) can be defined as the product of the anchor a with other radial basis function (RBF) kernels k off and k rel . The key input to the kernel is the relative position of the word with respect to the anchor and is defined as follows.

Number

Number

[0030] The kernel can be defined as follows.

Number

Number

Number

Number

[0031] Note that the convolutional kernel can act on any pair of words, regardless of whether they are in the same document, so d = d’ or d ≠ d’ is possible.

[0032] The linear transformations (matrices) A off and A rel act as inverse length scales. In one implementation, these are parameterized diagonally and σ is the length scale.

Number

Number

[0033] This allows the kernel to have different effects on each direction within the document. For example, the value of σ can be set so that the effect on the width of the document is more acceptable than the effect on the height.

[0034] The basis for the above kernel equation is the offset kernel k off The reason for this is that the further a word is from the anchor, the more it attenuates. In other words, the anchor only has a local effect. In contrast to this, the relative kernel k rel This method quantifies the spatial similarity between an anchor and each word. In other words, words are considered similar if they are located in the same position relative to the anchor. By taking the kernel product, it is possible to consider words similar only if they are close to the anchor and their positions relative to the anchor are similar.

[0035] As mentioned above, each convolution kernel may span a word pair; however, in an alternative embodiment, each convolution kernel may span a term pair (for example, each term consisting of multiple words) instead of, or in addition to, a single word pair.

[0036] As described above, in some embodiments, the process of determining anchors can be applied to more than two documents (e.g., multiple labeled and unlabeled documents), and each anchor may have multiple matching points in different documents. In these embodiments, the anchor is a=(a1,a2,…,a N) may be defined as, where N-1 is the number of labeled documents selected for data capture (and thus N is the total number of documents). Furthermore, in these embodiments, each convolution kernel can span all word pairs in all documents and all word pairs across all documents, and the entire kernel is the sum of all convolution kernels. This larger entire kernel (compared to when using only two documents) can provide the kernel machine with more information to identify one or more elements to capture in step S130. Each convolution kernel defined with respect to a set of documents can be represented as a symmetric positive semi-definite matrix having dimensions corresponding to the number of words in all documents in each set of documents. Thus, in this case, each entry in the matrix corresponds to each similarity between specific word pairs contained in documents within the set of documents. Similarly, since the entire kernel is generated by summing the convolution kernels, it can also be represented as a symmetric positive semi-definite matrix, where each entry in the matrix corresponds to each similarity between specific word pairs among the words contained in documents within the set of documents.

[0037] Returning to Figure 1, in step S130, the entire kernel in the kernel machine is used to identify one or more elements that are captured in the unlabeled document. The one or more elements that are captured correspond to one or more labeled elements and fields contained in the labeled document. The kernel machine has at least one of the following: a support vector machine (SVM), regularized kernel discriminant analysis (RKDA), or a Gaussian process (GP). The kernel machine may be trained on at least one of the one or more elements and one or more fields represented by each label in the labeled document. In some cases, the kernel machine may be trained on all of the one or more elements and one or more fields represented by each label in the labeled document in order to maximize the number of samples used for training. In some embodiments, the kernel machine may be trained using a subset of matrix entries for the entire kernel, where each matrix entry in the subset of matrix entries for the entire kernel corresponds to a pair of words in the labeled document. The identified elements may correspond to words that have a high (predicted) similarity / confidence score determined by the kernel machine. In these embodiments, the method may further include outputting similarity / confidence scores determined by a kernel machine.

[0038] Although not shown in Figure 1, in some embodiments, method 100 may further include receiving user input to confirm or reject a particular result after identifying one or more elements in step S130, and receiving user input to modify a particular result. In these embodiments, if user input confirming a result is received, method 100 may further include outputting the result (e.g., via a display) and performing further processing of the result (e.g., determining and / or grouping fields, as described below). If user input to modify a particular result is received, method 100 may further include performing the modification based on the user input and / or saving the modified result as at least part of a new labeled document.

[0039] In some embodiments, this method may further include determining the field corresponding to each of the one or more identified elements. This determination operation can also be performed using the entire kernel within the kernel machine. In some embodiments, it will be understood that the operation of identifying one or more elements in step S130 and the operation of determining the field for each of the one or more identified elements can be performed simultaneously as a single step / process within the kernel machine.

[0040] In some embodiments where multiple identified elements exist, the field determination operation for the identified elements involves grouping the multiple identified elements into one or more groups, where each group corresponds to a field. This grouping can be based on a word grouping algorithm. More specifically, the grouping of multiple identified elements may include the following: - Generate a linear text sequence of multiple identified elements in an unlabeled document (e.g., in reading order using a reading order algorithm). - Determine the classification of each element in the generated linear text sequence (e.g., using a kernel machine). -Group all of the largest consecutive subsequences of elements that share a classification in the generated linear text sequence into each group.

[0041] The classification determined for each element may be an argmax classification. The classification decision can be based on multiple possible classifications, each of which corresponds to either a field in the labeled document or an indication that the element is not associated with any field in the labeled document.

[0042] In embodiments where multiple identified elements are grouped together (for example, using the grouping algorithm described above), the method may further include assigning a row identifier to each element in the group based on the relative order of the elements within the group. This allows the elements belonging to the tabular format to be output directly (i.e., elements with the same row identifier are placed in the same row).

[0043] In some embodiments, one or more fields and one or more elements in a labeled document may be presented in an information structure. For example, in an invoice, one or more fields may be presented in a table header, and one or more elements may be presented below the table header, with each column corresponding to a field and each row corresponding to a purchase. In these embodiments, method 100 may further include outputting the identified one or more elements and their corresponding fields in the same (or similar) information structure. Alternatively, method 100 may further include outputting only one or more elements (regardless of whether the fields and / or elements are presented in the same / similar or different information structures).

[0044] This method may further include assigning a new label to each of the one or more identified elements, indicating the field to which each element corresponds.

[0045] In some embodiments, this method may further include converting each of the one or more identified elements into a normalized format. For example, all elements corresponding to date information may be returned in DD / MM / YYYY format. This operation may be based on field-independent rules (i.e., one rule that applies to all fields). For example, a rule to remove leading or trailing punctuation from each string of text of the identified elements. Alternatively, this operation may be based on one or more field-specific rules, which can be set based on the metadata of the labels in the labeled document.

[0046] In some embodiments, it will be understood that the use of labels in a labeled document is sufficient to indicate which elements are of interest, and therefore sufficient to indicate which corresponding elements in an unlabeled document should be identified and captured. However, in some embodiments, the method may further include defining one or more fields contained in the labeled document and obtaining a schema that defines whether elements corresponding to each field in the labeled document are displayed in a table, before determining one or more anchors in step S110. This schema can provide clearer information about what data should be captured from the unlabeled document. This schema can be used, for example, in determining anchors in S110 of method 100, so that one or more fields defined by the schema can each be used as anchor points.

[0047] Although not shown in Figure 1, in some embodiments, method 100 may further include receiving user input assigning one or more labels to elements of a labeled document before determining one or more anchors in step S110. In some implementations, receiving user input assigning labels may include receiving user selections of fields in the labeled document and receiving one or more user selections of one or more elements corresponding to the selected fields. For example, user selection of elements can be implemented by allowing the user to draw a rectangular box around an element on the document via a user interface, such as a touchscreen. The field corresponding to a label (which can be represented by a rectangular box) can be indicated by a visual indicator. For example, each rectangular box may be a color that indicates the corresponding field of the element. In this case, the rectangular boxes of elements corresponding to the same first field are the same first color, the rectangular box elements corresponding to the same second field are the same second color, and so on. Furthermore, in some embodiments, each label may include metadata. For example, metadata indicating whether the field is in numeric format, and / or metadata indicating whether the element corresponding to the field is displayed in tabular format.

[0048] In some embodiments, method 100 may further include obtaining a plurality of labeled document candidates before determining one or more anchors, and receiving user input to select a labeled document from the plurality of labeled document candidates. Alternatively, method 100 may further include obtaining a plurality of labeled document candidates before determining one or more anchors, and determining a labeled document from the obtained labeled document candidates based on a similarity metric between each of the plurality of labeled document candidates and an unlabeled document. In some embodiments, determining a labeled document from a plurality of labeled document candidates may be based on a similarity function and a best-match algorithm. The similarity function may be a function that processes document pairs (i.e., each labeled document candidate and an unlabeled document in this case) and returns a similarity metric based on that processing. An example of a similarity metric is a score that indicates the degree of similarity between two documents.

[0049] In some embodiments, the similarity function may be based on a bag-of-word model that compares pairs of documents based on the number of common words (which may or may not include duplicates) across the documents. For example, if one document consists of multiple tokens "aaabc" and another document consists of multiple tokens "aaccd", the document pair shares three tokens (two "a"s and one "c"), so the similarity function might return a similarity score of 3.

[0050] Once the similarity scores between all pairs of labeled and unlabeled documents have been determined, the best match algorithm can then be used to select the "best match" based on the similarity metrics between each of the multiple labeled document candidates and the unlabeled document. For example, the best match algorithm can be used to select the pair of documents with the highest similarity score.

[0051] In some cases, a similarity score threshold may be used to determine if there are no multiple labeled document candidates suitable for performing data capture by the method described herein. For example, if the similarity score threshold is set to 4 and the similarity scores of any document pair (each pair containing a labeled document candidate and an unlabeled document) are not 4 or greater, the method may further include providing an output indicating that none of the (current) labeled document candidates are available for data capture. The method may also further include obtaining a new set of multiple labeled document candidates and determining a labeled document based on similarity metrics between each of the new multiple labeled document candidates and an unlabeled document.

[0052] Alternatively, or in addition, in some cases, a similarity score threshold can be used to determine whether multiple labeled document candidates are suitable for performing data capture by the method described herein. For example, if the similarity score threshold is set to 4 and multiple document pairs (each pair containing a labeled document candidate and an unlabeled document) have a similarity score of 4 or higher, the step of determining anchors in step S110 may be based on multiple labeled documents (each corresponding to a similarity score higher than the threshold) and unlabeled documents. As described above, in this case, each anchor may have multiple matching points, each in a different document. Furthermore, in this case, the step of generating the entire kernel in step S120 may be based on multiple labeled and unlabeled documents. That is, each convolutional kernel may span multiple labeled documents (each corresponding to a similarity score higher than the threshold) and unlabeled documents.

[0053] Figure 2 is a block diagram of a system for performing data capture according to one embodiment of the present disclosure. More specifically, system 200 is for performing data capture on an unlabeled document based on a labeled document. A labeled document contains one or more elements, each labeled with respect to a field. Each element may be represented by a word, a string of words, a number, or a combination thereof. In some embodiments, a labeled document may further contain one or more fields indicated by one or more labels. Labelled documents and / or unlabeled documents may be PDF documents or images.

[0054] As shown in Figure 2, the system 200 comprises a decision unit 210, a generation unit 220, and a identification unit 230. The decision unit 210 is configured to determine one or more anchors. As described above, an anchor has a pair of points that match between a labeled document and an unlabeled document. In some implementations, the decision unit 210 may be configured to determine one or more new sets of anchors for each new pair of documents (each pair containing an unlabeled document and a labeled document).

[0055] In some embodiments, the decision unit 210 may be configured to determine the anchor by receiving user input to select a pair of points in an unlabeled document and a labeled document. Alternatively, in some embodiments, the decision unit 210 may be configured to determine the anchor as follows: - Extract all words from both labeled and unlabeled documents (e.g., using OCR). - Order the words contained in the labeled document into a linear text sequence to generate a first linear text sequence (optionally, remove any words corresponding to the labeled elements). - Order the words in an unlabeled document into a linear text sequence to generate a second linear text sequence. - Identify matches between the first linear text sequence and the second linear text sequence (if words corresponding to labeled elements have not been removed at this stage, those words are ignored in the match-finding step). - Specify the matched word as the anchor.

[0056] In this context, a match includes not only the intrinsic similarity of two words (e.g., visual similarity) but also similarity or a match in terms of linear order in each sequence for each word. In these embodiments, a match between a first text sequence and a second text sequence may exceed a predetermined threshold in terms of both intrinsic similarity and alignment similarity. Furthermore, the decision unit 210 may be configured to identify matches based on an alignment algorithm (e.g., an algorithm similar to or equivalent to the Needleman-Wunsch algorithm) (more specifically, to determine whether two words exceed an alignment similarity threshold).

[0057] In these embodiments, the decision unit 210 may be configured to order the words contained in the labeled and unlabeled documents using a reading order algorithm. Furthermore, in these embodiments, each of the one or more anchors may be defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0058] In some embodiments, the decision unit 210 may be configured to determine anchors by identifying visual features present in both labeled and unlabeled documents (e.g., using an image alignment algorithm) and designate the identified visual features as anchors. The image alignment algorithm may be based on at least one of the following: keypoint detectors (e.g., difference Gaussian, Harris, Good Features to Track (GFTT), locally invariant descriptors (e.g., scale-invariant feature transformation (SIFT), fastened robust features (SURF), oriented fast and rotated brief (ORB)), and keypoint matching processes (e.g., random sample consensus (RANSAC)).

[0059] In some embodiments, the determination unit 210 may be configured to determine anchors by placing a number of points around the boundary of an unlabeled document, placing the same number of points around the boundary of a labeled document, identifying matches between the points in the labeled document and the points in the unlabeled document, and designating the matched points as anchors.

[0060] In this embodiment, it is explained that one or more anchors are determined for a pair of documents (i.e., a labeled document and an unlabeled document), but it will be understood that the process described above for determining anchors by the determination unit 210 can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). Also, in this embodiment, it is explained that an anchor can include a single word to match, but in other embodiments, an anchor can include a term to match rather than a word. For example, the word "total amount" is treated as a single term rather than two separate words.

[0061] The generation unit 220 is configured to generate the entire kernel using one or more determined anchors. In some embodiments, the generation unit 210 may be configured to generate the entire kernel by generating a convolution kernel for each of the one or more anchors and then summing the one or more convolution kernels to generate the entire kernel. The entire kernel may be a weighted sum of one or more convolution kernels.

[0062] Each convolution kernel can span all word pairs within and between labeled and unlabeled documents. In some embodiments where the number of documents underlying the data capture exceeds two, each convolution kernel can span all word pairs within all documents and all word pairs across all documents. Each convolution kernel can measure the similarity of words within and between labeled and unlabeled documents (or all documents if more than two documents are involved). Similarity is a function of the distance from each point of an anchor to each word, and is the similarity of the relative positions to each point of the anchor. The mathematical details of the convolution kernels have been explained above with reference to Figure 1 and will not be repeated here for brevity. It will be understood that these formulas are applicable to this embodiment. Also, as previously mentioned with reference to Figure 1, the convolution kernels and the entire kernel defined with respect to a set of documents can each be represented as a symmetric positive semi-definite matrix with dimensions corresponding to the number of words in all documents in the set of documents, where each entry in the matrix corresponds to each similarity between a particular pair of words in the words within the documents in the set of documents.

[0063] As mentioned above, each convolution kernel may span a pair of words, but in an alternative embodiment, each convolution kernel may span not only a single pair of words, but also a pair of terms (e.g., consisting of multiple words).

[0064] The specific unit 230 is configured to use the entire kernel in the kernel machine to identify one or more elements to be captured in an unlabeled document. The one or more elements to be captured correspond to one or more labeled elements and fields contained in the labeled document. The kernel machine may include at least one of the following: a support vector machine (SVM), regularized kernel discriminant analysis (RKDA), or a Gaussian process (GP). The kernel machine can be trained on one or more elements and one or more fields indicated by each label in the labeled document. In some cases, the kernel machine may be trained on all of the one or more elements and one or more fields indicated by each label in the labeled document in order to maximize the number of samples used for training. In some embodiments, the kernel machine may be trained using a subset of matrix entries for the entire kernel, where each matrix entry in the subset of matrix entries for the entire kernel corresponds to a pair of words in the labeled document. The identified elements may correspond to words with a high (predictive) similarity / confidence score determined by the kernel machine. In these embodiments, the system 200 may include an output unit configured to output similarity / confidence scores determined by the kernel machine.

[0065] Although not shown in Figure 2, in some embodiments, the system 200 may further include a receiving unit configured to receive at least one of user inputs confirming or rejecting a particular result, and user inputs modifying a particular result, after the identification unit 230 has identified one or more elements. Furthermore, the system 200 may include an output unit configured to output the result (e.g., via a display) and / or an execution unit configured to perform further processing on the result (e.g., field determination and / or grouping; therefore the execution unit may be a determination unit) when user input confirming a result is received. When user input modifying a particular result is received, the execution unit may be configured to make the modification based on the user input, and / or a storage unit in the system 200 may be configured to store the modified result as at least part of a new labeled document.

[0066] In some embodiments, the decision unit 210 may be further configured to determine, for each of the identified elements, the field corresponding to each element. This determination can be performed using the entire kernel of a kernel machine, such as the kernel machine that the identification unit 230 uses to identify the elements. In some embodiments, it will be understood that the decision unit 210 can be configured to perform, for each of the identified elements, the identification of one or more elements and the field determination simultaneously as a single step / process within the kernel machine.

[0067] Although not shown in Figure 2, in some embodiments where multiple elements are identified, the decision unit 210 may be configured to determine the fields of the identified elements by grouping the multiple identified elements into one or more groups. In these embodiments, each group may correspond to a field. The grouping may be based on a word grouping algorithm. More specifically, the decision unit 210 may be configured to group the multiple identified elements as follows: - Generate a linear text sequence of multiple identified elements in an unlabeled document. - Determine the classification of each element in the generated linear text sequence (e.g., using a kernel machine). - In the generated linear text sequence, group all the largest consecutive subsequences of elements that share a classification into their respective groups.

[0068] The classification determined for each element may be an argmax classification. The classification determination by the decision unit 210 may be based on multiple possible classifications, each possible classification corresponding to either a field in the labeled document or an indication that the element is not associated with any field in the labeled document.

[0069] Furthermore, in these embodiments, the system 200 may further include an assignment unit configured to assign each row identifier to each element in a group based on the relative order of the elements in the group.

[0070] In some embodiments, one or more fields and one or more elements in a labeled document may be presented in an information structure. In these embodiments, the system 200 may further include an output unit configured to output the identified one or more elements and their corresponding fields in the same (or similar) information structure. Alternatively, the output unit may be configured to output only one or more elements (regardless of whether the fields and / or elements are presented in the same / similar or different information structures).

[0071] The system 200 may further include an assignment unit configured to assign to each of the one or more identified elements a new label indicating the field to which each element corresponds.

[0072] In some embodiments, the decision unit 210 may be further configured to convert each of the one or more identified elements into a normalized form. This operation may be based on field-independent rules or on one or more field-specific rules.

[0073] In some embodiments, it will be understood that the use of labels in a labeled document is sufficient to indicate which elements are of interest, and therefore sufficient to indicate which corresponding elements in an unlabeled document should be identified and captured. However, in some embodiments, the method may further include a capture unit configured to obtain a schema that defines one or more fields contained in a labeled document and whether elements corresponding to each field in the labeled document are displayed in a table, before the decision unit 210 determines one or more anchors. This schema can provide clearer information about the data to be captured from the unlabeled document. This schema can be used, for example, so that when the decision unit 210 determines anchors, it can use one or more fields defined by the schema as anchor points.

[0074] In some embodiments, the system 200 may further include a receiving unit configured to receive user input assigning one or more labels to elements in a labeled document before the decision unit 210 determines one or more anchors. In some implementations, the receiving unit may be configured to receive user input assigning labels by receiving user selections of fields in the labeled document and one or more user selections of one or more elements corresponding to the selected fields. For example, user selection of elements may be implemented, for example, by allowing the user to draw a rectangular box around the elements on the document via a user interface. The fields corresponding to the labels (which can be represented by rectangular boxes) can be indicated by visual indicators. For example, each rectangular box may be a color indicating the corresponding field of the element. Furthermore, in some embodiments, each label may contain metadata (e.g., metadata indicating whether the field is in numerical format, metadata indicating whether the element corresponding to the field is in tabular format, etc.).

[0075] In some embodiments, the system 200 may further include an acquisition unit configured to acquire a plurality of labeled document candidates. In these embodiments, the system 200 may further include a receiving unit configured to receive user input to select a labeled document from the plurality of labeled document candidates. Alternatively, in these embodiments, the decision unit 210 may further be configured to determine a labeled document from the acquired labeled document candidates based on a similarity metric between each of the plurality of labeled document candidates and an unlabeled document. More specifically, the decision unit 210 may be configured to determine a labeled document based on a similarity function and a best-match algorithm. The similarity function and the best-match algorithm have been described and explained in detail above with reference to Figure 1, so the technical functions of these components will not be repeated here and will be understood to be equivalent. The operations by the acquisition unit, receiving unit and decision unit described in these embodiments may be performed before the decision unit 210 determines one or more anchors.

[0076] Figures 3A and 3B illustrate exemplary labeled and unlabeled documents according to one embodiment of the present disclosure. As shown in Figures 3A and 3B, labeled document 310 and unlabeled document 320 are provided. In this embodiment, labeled document 310 and unlabeled document 320 are invoices (i.e., semi-structured documents).

[0077] In labeled document 310, information is presented in field element pair format or tabular format. For example, “Invoice Number” and “00123456” are displayed as field element pair, “Invoice Date” and “September 2, 2021” are displayed as field element pair, and “Total” and “4596” are displayed as field element pair. In these examples, “Invoice Number,” “Invoice Date,” and “Total” are considered fields, while “00123456,” “September 2, 2021,” and “4596” are considered corresponding elements.

[0078] In this example, the remaining information is displayed in a table format, with elements corresponding to one of the fields: “Contents,” “Quantity,” “Unit Price,” and “Total Amount.” More specifically, the elements “Canon 5D,” “Memory Card,” and “Lens” correspond to “Contents,” which are descriptions of the purchased items. Elements “1,” “2,” and “1” correspond to “Quantity,” with these numbers indicating the quantity of each purchased item. Elements “2499,” “149,” and “1799” correspond to “Unit Price,” with these numbers indicating the unit price of each purchased item. The elements “2499,” “298,” and “1799” correspond to “Total Amount,” with these numbers indicating the total amount (price) of each purchased item.

[0079] As shown in Figure 3A, all elements are labeled (indicated in the drawing by a box surrounding each element, along with corresponding reference numbers 311, 312, 313A-C, 314A-C, 315A-C, 316A-C, and 317). This can be achieved by receiving user input indicating which words and / or terms (or which parts of image pixels) in the document should be considered elements.

[0080] Labels indicate which field each element corresponds to. In this example, the label "00123456" (i.e., label 311) can indicate that each element corresponds to the "Invoice Number" field. Similarly, the label "September 2, 2021" (i.e., label 312) can indicate that each element corresponds to the "Invoice Date" field. Labels 313A-C indicate that the elements "Canon 5D", "Memory Card", and "Lens" correspond to the "Contents" field; labels 314A-C indicate that the elements "1", "2", and "1" correspond to the "Quantity" field; labels 315A-C indicate that the elements "2499", "149", and "1799" correspond to the "Amount" field; and label 317 indicates that the element "4596" corresponds to the "Total" field.

[0081] Next, referring to the unlabeled document 320 in Figure 3A, we can see that the invoice follows a similar information structure to the labeled document 310. Specifically, the unlabeled document 320 contains the same fields: "Invoice Number," "Invoice Date," "Contents," "Quantity," "Unit Price," "Amount," and "Total." By applying the computer implementation method 100 described above with reference to Figure 1, or by using the system 200 described above with reference to Figure 2, the elements within the unlabeled document can be identified and output.

[0082] An example of this output is shown in Figure 3B. The unlabeled document 320 is assigned labels 321, 322, 323, 324, 325, 326, and 327, corresponding to each identified element. In this case, the identified elements are "00123457", "September 3, 2021", "Sony ZV-1", "1", "699", "699", and "699". These identified elements correspond to the elements labeled in the labeled document 310, in their respective fields. Specifically, element "00123457" corresponds to element "00123456" in the field ("Invoice Number"), element "September 3, 2021" corresponds to element "September 2, 2021" in the field ("Invoice Date"), element "Sony ZV-1" corresponds to elements "Canon 5D", "Memory Card", and "Lens" in the field ("Contents"), and element "1" corresponds to elements "1", "2", and "1" in the field ("Quantity"). Corresponding to labels 314A-C, the unit price "699" (label 325) corresponds to the elements "2499", "149", and "1799" (labels 315A-C) within the field ("Unit Price"), the element "699" (label 326) corresponds to the elements "2499", "298", and "1799" (labels 316A-C) within the field ("Amount"), and the element "699" (label 326) corresponds to the element "4596" within the field ("Total").

[0083] Figure 4 shows an exemplary process for determining anchors between labeled and unlabeled documents in Figures 3A and 3B. Figure 4 shows the upper and lower linear text sequences. The upper linear text sequence can be generated by extracting all words from the labeled document 310 and ordering the extracted words into a linear text sequence (e.g., reading order). Similarly, the lower linear text sequence can be generated by extracting all words from the unlabeled document 320 and ordering the extracted words into a linear text sequence (e.g., reading order).

[0084] In this example, any words in the upper linear text sequence corresponding to the labeled elements (e.g., "00123456", "September 2, 2021", "Canon 5D", etc.) are ignored during the anchor determination process. Alternatively, in other embodiments or examples, words corresponding to the labeled elements may be removed from the linear text sequence during the anchor determination process.

[0085] Once linear text sequences are generated corresponding to the labeled document 310 and the unlabeled document 320, matches can be identified between the two linear text sequences. As indicated by the arrows between the upper and lower linear text sequences, in this example, 10 matches are identified (from left to right): “Invoice”, “Number”, “Invoice”, “Date”, “Content”, “Quantity”, “Unit”, “Price”, “Amount”, and “Total”. Note that, at least in this example, each single block of text can be considered a “word”. Therefore, “Invoice” and “Number” are considered two (separate) words, and “Unit cost” is considered a single field, but “Unit” and “Cost” are still considered two words.

[0086] Next, each identified match is designated as an anchor, and in this example, 10 anchors are displayed corresponding to the words “invoice,” “number,” “invoice,” “date,” “contents,” “quantity,” “unit,” “price,” “amount,” and “total.” While all matches identified in this example are exact matches, it will be understood that in other examples and embodiments, matches may not be 100% accurate (for example, due to OCR errors).

[0087] Figure 5 shows the anchors determined from the exemplary process shown in Figure 4. To illustrate one aspect of this disclosure, Figure 5 focuses only on the anchors indicated by arched arrows pointing to the word “Quantity” in both labeled document 510 (identical to labeled document 310 in Figures 3A and 3B) and unlabeled document 520 (identical to labeled document 320 in Figures 3A and 3B). These anchors are one of the matches identified by the process described above with reference to Figure 4.

[0088] As described above with reference to Figures 1 and 2, in some embodiments, the generation of an entire kernel involves generating a convolution kernel for each of one or more anchors, with each convolution kernel covering all pairs of words and / or terms within and between labeled and unlabeled documents. More specifically, each convolution kernel can measure the similarity of words and / or terms within and between labeled and unlabeled documents. Similarity is a function of the distance of each word and / or term from each point of the anchor, and is the similarity of the relative positions to each point of the anchor. Thus, in this example, the convolution kernel for the anchor "quantity" can measure the similarity of words and / or terms with respect to the anchor within labeled document 510, unlabeled document 520, and between labeled document 510 and unlabeled document 520.

[0089] As an example of how word similarity can be measured, Figure 5 focuses on the similarity between "1" in labeled document 510 and "1" in unlabeled document 520 (indicated by the downward arrows pointing to the two words). This similarity is a function of the distance between "1" and "quantity" in labeled document 510 (indicated by the short arrows in the document), the distance between "1" and "quantity" in unlabeled document 520 (indicated by the short arrows in the document), and the similarity of the relative position of the word "1" to the word "quantity" in each document.

[0090] As explained above, with reference to several formulas, in at least some implementations, words and / or terms are considered similar only if both are close to an anchor and their positions relative to the anchor are similar (by taking the product of the offset kernel and the relative kernel). In this case, the similarity measured by the convolution kernel will be high due to the proximity of "1" and "quantity" in labeled document 510, the proximity of "1" and "quantity" in unlabeled document 520, and the similarity of the relative positions of the word "1" to the word "quantity" in each document (both "1"s are located below the word "quantity"). On the other hand, the similarity measured for the same anchor between the word "4596" in labeled document 510 and the word "00123457" in unlabeled document 520 will be low because they are farther from the word "quantity" in each document and their relative positions to "quantity" are different.

[0091] Convolution kernels for other anchors can be generated in a similar manner, with each convolution kernel covering all pairs of words and / or terms within and between the labeled and unlabeled documents for each anchor. Once all convolution kernels have been generated, they can be summed up to produce a complete kernel for the purpose of identifying elements from the unlabeled document 520.

[0092] Figure 6 illustrates an exemplary labeled document and an exemplary unlabeled document according to one embodiment of the present disclosure. As shown in Figure 6, a labeled document 610 and an unlabeled document 620 are provided. In this embodiment, the labeled document and the unlabeled document are invoices (i.e., semi-structured documents).

[0093] In labeled document 610, some information is displayed in field element pair format. For example, "Invoice Reference Number" and "237481709" are displayed as field element pair, "Payment Total" and "£1901.98" are displayed as field element pair, and "Payment Due Date" and "May 16, 1995" are displayed as field element pair. In these examples, "Invoice Reference Number," "Payment Total," and "Payment Due Date" are considered fields, and "237481709," "£1901.98," and "May 16, 1995" are considered corresponding elements.

[0094] Some other information within the labeled document 610 is displayed in a tabular format, with elements corresponding to one of the following fields: “Contents,” “Price,” “Discount,” “Sales Tax,” and “Amount.” For example, elements such as “Energy Bar,” “100% Juice - Apple Grape,” and “Natural Mineral Water” correspond to “Contents,” and these are descriptions of the purchased items.

[0095] Furthermore, in this example, some information is presented as elements (the corresponding fields are not explicitly stated). For example, the element "Bridge Interiors, 56 Bridge Lane, Ilkley LS29 9EU" is an address and is therefore understood to correspond to the "Address" field. Although this field is not explicitly shown in the labeled document, label 612 assigned to this element indicates that this element corresponds to the "Address" field.

[0096] As shown in Figure 6, only some of the elements in labeled document 610 are labeled (indicated in the figure by boxes surrounding the elements corresponding to reference numbers 611, 612, 613, 614, and 615). This can be achieved by receiving user input indicating which words and / or terms (or which parts of image pixels) in the document should be considered elements. In this example, not all elements in labeled document 610 are assigned labels. This is because some elements (unlabeled) are not of interest, and therefore it is not necessary or required to capture these elements and their corresponding elements in fields. For example, although "Order" and "855496" are presented as a field element pair, "855496" is not assigned a label. As another example, none of the elements corresponding to "Price" are labeled in labeled document 610.

[0097] In this example, labels can indicate the field to which an element corresponds. For example, as mentioned above, label 612 can indicate that each element corresponds to the "Address" field. Similarly, the label "237481709" (i.e., label 611) indicates that each element corresponds to the "Invoice Reference Number" field, the label "£1901.98" (i.e., label 613) indicates that each element corresponds to the "Total Payment" field, the label "May 16, 1995" (i.e., label 614) indicates that each element corresponds to the "Due Date" field, and labels 615 for the various elements placed under "Contents" can indicate that these elements correspond to the "Contents" field.

[0098] Next, referring to the unlabeled document 620 in Figure 6, we can see that the invoice follows a similar information structure to the labeled document. Specifically, the unlabeled document 620 includes the same fields, such as "Invoice Reference Number," "Total Payment," "Payment Due Date," and "Contents." By applying the computer implementation method 100 described above with reference to Figure 1, or by using the system 200 described above with reference to Figure 2, the elements of the unlabeled document can be identified and output.

[0099] As an example of how anchors are determined in the data capture process, Figure 7A shows a text anchor determined based on the labeled and unlabeled documents in Figure 6. The labeled document 710 in Figure 7A corresponds to the labeled document 610 in Figure 6, and the unlabeled document 720 corresponds to the unlabeled document 620 in Figure 6. In this example, the text anchor is indicated by an arrow spanning the two documents, pointing to the word “Content” 711 in the labeled document 710 and the word “Content” 721 in the unlabeled document 721. This anchor can be determined in several steps. (1) Extract all words from the labeled document 710 and the unlabeled document 720 (e.g., using OCR). (2) The words contained in the labeled document 710 are ordered into a linear text sequence to generate a first linear text sequence. (3) The words contained in the unlabeled document 720 are ordered into a linear text sequence to generate a second linear text sequence. (4) Identify matches between the first linear text sequence and the second linear text sequence (ignore words corresponding to labeled elements, e.g., "energy bar"). (5) Specify the matched word as the anchor.

[0100] Once this anchor ("content") is determined, a convolution kernel for that anchor can be generated. The convolution kernel spans all pairs of words and / or terms within and between the labeled document 710 and the unlabeled document 720. The convolution kernel measures the similarity of words and / or terms within and between the labeled and unlabeled documents. Similarity is a function of the distance from each point of the anchor to each word, and is a similarity of the relative positions to each point of the anchor. For example, in these two documents, the term "energy bar" 712 and the word "sourdough bread mix" 722 in the labeled document would have a measured high similarity (with respect to the "content" anchor). This is because both are close to the word "content" in each document, and their relative positions to the word "content" are also similar (i.e., "energy bar" and "sourdough bread mix" are directly below "content" in each document).

[0101] Convolution kernels for other anchors can be generated in a similar manner, with each convolution kernel covering all pairs of words and / or terms within and between the labeled and unlabeled documents for each anchor. Once all convolution kernels have been generated, they can be summed up to produce a complete kernel for the purpose of identifying elements from the unlabeled document 720.

[0102] As another example of an anchor determination method in a data capture process, Figure 7B shows a visual anchor determined based on a labeled document 710 and an unlabeled document 720. In this example, the visual anchor is indicated by an arrow spanning the two documents, pointing to the horizontal dividing line 713 in the labeled document 710 and the horizontal dividing line 723 in the unlabeled document 720. This anchor can be determined by identifying a visual feature present in both the labeled and unlabeled documents and designating the identified visual feature as the anchor. In this example, the identified visual feature is the horizontal dividing line in the two documents. The identification of the visual feature can be performed based on an image alignment algorithm, such as one based on SIFT.

[0103] Once this visual anchor is determined, a convolution kernel can be generated for that anchor. Here, the convolution kernel spans all pairs of words and / or terms within and between the labeled document 710 and the unlabeled document 720. The convolution kernel measures the similarity of words and / or terms within and between the labeled document and the unlabeled document. The similarity is a function of the distance from each point of the anchor to each word and the similarity of the relative position of each anchor to the point.

[0104] For example, the term "100% Juice - Apple Grape" in labeled document 710 and the term "Protein Plus Bar - Cookies & Cream Flavor" in unlabeled document 720 would have a high measured similarity (in terms of visual anchors) because these terms are relatively close to the horizontal divider in each document, and their relative positions to the horizontal divider are similar. In contrast, the term "Pepper Mayonnaise" in labeled document 710 and the term "Onion Rings" in unlabeled document would have a low measured similarity (in terms of visual anchors) because at least the term "Pepper Mayonnaise" is far from the horizontal divider, and the relative positions of these terms ("Pepper Mayonnaise" vs. "Onion Rings") are significantly different.

[0105] Figure 8 shows a labeled document 810, a first unlabeled document 820, and a second unlabeled document 830. Labeled document 810 corresponds to labeled document 610 shown in Figure 6, and the first unlabeled document 820 corresponds to unlabeled document 620 shown in Figure 6. The second unlabeled document 630 is also an invoice, employing a similar information structure to the other two documents. In this example, anchors have been determined for labeled document 810 and unlabeled document 820, indicated by arrows pointing to box 816 in labeled document 810 and box 826 in the first unlabeled document 820, and anchor symbols within these two documents. These anchors can be used as a criterion for determining the anchor for a new unlabeled document (for example, one belonging to the same batch of unlabeled documents as 820). In this case, the new unlabeled document is the second unlabeled document 830. As shown in Figure 8, the new anchor, indicated by the arrows pointing to box 816 in the labeled document 810 and box 836 in the unlabeled document 830, and the anchor symbols in the two documents, can thus be determined.

[0106] Figure 9 shows a topology for performing data capture according to one embodiment of the present disclosure. The topology 900 may include client computing devices 905 and a server 920 configured to communicate via a network 910.

[0107] Network 910 may be a wired or wireless network such as the Internet, intranet, local area network (LAN), wide area network (WAN), near-field communication (NFC) network, Bluetooth®, infrared, radio frequency, cellular network, or other types of networks. It will be understood that Network 910 may be a combination of several different types of wired or wireless networks.

[0108] Each client computing device 905 may be a smartphone, tablet computer, laptop computer, computer, personal data assistant, or other type of mobile device having a hardware processor configured to process instructions and connected to one or more parts of the network 910. Each client computing device 905 may have a graphical user interface configured to allow a user to interact with the processor of the client computing device 905.

[0109] Server 920 may include a physical computing device located in a specific location, or it may be deployed in a cloud computing network environment. In this disclosure, “cloud computing” may be defined as a model that enables ubiquitous, convenient, on-demand network access to shared, pooled, configurable computing resources (e.g., networks, servers, storage, applications, and services) that are rapidly provisioned by virtualization, released with minimal administrative work or interaction with service providers, and subsequently scaled as appropriate. Cloud models can consist of various characteristics (e.g., on-demand self-service, extensive network access, resource pooling, rapid scalability, measured services), service models (e.g., Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS)), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.). Server 920 may include any combination of media usable by or readable by one or more computers. In some embodiments, Server 920 may be configured to perform data capture. For example, Server 920 may perform at least part of Method 100 described above with reference to Figure 1.

[0110] It should be understood that this explanation is merely illustrative. Modifications and changes can be made to the described embodiments without departing from the scope of disclosure as defined in the claims.

[0111] While this technology is described in detail based on the most practical and preferred implementation currently available, it should be understood that such detailed descriptions are for illustrative purposes only, and the technology is not limited to the disclosed implementation, but rather aims to cover modifications and equivalent configurations included in the attached claims. For example, it should be understood that, to the extent possible, the technology assumes that one or more functions of any implementation can be combined with one or more functions of any other implementation.

[0112] Throughout this specification, the expressions “one embodiment,” “an embodiment,” “an example,” or “an example” mean that a particular feature, structure, or characteristic described in relation to an embodiment or example is included in at least one embodiment of this disclosure. Therefore, the expressions “one embodiment,” “an embodiment,” “an example,” or “an example” in various parts of this specification do not necessarily all refer to the same embodiment or example. Furthermore, a particular feature, structure, or characteristic may be combined in any suitable combination and / or partial combination in one or more embodiments or examples. Also, please note that the accompanying drawings are generally intended for those skilled in the art and are not necessarily drawn to scale.

[0113] Embodiments provided in this disclosure can be embodied as apparatus, methods, or computer program products. For example, in some embodiments, a computer program product may be provided that, when executed by a processor, causes the processor to perform the method described with respect to Figure 1. In other examples, in some embodiments, a computer-readable storage medium may be provided that, when executed by a computer, causes the computer to perform the steps of the method described with respect to Figure 1. Thus, embodiments can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which can be generally referred to as “modules” or “systems.” Furthermore, embodiments of this disclosure can take the form of computer program products embodied in any tangible medium of representation, in which computer-readable program code is embodied in any physical medium.

[0114] While described in relation to exemplary computing system environments, embodiments of the disclosure can operate in a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use in embodiments of the disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the systems or devices described above.

[0115] Any combination of media usable by or readable by one or more computers can be used. For example, computer-readable media may include one or more of the following: portable computer diskettes, hard disks, random access memory (RAM) devices, read-only memory (ROM) devices, erasable and programmable read-only memory (EPROM or flash memory) devices, portable compact disk read-only memory (CDROM), optical storage devices, and magnetic storage devices. Computer program code for performing operations of embodiments of this disclosure can be written in any combination of one or more programming languages.

[0116] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a block diagram and / or flowchart may represent a module, segment, or portion of code consisting of one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that performs a specified function or operation. These computer program instructions may also be stored on a computer-readable medium that can instruct a computer or other programmable data processing device to function in a particular way. Instructions stored on a computer-readable medium produce a product containing instruction means that implement the functions / operations specified in the flowchart and / or block diagram.

[0117] The following is a list of embodiments of the present disclosure.

[0118] 1. A computer implementation method for performing data capture in an unlabeled document based on a labeled document containing one or more elements, each labeled for a field, Determining one or more anchors having a pair of matching points between the labeled document and the unlabeled document, The entire kernel is generated using the one or more anchors determined above, Using the entire kernel within the kernel machine to identify one or more elements to be captured in the unlabeled document, Includes, A method wherein the one or more elements to be captured correspond in fields to the one or more labeled elements contained in the labeled document.

[0119] 2. Generating the entire kernel is A convolution kernel is generated for each of the one or more anchors mentioned above, The method according to Embodiment 1, comprising generating the entire kernel by adding the one or more convolution kernels.

[0120] 3. The method according to Embodiment 2, wherein the entire kernel is a weighted sum of one or more convolution kernels.

[0121] 4. The method of Embodiment 2 or Embodiment 3, wherein each convolution kernel covers all word pairs within and between the labeled and unlabeled documents.

[0122] 5. The method according to any one of embodiments 2 to 4, wherein each convolution kernel measures the similarity of words within and between the labeled and unlabeled documents, the similarity being a function of the distance from each point of the anchor to each word, and the similarity of the relative position of the anchor to each point.

[0123] 6. A method according to any one of Embodiments 2 to 5, The labeled document is defined as d=d1, the unlabeled document as d=d2, where d is a document, and each document is a word w d =w d,1 ,…,w d,nd Defined as a sequence of n d This is the number of words in each document, Here, each convolution kernel is k(w d,i ,w d’,j ;a)=k off (z)k off (z')k rel (z,z') is defined as (w d,i ,w d’,j ) is expressed as, 'a' represents each anchor, defined as a=(a1,a2), where a1 is the position of the corresponding point of the anchor in d1, and a2 is the position of the corresponding point of the anchor in d2. Here,

number

number

number

number

number

[0124] 7. A method according to Embodiment 6, wherein d = d' and / or d ≠ d'.

[0125] 8. A method according to any one of the prior embodiments, After identifying one or more elements in the unlabeled document, Receiving user input to confirm or reject the aforementioned specific result, and Receiving user input to correct the aforementioned specific result, A method further comprising at least one of the following.

[0126] 9. A method according to any one of the prior embodiments, The method wherein the labeled document further includes the one or more fields indicated by the one or more labels.

[0127] 10. A method according to any one of the prior embodiments, A method further comprising determining, for each of the one or more elements identified, the field to which each element corresponds, wherein the determination uses the entire kernel in a kernel machine.

[0128] 11. The method according to Embodiment 10, A method comprising identifying a plurality of elements in the unlabeled document, and determining the field to which each of the identified elements corresponds, by grouping the plurality of identified elements into one or more groups, where each group corresponds to a field.

[0129] 12. The method described in Embodiment 11, The method for grouping the aforementioned multiple identified elements is based on a word grouping algorithm.

[0130] 13. The method according to Embodiment 11 or Embodiment 12, Grouping the aforementioned multiple identified elements is To generate a linear text sequence of the multiple identified elements in the unlabeled document, The process involves determining the argmax classification of each element in the generated linear text sequence, The process involves grouping all of the longest continuous subsequences of elements that share the argmax classification in the generated linear text sequence into groups, Methods that include...

[0131] 14. A method according to any one of embodiments 11 to 13, A method further comprising assigning a row identifier to each element in a group based on the relative order of the elements in the group.

[0132] 15. A method according to any one of embodiments 9 to 14, A method comprising presenting the one or more fields and one or more elements in the labeled document in an information structure, and further outputting the identified one or more elements and their corresponding fields in the same information structure.

[0133] 16. A method according to any one of Embodiments 9 to 15, A method further comprising defining the one or more fields contained in the labeled document and obtaining a schema that defines whether elements corresponding to each field in the labeled document are displayed in a table, before determining one or more anchors.

[0134] 17. A method according to any one of the prior embodiments, Determining the anchor is Extracting all words from the labeled document and the unlabeled document, The words contained in the labeled document are ordered into a linear text sequence to generate a first linear text sequence, and all words corresponding to the labeled elements are deleted. The words contained in the aforementioned unlabeled document are ordered into a linear text sequence to generate a second linear text sequence. Identifying a match between the first linear text sequence and the second linear text sequence, and Specify the aforementioned matched word as the anchor. Methods that include...

[0135] 18. The method described in Embodiment 17, A method wherein the match between the first linear text sequence and the second linear text sequence exceeds a predetermined similarity threshold and a predetermined alignment threshold.

[0136] 19. The method described in Embodiments 17 to 18, A method for extracting the words from the labeled and unlabeled documents, based on optical character recognition (OCR).

[0137] 20. A method according to any one of embodiments 17 to 19, A method for performing ordering on words contained in the labeled document and the unlabeled document using a reading order algorithm.

[0138] 21. A method according to any one of Embodiments 17 to 20, A method in which each of the one or more anchors is defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0139] 22. The method according to any one of Embodiments 17 to 21, A method for identifying matches between the first linear text sequence and the second linear text sequence, based on an alignment algorithm.

[0140] 23. The method described in Embodiment 22, The alignment algorithm is a method that is similar to or equivalent to the Needleman-Wunsch algorithm.

[0141] 24. A method according to any one of the prior embodiments, Determining the anchor is Identifying visual features present in both the labeled document and the unlabeled document, and Designating the aforementioned identified visual feature as an anchor, Methods that include...

[0142] 25. The method described in Embodiment 24, Identifying the aforementioned visual features is a method based on an image alignment algorithm.

[0143] 26. The method according to Embodiment 25, The image alignment algorithm is based on at least one of a keypoint detector, a locally invariant descriptor, and a keypoint matching process.

[0144] 27. A method according to any one of the prior embodiments, Determining the anchor is Placing multiple points around the boundary of the unlabeled document and placing the same number of points around the boundary of the labeled document, Identifying matches between points in the labeled document and points in the unlabeled document, The matching point is designated as the anchor, Methods that include...

[0145] 28. A method according to any one of the prior embodiments, A method further comprising receiving user input assigning one or more labels to the elements of the labeled document before determining one or more anchors.

[0146] 29. A method according to any one of the prior embodiments, Before determining one or more anchors, Obtaining multiple labeled document candidates, From the acquired labeled document candidates, the labeled document is determined based on the similarity measurement value between each of the multiple labeled document candidates and the unlabeled document. Methods that further include this.

[0147] 30. A method according to any one of the prior embodiments, The kernel machine is a method comprising at least one of the following: support vector machine, SVM, regularized kernel discriminant analysis, RKDA, Gaussian process, or GP.

[0148] 31. A method according to any one of the prior embodiments, The method involves the labeled document being in PDF document or image format.

[0149] 32. A system for performing data capture in an unlabeled document based on a labeled document containing one or more elements, each labeled for a field, A determination unit configured to determine one or more anchors having a pair of corresponding points that match between the labeled document and the unlabeled document, A generation unit configured to generate the entire kernel using one or more anchors determined above, The kernel machine is configured to use the entire kernel to identify one or more elements to be captured in the unlabeled document containing information to be captured, and the one or more elements to be captured are specific units that correspond in fields to the one or more labeled elements contained in the labeled document, A system equipped with these features.

[0150] 33. A computer-readable storage medium, A storage medium that, when executed by a computer, includes instructions causing the computer to perform a step of any one of the methods of Embodiments 1 to 31.

Claims

1. A computer implementation method (100) for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the method Step (S110): A step of determining one or more anchors, wherein each anchor has a pair of location points of matching words or visual features in the labeled document and the unlabeled document. Step (S120) to generate an overall kernel using one or more anchors determined above, Step (S130): Identifying one or more elements to be captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements contained in the labeled document. A method of having.

2. The step of generating the overall kernel (S120) is, A step of generating a convolution kernel for each of the one or more anchors mentioned above. A step of generating the overall kernel by adding one or more of the aforementioned convolution kernels, The method according to claim 1, comprising having

3. The method according to claim 2, wherein each convolution kernel measures the similarity of words within and between the labeled and unlabeled documents, the similarity being a function of the distance from each point of the anchor to each word and the similarity of the relative position of the anchor to each point.

4. The method according to claim 1, wherein the labeled document further comprises the one or more fields indicated by the one or more labels.

5. The one or more fields and one or more elements within the labeled document are represented by an information structure. The method further comprises the step of outputting the identified one or more elements and their corresponding fields in the same information structure. The method according to claim 4.

6. The method further includes a step of obtaining a schema, which defines one or more fields contained in the labeled document and defines whether elements corresponding to each field in the labeled document are presented in a table, prior to the step of determining the one or more anchors (S110). The method according to claim 4.

7. The method further comprises the step of determining, for each of the one or more identified elements, the field to which the element corresponds, wherein the determination uses the kernel machine. The method according to claim 1.

8. The step of identifying multiple elements in the unlabeled document and determining the field to which each identified element corresponds includes the step of grouping the multiple identified elements into one or more groups, where each group corresponds to a field. The method according to claim 7.

9. The step of grouping the aforementioned multiple identified elements is: A step of generating a linear text sequence of the plurality of identified elements in the unlabeled document, A step of determining the argmax classification of each element in the generated linear text sequence, The step of grouping all of the longest continuous subsequences of elements that share the argmax classification in the generated linear text sequence into groups, Having, The method according to claim 8.

10. The one or more fields and one or more elements within the labeled document are represented by an information structure. The method further comprises the step of outputting the identified one or more elements and their corresponding fields in the same information structure. The method according to claim 7.

11. The method further includes a step of obtaining a schema, which defines one or more fields contained in the labeled document and defines whether elements corresponding to each field in the labeled document are presented in a table, prior to the step of determining the one or more anchors (S110). The method according to claim 7.

12. The step of determining the anchor (S110) is, A step of extracting all words from the labeled document and the unlabeled document, A step of generating a first linear text sequence by rearranging the words contained in the labeled document into a linear text sequence and deleting all words corresponding to the labeled elements, A step of generating a second linear text sequence by rearranging the words contained in the aforementioned unlabeled document into a linear text sequence, Steps to identify a match between the first linear text sequence and the second linear text sequence, The step of designating the matching word as the anchor, Having, The method according to claim 1.

13. The step of determining the anchor (S110) is, A step of identifying visual features present in both the labeled document and the unlabeled document, The step of designating the identified visual feature as the anchor, Having, The method according to claim 1.

14. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 13.

15. The step of determining the anchor (S110) is, The steps include: placing multiple points around the boundary of the unlabeled document and placing the same number of points around the boundary of the labeled document; A step of identifying a match between a point on the labeled document and a point on the unlabeled document. The step of designating the aforementioned matching point as the anchor, Having, The method according to claim 1.

16. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 15.

17. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 1.

18. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 13.

19. A system (200) that performs data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the system A decision unit (210) configured to determine one or more anchors, wherein the anchors have a pair of location points of matching words or visual features in the labeled document and the unlabeled document. A generation unit (220) configured to generate an overall kernel using one or more of the determined anchors, An identification unit (230) that identifies one or more elements captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements captured correspond in a field to the one or more labeled elements contained in the labeled document, A system (200) equipped with the following features.

20. A non-temporary computer-readable storage medium, which, when executed by a computer, includes instructions causing the computer to perform a method (100) of performing data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the method Step (S110): A step of determining one or more anchors, wherein each anchor has a pair of location points of matching words or visual features in the labeled document and the unlabeled document. Step (S120) to generate an overall kernel using one or more anchors determined above, Step (S130): Identifying one or more elements to be captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements contained in the labeled document. A computer-readable storage medium having [a certain characteristic].

Images