Method and system for performing data capture

Abstract

A method (100) for performing data capture on an unlabeled document based on a labeled document including one or more elements each labeled for a field is provided. The method (100) includes determining (S110) one or more anchors having a pair of matching points between the labeled document and the unlabeled document, generating (S120) an entire kernel using the determined one or more anchors, and identifying (S130) one or more elements to be captured in the unlabeled document using the entire kernel in a kernel machine. The one or more elements to be captured correspond to one or more labeled elements included in the labeled document in a field.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for performing data capture. More specifically, the present disclosure relates to systems and computer-implemented methods for performing data capture on one or more unlabeled documents based on one or more labeled documents.

Background Art

[0002] Various types of documents are widely used for collecting and recording information for many purposes and fields such as medical, commercial, educational, and government. Today, because computers and communication networks are widely used, these documents are now usually created electronically and provided to be digitally generated and shared. These documents usually contain data in a structured or semi-structured format. Structured documents may have codes embedded therein to enable information to be placed in a specific format. Semi-structured documents are documents that do not follow a strict layout like structured documents, such as invoices, bank transaction statements, utility bills, passports, etc., and may not be limited to specific data fields like structured documents.

[0003] To utilize information from these electronic documents to perform desired operations (such as enterprise-related operations), it is necessary to appropriately extract the information. However, at least for semi-structured electronic documents, since these documents often do not have a clearly defined layout, it may be difficult to reliably analyze and extract the necessary information.

Summary of the Invention

[0004] Data capture is the task of extracting relevant information from documents. The one-shot data capture (OSDC) technology described in this disclosure is designed to extract information from structured and semi-structured documents. The data to be captured may include the values of key-value pairs within the document, such as claim numbers, total amounts, and / or values within tables in the document. In the context of this disclosure, key-value pairs may sometimes be referred to as "field-element pairs", where the key may be called the "field" and the value may be called the "element". In its simplest form, the OSDC technology operates on pairs of labeled and unlabeled documents. The task in this case is to extract the same type of information from the unlabeled document as that which has been labeled in the labeled document.

[0005] The embodiments described herein are directed to methods and systems for performing data capture on unlabeled documents based on labeled documents.

[0006] According to one aspect of the present disclosure, there is provided a computer-implemented method for performing data capture on an unlabeled document based on a labeled document that includes one or more elements each labeled with respect to a field. The computer-implemented method includes determining one or more anchors having a matching pair of points between the labeled document and the unlabeled document, generating an entire kernel using the determined one or more anchors, and using the entire kernel within a kernel machine to identify one or more elements to be captured within the unlabeled document, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements included in the labeled document.

[0007] According to another aspect of the present disclosure, a system for performing data capture on a document without labels based on a labeled document including one or more elements each labeled for a field is provided. The system includes a determination unit configured to determine one or more anchors having a pair of corresponding points that match between the labeled document and the document without labels, a generation unit configured to generate an entire kernel using the determined one or more anchors, and a specific unit configured to use the entire kernel in a kernel machine to identify one or more elements to be captured including information to be captured in the document without labels, wherein the one or more elements to be captured correspond to the one or more labeled elements included in the labeled document in a field.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided. The storage medium includes instructions that, when executed by a computer, cause the computer to execute the steps of the method described in the present disclosure.

[0009] Optional features are described in the appended dependent claims.

[0010] These and other aspects of the present disclosure will be better recognized and understood when considered in conjunction with the following description and the accompanying drawings. The following description gives, by way of illustration and not limitation, various embodiments of the present disclosure and some of their specific details. Many substitutions, changes, additions or rearrangements are possible within the scope of the present disclosure, and the present disclosure includes all such substitutions, changes, additions or rearrangements.

Brief Description of the Drawings

[0011] Hereinafter, embodiments of the present disclosure will be described by way of example only with reference to the accompanying drawings.

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6-1

Figure 6-2

Figure 7A

Figure 7B

Figure 8-1

Figure 8-2

Figure 8-3

Figure 9

[0012] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments may be practiced without these specific details. In other instances, well-known materials or methods have not been described in detail so as not to obscure the present disclosure.

[0013] FIG. 1 is a flowchart of a computer-implemented method 100 for performing data capture according to an embodiment of the present disclosure. More specifically, the illustrated method 100 is for performing data capture in a document without labels based on a labeled document. The labeled document includes one or more elements each labeled with respect to a field. Each element may be represented by a word, a string of words, a number, or a combination thereof. In some embodiments, the labeled document may further include one or more fields indicated by one or more labels. The labeled document and / or the document without labels may be a PDF document or an image. In some implementations, this method may be performed by a client computing device, a network, a server, or a combination thereof, such as the client computer device 905, network 910, and server 920 shown in FIG. 9.

[0014] Method 100 begins at step S110, where one or more anchors are determined. An anchor comprises a pair of points that match between the labeled document and the document without labels. In some implementations, a new set of one or more anchors is determined for each new pair of documents (the pair includes a document without labels and a labeled document).

[0015] In some embodiments, determining the anchor in step S110 may include receiving user input selecting a pair of points in the document without labels and the labeled document. Alternatively, in some embodiments, determining the anchor in step S110 may include the following steps. - Extract all words from labeled and unlabeled documents (e.g., using OCR). - Order the words contained in the labeled document into a linear text sequence to generate a first linear text sequence (optionally, delete any words corresponding to labeled elements). - Order the words contained in the unlabeled document into a linear text sequence to generate a second linear text sequence. - Identify the matches between the first linear text sequence and the second linear text sequence (if words corresponding to labeled elements have not been deleted at this stage, these words are ignored in the match identification step). - Designate the matched words as anchors.

[0016] Matches in this context include the inherent similarity of two words and their alignment (i.e., similarity in terms of the linear order of each word in each sequence). In these embodiments, the match between the first text sequence and the second text sequence may exceed a predetermined threshold both in terms of the inherent similarity (e.g., visual features) and alignment (i.e., how similar they are in terms of the linear order of each in each text sequence). In other words, the match may be two words that have the same or similar appearance and the linear order in each text sequence is the same or similar. As an example of visual similarity, the words "bam" and "barn" can be determined to exceed a predetermined visual similarity threshold by visual similarity (e.g., considering "m" and "rn" as OCR (Optical Character Recognition) errors). As an example of alignment, the 8th word of the first text subsequence will likely have a higher degree of alignment with the 8th word of the second text subsequence than with the 7th word of that text subsequence. In some embodiments, the identification of the match (more specifically, the determination of whether two words exceed the alignment threshold) can be based on an alignment algorithm such as an algorithm similar to or equivalent to the Needleman-Wunsch algorithm.

[0017] Alternatively or additionally, in these embodiments, the ordering of words included in labeled and unlabeled documents can be performed using a reading order algorithm (i.e., any algorithm that attempts to order words as a human would read them).

[0018] Alternatively or additionally, in these embodiments, each of one or more anchors may be defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0019] In some embodiments, the determination of the anchors in step S110 may include identifying visual features present in both the labeled and unlabeled documents (e.g., using an image alignment algorithm), and designating the identified visual features as the anchors. The image alignment algorithm may be based on at least one of a keypoint detector (e.g., Difference of Gaussians, Harris, Good Features to Track (GFTT), local invariant descriptors (e.g., Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), etc.), and a keypoint matching process (e.g., Random Sample Consensus (RANSAC)).

[0020] In some embodiments, the determination of the anchors in step S110 may include placing a plurality of points around the boundary of the unlabeled document, placing the same number of points around the boundary of the labeled document, identifying the matches between the points of the labeled document and the points of the unlabeled document, and designating the matched points as the anchors.

[0021] In this embodiment, it is described that one or more anchors are determined for a pair of documents (i.e., a labeled document and an unlabeled document). However, it will be understood that the process of determining the above-mentioned anchors can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). For example, an anchor may have multiple points, and these points may match (i.e., exceed a visual similarity threshold and an alignment threshold), and the multiple points are in different documents respectively.

[0022] Also, in this embodiment, it is described that an anchor can include a single word (consisting of numbers or may or may not include numbers). However, in other embodiments, an anchor can include terms that match in addition to or instead of words. For example, multiple words such as "Total amount" are treated as a single term rather than two separate words. As a result, each anchor that matches the term "Total amount" matches in a pair of documents.

[0023] Returning to FIG. 1, in step S120, the entire kernel is generated using the one or more determined anchors. In some embodiments, generating the entire kernel in step S120 may include generating a convolutional kernel for each of the one or more anchors determined in step S110 and summing the one or more convolutional kernels to generate the entire kernel. The entire kernel may be a weighted sum of the one or more convolutional kernels.

[0024] Each convolutional kernel can span all word pairs within and between labeled and unlabeled documents. In other embodiments where more than two documents are involved in data capture, each convolutional kernel can span all word pairs within and across all pairs of more than two documents. Each convolutional kernel can measure the word similarity within and between a labeled document and an unlabeled document (or, when more than two documents are involved, between all documents), where the similarity is a function of the distance from each point of the anchor to each word and is the similarity of the relative positions with respect to each point of the anchor.

[0025] To describe the convolutional kernel in more technical detail, define a document d as a sequence of words w d =w d,1 ,…,w d,nd where n d is the number of words in the document.

[0026] If the document is labeled, a label l d =l1,…,l nd may be provided for each word, and the labels belong to one of K categories (each category corresponding to a field).

Number

[0027] Furthermore, each word may have a position on the page of the document. The position of the i'-th word w d,i’ in the d'-th document is designated as x d,i .

[0028] An anchor a spanning two documents d1 and d2 can be represented as a pair of positions a=(a1,a2) (one in each document). Here, a1 is the position in d1 and a2 is the position in d2. The labeled document is defined as d=d1 and the unlabeled document is defined as d=d2.

[0029] A pair of words (w d,i , w d’,j ) The convolution kernel k acting on can be defined as the product of the other radius basis function (RBF) kernels k off and k rel with respect to the anchor a. The key input to the kernel is the relative position of the word with respect to the anchor and is defined as follows.

Number

Number

[0030] The kernel can be defined as follows.

Number

Number

Number

Number

[0031] Note that since the convolution kernel can act on any pair of words regardless of whether they are in the same document, d = d' or d ≠ d' is possible.

[0032] The linear transformation (matrix) A off and A rel act as inverse length scales. In one implementation, these are parameterized diagonally and σ is the length scale.

Number

Number

[0033] This allows the kernel to have different effects on each direction within the document. For example, the value of σ can be set such that the influence on the width is more acceptable than the influence on the height within the document.

[0034] The basis of the above kernel equation is that the offset kernel k off decays as the word moves away from the anchor. That is, the anchor has only a local effect. In contrast, the relative kernel k rel quantifies the similarity of the spatial relationship between the anchor and each word. That is, words are similar when they are placed at the same position relative to the anchor. By taking the product of the kernels, it can be made such that they are considered similar only when both are close to the anchor and their positions relative to the anchor are similar.

[0035] As mentioned above, while each convolutional kernel can span word pairs, in an alternative embodiment, the convolutional kernels can each span term pairs (e.g., each term is composed of multiple words) instead of, or in addition to, single-word pairs.

[0036] As described above, in some embodiments, the process of determining the anchor can be applied to more than two documents (e.g., multiple labeled documents and unlabeled documents), and each anchor may have multiple matching points in different documents. In these embodiments, the anchor is a=(a1,a2,…,a N) may be defined, where N-1 is the number of labeled documents selected for data capture (therefore, N is the total number of documents). Further, in these embodiments, each convolutional kernel can span all word pairs within all documents and all word pairs across all documents, and the overall kernel is the sum of all convolutional kernels. This larger overall kernel (compared to the case of using only two documents) can provide more information by the kernel machine for identifying one or more elements to be captured in step S130. Each convolutional kernel defined with respect to a document set can be represented as a symmetric positive semi-definite matrix having dimensions corresponding to the number of words in all documents within each document set. Thus, in this case, each entry in the matrix corresponds to each similarity between specific word pairs included in the documents within the document set. Similarly, since the overall kernel is generated by summing the convolutional kernels, this can also be represented as a symmetric positive semi-definite matrix, and each entry in that matrix corresponds to each similarity between specific word pairs among the words included in the documents within the document set.

[0037] Returning to FIG. 1, in step S130, using the entire kernel in the kernel machine, one or more elements captured in the unlabeled document are identified. The one or more elements captured correspond in fields to one or more labeled elements included in the labeled document. The kernel machine has at least one of a support vector machine (SVM), a regularized kernel discriminant analysis (RKDA), and a Gaussian process (GP). The kernel machine may be trained based on at least one of one or more elements and one or more fields indicated by each label in the labeled document. In some cases, the kernel machine may be trained based on all of the one or more elements and one or more fields indicated by each label in the labeled document in order to maximize the number of samples used for training. In some embodiments, the kernel machine may be trained using a subset of the matrix entries for the entire kernel, where the entries of each matrix in the subset of the matrix entries for the entire kernel correspond to word pairs in the labeled document. The identified elements may correspond to words having a high (predicted) similarity / reliability score determined by the kernel machine. In these embodiments, the method may further include outputting the similarity / reliability score determined by the kernel machine.

[0038] Although not shown in FIG. 1, in some embodiments, method 100 may further include, after identifying one or more elements in step S130, receiving a user input to confirm or reject a specific result, and receiving a user input to modify the specific result. In these embodiments, if a user input to confirm the result is received, method 100 may further include outputting the result (e.g., via a display), and may also perform further processing of the result (e.g., determination and / or grouping of fields as described below). If a user input to modify the specific result is received, method 100 may further include performing a modification based on the user input and / or saving the modified result as at least part of a newly labeled document.

[0039] In some embodiments, this method may further include, for each of the one or more identified elements, determining the field to which each element corresponds. This determination operation can also be performed using the entire kernel within the kernel machine. It will be understood that in some embodiments, the operation of identifying one or more elements in step S130 and the operation of determining the field for each of the one or more identified elements can be performed simultaneously as a single step / process within the kernel machine.

[0040] In some embodiments where there are multiple identified elements, the operation of field determination for the identified elements includes grouping the multiple identified elements into one or more groups, with each group corresponding to a field. This grouping can be based on a word grouping algorithm. More specifically, the grouping of the multiple identified elements can include the following. - Generate a linear text sequence of the multiple identified elements in the document without labels (e.g., in reading order using a reading order algorithm). - Determine the classification of each element in the generated linear text sequence (e.g., using a kernel machine). - Group all of the maximum consecutive subsequences of elements that share a classification in the generated linear text sequence into each group.

[0041] The classification determined for each element may be an argmax classification. The determination of the classification can be made based on a plurality of possible classifications, where each possible classification corresponds to a field in the labeled document or an indication indicating that the element is not associated with any field in the labeled document.

[0042] In embodiments where a plurality of identified elements are grouped (e.g., using the grouping algorithm described above), the method may further include assigning a respective row identifier to each element within the group based on the relative order of the elements within the group. Thereby, the elements belonging to the tabular form can be output as they are (i.e., the elements having the same row identifier are arranged in the same row).

[0043] In some embodiments, one or more fields and one or more elements in the labeled document may be presented in an information structure. For example, in a claim, one or more fields may be presented as table headers, one or more elements may be presented under the table headers, each column corresponds to a field, and each row corresponds to a purchased item. In these embodiments, method 100 may further include outputting the identified one or more elements and their corresponding fields in the same (or a similar) information structure. Alternatively, method 100 may further include outputting only the one or more elements (regardless of whether the fields and / or elements are presented in the same / similar information structure or different information structures).

[0044] This method may further include assigning a new label indicating the field to which each element corresponds to each of the identified one or more elements.

[0045] In some embodiments, the method may further include converting each of the identified one or more elements into a normalized form. For example, all elements corresponding to date information may be returned in the format DD / MM / YYYY. This operation may be based on field-independent rules (i.e., one rule applicable to all fields). For example, a rule such as removing leading or trailing punctuation marks in each string of text of the identified element. Alternatively, this operation can also be performed based on one or more field-specific rules, and the field-specific rules can be set based on the metadata of the labels in the labeled document.

[0046] In some embodiments, the use of labels in a labeled document is sufficient to indicate which elements are of interest, and thus it will be understood that it is sufficient to identify and capture which corresponding elements in a label-free document. However, in some embodiments, the method may further include, before determining one or more anchors in step S110, defining one or more fields included in the labeled document and obtaining a schema that defines whether the elements corresponding to each field of the labeled document are displayed in a table. This schema can provide more explicit information about what data should be captured from the label-free document. This schema can be used, for example, in the determination of the anchors in S110 of method 100, such that each of the one or more fields defined by the schema can be used as an anchor point.

[0047] Although not shown in FIG. 1, in some embodiments, method 100 may further include receiving user input to assign one or more labels to elements of the labeled document before determining one or more anchors in step S110. In some implementations, receiving user input to assign labels may include receiving a user selection of fields within the labeled document and receiving a user selection of one or more elements corresponding to the selected fields. For example, the user selection of an element can be implemented by enabling the user to draw a rectangular box around the element on the document via a user interface such as a touch screen. The field corresponding to the label (which can be represented by the rectangular box) can be indicated by a visual indicator. For example, each rectangular box can be the color indicating the corresponding field of the element. In this case, the rectangular boxes of the elements corresponding to the same first field are the same first color, the rectangular box elements corresponding to the same second field are the same second color, and so on. Further, in some embodiments, each label may include metadata. For example, metadata for indicating whether the field is in numerical format and / or metadata for indicating whether the element corresponding to the field is displayed in tabular form, etc.

[0048] In some embodiments, method 100 may further include obtaining a plurality of labeled document candidates before determining one or more anchors and receiving user input to select a labeled document from the plurality of labeled document candidates. Alternatively, method 100 may further include obtaining a plurality of labeled document candidates before determining one or more anchors and determining a labeled document based on a similarity measure value between each of the obtained labeled document candidates and a document without a label. In some embodiments, determining a labeled document from a plurality of labeled document candidates may be based on a similarity function and a best match algorithm. The similarity function may be a function that can process a document pair (i.e., in this case, each labeled document candidate and the document without a label) and return a similarity measure value based on the processing. An example of the similarity measure value may be a score indicating the degree of similarity between two documents.

[0049] In some embodiments, the similarity function may be based on a bag-of-words model that compares document pairs by the number of common words across documents (either including or not including duplicate words). For example, if one document is composed of multiple tokens "a a a b c" and another document is composed of multiple tokens "a a c c d", the document pair shares three tokens (two "a"s and one "c"), so the similarity function may return a similarity score of 3.

[0050] Once the similarity scores for all pairs of the labeled document candidates and the document without a label are determined in this way, then a "best match" can be selected based on the similarity measure values between each of the plurality of labeled document candidates and the document without a label using a best match algorithm. For example, a best match algorithm can be used to select the document pair with the highest similarity score.

[0051] In some cases, a similarity score threshold may be used to determine that there are no multiple labeled document candidates suitable for performing data capture by the method described herein. For example, if the similarity score threshold is set to 4 and none of the similarity scores of the document pairs (each pair includes a labeled document candidate and an unlabeled document) are 4 or higher, the method may further include providing an output indicating that none of the (current) labeled document candidates can be used for data capture. Additionally, the method may further include obtaining a new set of multiple labeled document candidates and determining a labeled document based on the similarity measurement values between each of the new multiple labeled document candidates and the unlabeled document.

[0052] Alternatively, or in addition, in some cases, a similarity score threshold may be used to determine that multiple labeled document candidates are suitable for performing data capture by the method described herein. For example, if the similarity score threshold is set to 4 and multiple document pairs (each pair includes a labeled document candidate and an unlabeled document) have a similarity score of 4 or higher, the step of determining an anchor in step S110 may be based on multiple labeled documents (each corresponding to a similarity score higher than the threshold) and the unlabeled document. As described above, in this case, each anchor may have multiple matching points in different documents. Further, in this case, the step of generating the entire kernel in step S120 may be based on multiple labeled documents and the unlabeled document. That is, each convolutional kernel can span multiple labeled documents (each corresponding to a similarity score higher than the threshold) and the unlabeled document.

[0053] FIG. 2 is a block diagram of a system for performing data capture according to an embodiment of the present disclosure. More specifically, system 200 is for performing data capture on a document without labels based on a labeled document. The labeled document includes one or more elements each labeled with respect to a field. Each element may be represented by a word, a string of words, a numerical value, or a combination thereof. In some embodiments, the labeled document may further include one or more fields indicated by one or more labels. The labeled document and / or the document without labels may be a PDF document or an image.

[0054] As shown in FIG. 2, system 200 includes a determination unit 210, a generation unit 220, and an identification unit 230. The determination unit 210 is configured to determine one or more anchors. As described above, an anchor has a pair of matching points between the labeled document and the document without labels. In some implementations, the determination unit 210 may be configured to determine a new set of one or more anchors for each new pair of documents (the pair includes a document without labels and a labeled document).

[0055] In some embodiments, the determination unit 210 may be configured to determine an anchor by receiving a user input that selects a pair of points in the document without labels and the labeled document. Alternatively, in some embodiments, the determination unit 210 may be configured to determine an anchor as follows. - Extract all words in the labeled document and the document without labels (e.g., using OCR). - Order the words included in the labeled document into a linear text sequence to generate a first linear text sequence (optionally, delete any words corresponding to labeled elements). - Order the words included in the document without labels into a linear text sequence to generate a second linear text sequence. - Identify a match between a first linear text sequence and a second linear text sequence (if words corresponding to labeled elements have not been removed at this stage, those words are ignored in the match identification step). - Designate the matched words as anchors.

[0056] A match in this context includes not only the inherent similarity (e.g., visual similarity) of two words, but also similarity or match in terms of the linear order in each sequence for each word. In these embodiments, a match between a first text sequence and a second text sequence may be one that exceeds a predetermined threshold from both the perspectives of inherent similarity and alignment similarity. Further, the determination unit 210 may be configured to identify a match (more specifically, determine whether two words exceed an alignment similarity threshold) based on an alignment algorithm (e.g., an algorithm similar to or equivalent to the Needleman-Wunsch algorithm).

[0057] In these embodiments, the determination unit 210 may be configured to order the words included in the labeled document and the unlabeled document using a reading order algorithm. Further, in these embodiments, each of one or more anchors may be defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0058] In some embodiments, the determination unit 210 may be configured to determine an anchor by identifying visual features present in both labeled and unlabeled documents (e.g., using an image alignment algorithm), and designate the identified visual features as the anchor. The image alignment algorithm may be based on at least one of a keypoint detector (e.g., Difference of Gaussians, Harris, Good Features to Track (GFTT), local invariant descriptors (e.g., Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), etc.), and a keypoint matching process (e.g., Random Sample Consensus (RANSAC)).

[0059] In some embodiments, the determination unit 210 may be configured to determine an anchor by placing a plurality of points around the boundary of the unlabeled document, placing the same number of points around the boundary of the labeled document, identifying a match between the points of the labeled document and the points of the unlabeled document, and designating the matched points as the anchor.

[0060] In this embodiment, it is described that one or more anchors are determined for a pair of documents (i.e., a labeled document and an unlabeled document). However, it will be understood that the process for determining an anchor by the determination unit 210 described above can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). Also, in this embodiment, it is described that an anchor can include a single matching word, but in other embodiments, an anchor can include a matching term rather than a word. For example, the term "total amount" is treated as a single term rather than two separate words.

[0061] The generation unit 220 is configured to generate the entire kernel using the determined one or more anchors. In some embodiments, the generation unit 210 may be configured to generate the entire kernel by generating a convolutional kernel for each of the one or more anchors and summing the one or more convolutional kernels. The entire kernel may be a weighted sum of the one or more convolutional kernels.

[0062] Each convolutional kernel can span all pairs of words within and between the labeled and unlabeled documents. In some embodiments where the number of documents underlying the data capture exceeds two, each convolutional kernel can span all pairs of words within all documents and all pairs of words across all documents. Each convolutional kernel can measure the similarity of words within and between a labeled document and an unlabeled document (or, if more than two documents are involved, all documents). The similarity is a function of the distance from each point of the anchor to each word and is the similarity of the relative positions with respect to each point of the anchor. Since the mathematical details of the convolutional kernel were described above with reference to FIG. 1, they will not be repeated here for the sake of brevity. It will be understood that these formulas are applicable to the present embodiment. Also, as described above with reference to FIG. 1, the convolutional kernel and the entire kernel defined with respect to a set of documents can each be represented as a symmetric positive semi-definite matrix having dimensions corresponding to the number of words in all documents within the set of documents, and each entry of each matrix within the matrix corresponds to each similarity between a particular pair of words in the words within the documents within the set of documents.

[0063] Although it has been described above that each convolutional kernel may span pairs of words, in alternative embodiments, the convolutional kernels may span, instead of or in addition to, pairs of terms (e.g., composed of multiple words).

[0064] The specific unit 230 is configured to identify one or more elements to be captured in the unlabeled document using the entire kernel in the kernel machine. The one or more elements to be captured correspond in fields to one or more labeled elements included in the labeled document. The kernel machine may include at least one of a support vector machine (SVM), a regularized kernel discriminant analysis (RKDA), and a Gaussian process (GP). The kernel machine can be trained based on one or more elements and one or more fields indicated by each label in the labeled document. In some cases, the kernel machine may be trained based on all of the one or more elements and one or more fields indicated by each label in the labeled document to maximize the number of samples used for training. In some embodiments, the kernel machine may be trained using a subset of the matrix entries for the entire kernel, where the entries of each matrix in the subset of the matrix entries for the entire kernel correspond to pairs of words in the labeled document. The identified elements may correspond to words having a high (predicted) similarity / reliability score determined by the kernel machine. In these embodiments, the system 200 may include an output unit configured to output the similarity / reliability score determined by the kernel machine.

[0065] Although not shown in FIG. 2, in some embodiments, system 200 may further include a receiving unit configured to receive at least one of a user input to confirm or reject a particular result and a user input to modify a particular result after the identification unit 230 has identified one or more elements. Further, system 200 may include an output unit (e.g., via a display) configured to output the result and / or an execution unit (e.g., field determination and / or grouping, and thus the execution unit may be a determination unit) configured to perform further processing of the result when a user input to confirm the result is received. When a user input to modify a particular result is received, the execution unit may be configured to perform the modification based on the user input, and / or the storage unit within system 200 may be configured to store the modified result as at least part of a new labeled document.

[0066] In some embodiments, the determination unit 210 may be further configured to determine, for each of the one or more identified elements, the field to which each element corresponds. This determination can be performed using the entire kernel of a kernel machine, such as the kernel machine used by the identification unit 230 to identify elements. It will be appreciated that in some embodiments, the determination unit 210 can be configured to perform the identification of the one or more elements and the field determination simultaneously as a single step / process within the kernel machine for each of the one or more identified elements.

[0067] Although not shown in FIG. 2, in some embodiments where there are multiple identified elements, the determination unit 210 may be configured to determine the field of the identified elements by grouping the multiple identified elements into one or more groups. In these embodiments, each group may correspond to a field. The grouping may be based on a word grouping algorithm. More specifically, the determination unit 210 may be configured to group the multiple identified elements as follows. - Generate a linear text sequence of the multiple identified elements in the unlabeled document. - Determine the classification of each element in the generated linear text sequence (e.g., using a kernel machine). - Group all the maximum continuous subsequences of elements sharing the classification in the generated linear text sequence into each group.

[0068] The classification determined for each element may be an argmax classification. The determination of the classification by the determination unit 210 may be made based on multiple possible classifications, and each possible classification corresponds to a field of the labeled document or an indication indicating that the element is not associated with any field of the labeled document.

[0069] Furthermore, in these embodiments, the system 200 may further include an assignment unit configured to assign each row identifier to each element in the group based on the relative order of the elements in the group.

[0070] In some embodiments, one or more fields and one or more elements in a labeled document may be presented in an information structure. In these embodiments, system 200 may further comprise an output unit configured to output the identified one or more elements and their corresponding fields in the same (or a similar) information structure. Alternatively, the output unit may be configured to output only the one or more elements (regardless of whether the fields and / or elements are presented in the same / similar information structure or in different information structures).

[0071] System 200 may further comprise an assignment unit configured to assign to each of the identified one or more elements a new label indicating the field to which each element corresponds.

[0072] In some embodiments, determination unit 210 may be further configured to convert each of the one or more identified elements into a normalized form. This operation may be based on field-independent rules or on one or more field-specific rules.

[0073] In some embodiments, it will be understood that the use of labels in a labeled document is sufficient to indicate which elements are of interest and thus which corresponding elements in a label-free document should be identified and captured. However, in some embodiments, the method may further comprise an acquisition unit configured to define one or more fields included in the labeled document and to obtain a schema that defines whether the elements corresponding to each field of the labeled document are shown in a table before determination unit 210 determines one or more anchors. This schema can provide clearer information about the data to be captured from the label-free document. This schema can be used, for example, such that when determination unit 210 determines an anchor, each of the one or more fields defined by the schema can be used as an anchor point.

[0074] In some embodiments, system 200 may further include a receiving unit configured to receive user input for assigning one or more labels to elements of the labeled document before the determining unit 210 determines one or more anchors. In some implementations, the receiving unit may be configured to receive user input for assigning labels by receiving a user selection of a field within the labeled document and receiving one or more user selections of one or more elements corresponding to the selected field. For example, the user selection of an element may be implemented, for example, via a user interface, by enabling the user to draw a rectangular box around the element on the document. The field corresponding to the label (which can be represented by the rectangular box) can be indicated by a visual indicator. For example, each rectangular box can be the color of the corresponding field of the element. Further, in some embodiments, each label may include metadata (e.g., metadata indicating whether the field is in numerical format, metadata indicating whether the element corresponding to the field is in tabular form, etc.).

[0075] In some embodiments, system 200 may further comprise an acquisition unit configured to acquire a plurality of labeled document candidates. In these embodiments, system 200 may further comprise a receiving unit configured to receive user input for selecting a labeled document from the plurality of labeled document candidates. Alternatively, in these embodiments, determination unit 210 may be further configured to determine a labeled document based on a similarity measurement value between each of the acquired labeled document candidates and the unlabeled document. More specifically, determination unit 210 may be configured to determine a labeled document based on a similarity function and a best match algorithm. Since the similarity function and the best match algorithm are described and explained in detail above with reference to FIG. 1, it is understood that the technical functions of these components are not repeated here and are equally applicable. The operations by the acquisition unit, the receiving unit, and the determination unit described in these embodiments may be performed before determination unit 210 determines one or more anchors.

[0076] FIGS. 3A and 3B are diagrams illustrating an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure. As shown in FIGS. 3A and 3B, a labeled document 310 and an unlabeled document 320 are provided. In this embodiment, the labeled document 310 and the unlabeled document 320 are bills of lading (i.e., semi-structured documents).

[0077] In the labeled document 310, the information is presented in a field element pair format or a table format. For example, "Bill of Lading Number" and "00123456" are presented as a field element pair, "Date of Bill of Lading" and "September 2, 2021" are presented as a field element pair, and "Total" and "4596" are presented as a field element pair. In these examples, "Bill of Lading Number", "Date of Bill of Lading", and "Total" are regarded as fields, while "00123456", "September 2, 2021", and "4596" are regarded as corresponding elements.

[0078] In this example, the remaining information is presented in tabular form, and the elements correspond to one of the fields of "Content", "Quantity", "Unit Price", and "Amount". More specifically, the elements "Canon 5D", "Memory Card", and "Lens" correspond to "Content". These are descriptions of the purchased items. The elements "1", "2", and "1" correspond to "Quantity", and these numbers indicate the quantity of each purchased item. The elements "2499", "149", and "1799" correspond to "Unit Price", and these numbers indicate the unit price of each purchased item. The elements "2499", "298", "1799" correspond to "Amount", and these numbers indicate the total amount (price) of each purchased item.

[0079] As shown in FIG. 3A, all elements are labeled (shown in the drawing by boxes surrounding each element, along with the corresponding reference numbers 311, 312, 313A-C, 314A-C, 315A-C, 316A-C, and 317). This can be achieved by receiving user input indicating which words and / or terms (or which parts of the image pixels) within the document are to be considered as elements.

[0080] The label indicates which field the element corresponds to. In this example, the label "00123456" (i.e., label 311) can indicate that each element corresponds to the "Claim Number" field. Similarly, the label "September 2, 2021" (i.e., label 312) can indicate that each element corresponds to the "Filing Date" field. Labels 313A-C indicate that the elements "Canon 5D", "Memory Card", and "Lens" correspond to the field "Content", labels 314A-C indicate that the elements "1", "2", and "1" correspond to the field "Quantity", labels 315A-C indicate that the elements "2499", "149", and "1799" correspond to the field "Unit Price", and label 317 can indicate that the element "4596" corresponds to the field "Total".

[0081] Next, referring to the unlabeled document 320 in FIG. 3A, it can be seen that the claim follows the same information structure as the labeled document 310. Specifically, the unlabeled document 320 contains the same fields of "claim number", "claim date", "content", "quantity", "unit price", "amount", and "total". By applying the computer-implemented method 100 described above with reference to FIG. 1 or using the system 200 described above with reference to FIG. 2, the elements within the unlabeled document can be identified and output.

[0082] An example of this output is shown in FIG. 3B. Labels 321, 322, 323, 324, 325, 326, 327 corresponding to each identified element are assigned to the unlabeled document 320. In this case, the identified elements are "00123457", "September 3, 2021", "Sony ZV-1", "1", "699", "699", and "699". These identified elements correspond to the labeled elements in the labeled document 310 and their fields. Specifically, the element "00123457" corresponds to the element "00123456" within the field ("claim number"), the element "September 3, 2021" corresponds to the element "September 2, 2021" within the field ("claim date"), the element "Sony ZV-1" corresponds to the elements "Canon 5D", "memory card", and "lens" within the field ("content"), the element "1" corresponds to the elements "1", "2", and "1" (labels 314A~C) within the field ("quantity"), the unit price "699" (label 325) corresponds to the elements "2499", "149", "1799" (labels 315A~C) within the field ("unit price"), the element "699" (label 326) corresponds to the elements "2499", "298", and "1799" (labels 316A~C) within the field ("amount"), and the element "699" (label 326) corresponds to the element "4596" within the field ("total").

[0083] FIG. 4 is a diagram showing an exemplary process for determining an anchor between the labeled document of FIGS. 3A and 3B and the unlabeled document. FIG. 4 shows an upper linear text sequence and a lower linear text sequence. The upper linear text sequence can be generated by extracting all the words in the labeled document 310 and ordering the extracted words in a linear text sequence (e.g., reading order). Similarly, the lower linear text sequence can be generated by extracting all the words in the unlabeled document 320 and ordering the extracted words in a linear text sequence (e.g., reading order).

[0084] In this example, any word in the upper linear text sequence corresponding to a labeled element (e.g., "00123456", "September 2, 2021", "Canon 5D", etc.) is ignored in the process of determining the anchor. Alternatively, in other embodiments or examples, the words corresponding to the labeled elements can be removed from the linear text sequence in the process of determining the anchor.

[0085] Once the linear text sequences corresponding to the labeled document 310 and the unlabeled document 320 are generated, a match can be identified between the two linear text sequences. As indicated by the arrows between the upper and lower linear text sequences, in this example (from left to right), 10 matches of "claim", "number", "claim", "date", "content", "quantity", "unit", "price", "amount", and "total" are identified. Note that in at least this example, each of the single blocks of text can be considered a "word". Thus, "claim" and "number" are considered two (separate) words, "unit cost" is considered as one field in total, but still "unit" and "cost" are considered two words as well.

[0086] Next, each identified match is designated as an anchor, and in this example, ten anchors corresponding to the words "claim", "number", "claim", "date", "content", "quantity", "unit", "price", "amount", and "total" are displayed respectively. Although all the matches identified in this example are exact matches, it will be understood that in other examples and embodiments, the matches may not be 100% exact (e.g., due to OCR errors).

[0087] FIG. 5 is a diagram showing the anchors determined from the exemplary process shown in FIG. 4. To illustrate one aspect of the present disclosure, in FIG. 5, focus is placed only on the anchor indicated by the arched arrow referring to the word "quantity" in both the labeled document 510 (identical to the labeled document 310 in FIGS. 3A and 3B) and the unlabeled document 520 (identical to the unlabeled document 320 in FIGS. 3A and 3B). This anchor is one of the matches identified by the process described above with reference to FIG. 4.

[0088] As described above with reference to FIGS. 1 and 2, in some embodiments, the generation of the entire kernel includes generating a convolutional kernel for each of one or more anchors, and each convolutional kernel spans all pairs of all words and / or terms within and between the labeled document and the unlabeled document. More specifically, each convolutional kernel can measure the similarity of words and / or terms within and between the labeled document and the unlabeled document. The similarity is a function of the distance of each word and / or each term from each point of the anchor and is the similarity of the relative positions with respect to each point of the anchor. Thus, in this example, the convolutional kernel for the "quantity" anchor can measure the similarity of words and / or terms related to the anchor within the labeled document 510, the unlabeled document 520, and between the labeled document 510 and the unlabeled document 520.

[0089] As an example of a method for measuring the similarity of words, in FIG. 5, the similarity between "1" in the labeled document 510 and "1" in the unlabeled document 520 is focused on (indicated by the lower arrow pointing to the two words). This similarity is a function of the distance between "1" and "quantity" in the labeled document 510 (indicated by the short arrow within the document), the distance between "1" and "quantity" in the unlabeled document 520 (indicated by the short arrow within the document), and the similarity of the relative positions of the word "1" with respect to the word "quantity" in each document.

[0090] As described above, as explained with reference to several mathematical formulas, in at least some implementations, words and / or terms are considered similar only if both are close to the anchor and have similar positions with respect to the anchor (by taking the product of the offset kernel and the relative kernel). In this case, due to the proximity of "1" and "quantity" in the labeled document 510, the proximity of "1" and "quantity" in the unlabeled document 520, and the similarity of the relative positions of the word "1" with respect to the word "quantity" in each document (both "1"s are located below the word "quantity"), the similarity measured by the convolution kernel becomes high. On the other hand, the measured similarity regarding the same anchor between the word "4596" in the labeled document 510 and the word "00123457" in the unlabeled document 520 is low because both are far from the word "quantity" in their respective documents and have different relative positions with respect to "quantity".

[0091] Convolution kernels for other anchors can also be generated in a similar manner, and each convolution kernel covers all pairs of all words and / or terms within and between the labeled and unlabeled documents for each anchor. Once all the convolution kernels are generated, they can be summed up to generate the overall kernel for the purpose of identifying elements from the unlabeled document 520.

[0092] FIG. 6 is a diagram illustrating an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure. As shown in FIG. 6, a labeled document 610 and an unlabeled document 620 are provided. In this embodiment, the labeled document and the unlabeled document are invoices (i.e., semi-structured documents).

[0093] In the labeled document 610, some information is presented in a field-element pair format. For example, "Invoice reference number" and "237481709" are presented as a field-element pair, "Total payment" and "£1901.98" are presented as a field-element pair, and "Payment due date" and "16 May 1995" are presented as a field-element pair. In these examples, "Invoice reference number", "Total payment", and "Payment due date" are regarded as fields, and "237481709", "£1901.98", and "16 May 1995" are regarded as the corresponding elements.

[0094] Some other information within the labeled document 610 is presented in a table format, and the elements correspond to one of the fields of "Contents", "Price", "Discount", "VAT", and "Amount". For example, elements such as "Energy bar", "100% juice apple grape", and "Natural mineral water" correspond to "Contents", and these are descriptions of the purchased items.

[0095] Furthermore, in this example, some information is presented as an element (the corresponding field is not explicitly stated). For example, since the element "Bridge Interiors, 56 Bridge Lane, Ilkley LS29 9EU" is an address, it is understood to correspond to the "Address" field. Although this field is not explicitly shown in the labeled document, the label 612 assigned to this element can indicate that this element corresponds to the "Address" field.

[0096] As shown in FIG. 6, only some of the elements of the labeled document 610 are labeled (in the figure, shown by boxes surrounding each element corresponding to reference numerals 611, 612, 613, 614, and 615). This can be achieved by receiving user input indicating which words and / or terms (or which parts of the image pixels) within the document are to be considered elements. In this example, not all elements within the labeled document 610 have a label assigned to them. This is because some elements (without labels) are not of interest, and thus it is not necessary or not required to capture these elements and their corresponding elements in the fields. For example, although "order form" and "855496" are presented as a field element pair, no label is assigned to "855496". As another example, none of the elements corresponding to "price" are labeled in the labeled document 610.

[0097] In this example, the label can indicate the field to which the element corresponds. For example, as described above, the label 612 can indicate that each element corresponds to the "address" field. Similarly, the label for "237481709" (i.e., label 611) indicates that each element corresponds to the "invoice reference number" field, the label for "1901.98 pounds" (i.e., label 613) indicates that each element corresponds to the "total payment" field, the label for "May 16, 1995" (i.e., label 614) indicates that each element corresponds to the "payment due date" field, and the label 615 for the various elements placed under "content" can indicate that these elements correspond to the "content" field.

[0098] Next, referring to the unlabeled document 620 of FIG. 6, it can be seen that the claim follows the same information structure as the labeled document. Specifically, the unlabeled document 620 includes the same fields, such as "claim reference number", "total payment", "payment due date", and "content". By applying the computer-implemented method 100 described above with reference to FIG. 1, or using the system 200 described above with reference to FIG. 2, the elements of the unlabeled document can be identified and output.

[0099] As an example of the manner of anchor determination in the data capture process, FIG. 7A shows text anchors determined based on the labeled document and the unlabeled document of FIG. 6. The labeled document 710 in FIG. 7A corresponds to the labeled document 610 in FIG. 6, and the unlabeled document 720 corresponds to the unlabeled document 620 in FIG. 6. The text anchors in this example are indicated by arrows spanning the two documents, and point to the word "content" 711 in the labeled document 710 and the word "content" 721 in the unlabeled document 721. This anchor can be determined in several steps. (1) Extract all the words from the labeled document 710 and the unlabeled document 720 (for example, using OCR). (2) Order the words included in the labeled document 710 into a linear text sequence to generate a first linear text sequence. (3) Order the words included in the unlabeled document 720 into a linear text sequence to generate a second linear text sequence. (4) Identify the matches between the first linear text sequence and the second linear text sequence (ignoring words corresponding to labeled elements, such as "energy bar"). (5) Designate the matched words as anchors.

[0100] Once this anchor (the "content") is determined, a convolutional kernel for the anchor can be generated. The convolutional kernel spans all pairs of all words and / or terms within and between the labeled document 710 and the unlabeled document 720. The convolutional kernel measures the similarity of words and / or terms within and between the labeled and unlabeled documents. The similarity is a function of the distance from each point of the anchor to each word, and is the similarity of the relative positions with respect to each point of the anchor. For example, in these two documents, the term "energy bar" 712 in the labeled document and the word "sawdust bread mix" 722 will have a high similarity measured (with respect to the anchor of the "content"). This is because both are close to the word "content" in each document, and their relative positions with respect to the word "content" are also similar (i.e., "energy bar" and "sawdust bread mix" are directly below "content" in each document).

[0101] Convolutional kernels for other anchors can also be generated in a similar way, and each convolutional kernel spans all pairs of all words and / or terms within and between the labeled and unlabeled documents for each anchor. Once all the convolutional kernels are generated, they can be summed to generate an overall kernel for the purpose of identifying elements from the unlabeled document 720.

[0102] As another example of the manner of anchor determination in the data capture process, FIG. 7B shows a visual anchor determined based on a labeled document 710 and an unlabeled document 720. The visual anchor in this example is indicated by an arrow spanning the two documents, pointing to the horizontal dividing line 713 of the labeled document 710 and the horizontal dividing line 723 of the unlabeled document 720. This anchor can be determined by identifying visual features that exist in both the labeled and unlabeled documents and designating the identified visual features as the anchor. In this example, the identified visual features are the horizontal dividing lines in the two documents. The identification of visual features can be performed based on an image alignment algorithm, such as one based on SIFT.

[0103] Once this visual anchor is determined, a convolutional kernel for the anchor can be generated. Here, the convolutional kernel spans all pairs of all words and / or terms within and between the labeled document 710 and the unlabeled document 720. The convolutional kernel measures the similarity of words and / or terms within and between the labeled and unlabeled documents. The similarity is a function of the distance from each point of the anchor to each word and the similarity of the relative positions with respect to the points of each anchor.

[0104] For example, the term "100% Juice Apple Grape" in the labeled document 710 and the term "Protein Plus Bar Cookie & Cream Flavor" in the unlabeled document 720 will likely have a high measured similarity (with respect to the visual anchor) due to the fact that these terms are relatively close to the horizontal dividing line in their respective documents and the relative positions with respect to the horizontal dividing line are similar. In contrast, the term "Pepper Mayonnaise" in the labeled document 710 and the term "Onion Ring" in the unlabeled document will likely have a low measured similarity (with respect to the visual anchor) because at least the term "Pepper Mayonnaise" is far from the horizontal dividing line and the relative positions of these terms ("Pepper Mayonnaise" vs. "Onion Ring") are significantly different.

[0105] FIG. 8 shows a labeled document 810, a first unlabeled document 820, and a second unlabeled document 830. The labeled document 810 corresponds to the labeled document 610 shown in FIG. 6, and the first unlabeled document 820 corresponds to the unlabeled document 620 shown in FIG. 6. The second unlabeled document 630 is also a claim adopting an information structure similar to that of the other two documents. In this example, anchors are determined for the labeled document 810 and the unlabeled document 820, which are indicated by an arrow pointing from a box 816 in the labeled document 810 to a box 826 in the first unlabeled document 820, and anchor symbols in these two documents. This anchor can be used as a criterion when determining the anchor of a new unlabeled document (for example, belonging to a batch of unlabeled documents identical to 820). In this case, the new unlabeled document is the second unlabeled document 830. As shown in FIG. 8, a new anchor indicated by an arrow pointing from a box 816 in the labeled document 810 to a box 836 in the unlabeled document 830, and anchor symbols in the two documents can be determined in this way.

[0106] FIG. 9 is a diagram showing a topology for performing data capture according to an embodiment of the present disclosure. The topology 900 may include a client computing device 905 and a server 920 configured to be communicably connected via a network 910.

[0107] The network 910 may be a wired or wireless network such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a near field communication (NFC) network, Bluetooth (registered trademark), infrared, radio frequency, a cellular network, or other types of networks. It will be understood that the network 910 may be a combination of multiple different types of wired or wireless networks.

[0108] Each client computing device 905 may be a smartphone, a tablet computer, a laptop computer, a computer, a personal data assistant, or any other type of mobile device having a hardware processor configured to process instructions and connected to one or more portions of network 910. Each client computing device 905 may include a graphical user interface configured to enable a user to interact with the processor of the client computing device 905.

[0109] Server 920 may include a physical computing device located in a particular location or may be deployed in a cloud computing network environment. In the present disclosure, "cloud computing" may be defined as a model that enables ubiquitous and convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that are rapidly provisioned via virtualization, released with minimal management effort or interaction with a service provider, and then scaled as appropriate. The cloud model can be composed of various characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service), service models (e.g., software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS)), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.). Server 920 may include any combination of one or more computer-usable or computer-readable media. In some embodiments, server 920 may be configured to perform data capture. For example, server 920 may perform at least a portion of method 100 described above with reference to FIG. 1.

[0110] It will be understood that this description is illustrative only. Modifications and changes can be made to the described embodiments without departing from the scope of the disclosure defined by the claims.

[0111] This technology has been described in detail based on the currently most practical and preferred implementation. However, such detailed descriptions are for illustrative purposes only. This technology is not limited to the disclosed implementation. On the contrary, it is intended to cover modifications and equivalent configurations included in the appended claims. For example, it is understood that this technology assumes that, to the extent possible, one or more functions of any implementation can be combined with one or more functions of any other implementation.

[0112] Throughout this specification, the expressions "one embodiment", "an embodiment", "an example", or "an illustration" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present disclosure. Therefore, the expressions "in one embodiment", "in an embodiment", "an example", or "an illustration" in various places in this specification do not necessarily all refer to the same embodiment or example. Furthermore, a particular function, structure, or characteristic can be combined in any suitable combination and / or partial combination in one or more embodiments or examples. Also, the accompanying drawings are generally for the purpose of explanation for those skilled in the art, and it should be understood that the drawings are not necessarily drawn to scale.

[0113] Embodiments according to the present disclosure can be embodied as an apparatus, a method, or a computer program product. For example, in some embodiments, when a program is executed by a processor, a computer program product may be provided that includes instructions for causing the processor to perform the method described with respect to FIG. 1. As another example, in some embodiments, a computer-readable storage medium may be provided that includes instructions for causing a computer to perform the steps of the method described with respect to FIG. 1 when executed by the computer. Accordingly, the present embodiments can take any form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which can generally be referred to as a "module" or a "system". Further, embodiments of the present disclosure can take the form of a computer program product embodied in any tangible expression medium in which computer-usable program code is embodied in any physical medium.

[0114] Although described in relation to an exemplary computing system environment, embodiments of the present disclosure are operable in a number of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use in aspects of the present disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, handheld or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set-top boxes, programmable household appliances, cellular phones, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0115] Any combination of one or more computer-usable or computer-readable media can be utilized. For example, computer-readable media can include one or more of a portable computer diskette, hard disk, random access memory (RAM) device, read-only memory (ROM) device, erasable programmable read-only memory (EPROM or Flash memory) device, portable compact disc read-only memory (CDROM), optical storage device, and magnetic storage device. The computer program code for performing the operations of the embodiments of the present disclosure can be written in any combination of one or more programming languages.

[0116] Flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the block diagrams and / or flowcharts may represent a module, segment, or portion of code that comprises one or more executable instructions for implementing the specified logical function. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions. These computer program instructions can also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner. The instructions stored in the computer-readable medium produce an article of manufacture that includes instruction means for implementing the functions / acts specified in the flowchart and / or block diagram.

[0117] The following is a list of embodiments of the present disclosure.

[0118] 1. A computer-implemented method for data capture in an unlabeled document based on a labeled document including one or more elements each labeled for a field, comprising: determining one or more anchors having a pair of matching points between the labeled document and the unlabeled document; generating an entire kernel using the determined one or more anchors; identifying one or more elements to be captured in the unlabeled document using the entire kernel in a kernel machine; including; wherein the one or more elements to be captured correspond in a field to the one or more labeled elements included in the labeled document.

[0119] 2. The method according to embodiment 1, wherein generating the entire kernel comprises: generating a convolutional kernel for each of the one or more anchors; generating the entire kernel by adding the one or more convolutional kernels.

[0120] 3. The method according to embodiment 2, wherein the entire kernel is a weighted sum of the one or more convolutional kernels.

[0121] 4. The method according to embodiment 2 or 3, wherein each convolutional kernel spans all pairs of words within and between the labeled document and the unlabeled document.

[0122] 5. The method according to any one of embodiments 2 to 4, wherein each convolutional kernel measures a similarity of words within and between the labeled document and the unlabeled document, the similarity being a function of a distance from each point of the anchor to each word, and a similarity of relative positions with respect to each point of the anchor.

[0123] 6. The method according to any one of embodiments 2 to 5, comprising: The labeled document is defined as d = d1, and the unlabeled document is defined as d = d2, where d is a document, and each document is a sequence of words w d = w d,1 ,…, w d,nd and n d is the number of words in each document, where each convolutional kernel is defined as k(w d,i , w d’,j ; a) = k off (z)k off (z’)k rel (z, z’), and each pair of words is represented as (w d,i , w d’,j ), a is the respective anchor, defined as a = (a1, a2), where a1 is the position of the corresponding point of the anchor in d1 and a2 is the position of the corresponding point of the anchor in d2, where

Number

Number

Number

Number

Number

[0124] 7. A method according to Embodiment 6, wherein d = d’ and / or d ≠ d’.

[0125] 8. A method according to any one of the preceding embodiments, after identifying the one or more elements within the unlabeled document, Receiving user input to confirm or reject the specific result, and Receiving user input to correct the specific result, A method further comprising at least one of the above.

[0126] 9. The method according to any one of the preceding embodiments, wherein the labeled document further includes the one or more fields indicated by the one or more labels.

[0127] 10. The method according to any one of the preceding embodiments, further comprising, for each of the one or more identified elements, determining the corresponding field for each element, the determination using the entire kernel in the kernel machine.

[0128] 11. The method according to embodiment 10, wherein a plurality of elements are identified in the unlabeled document, and determining the corresponding field for each identified element includes grouping the plurality of identified elements into one or more groups, each group corresponding to a field.

[0129] 12. The method according to embodiment 11, wherein grouping the plurality of identified elements is based on a word grouping algorithm.

[0130] 13. The method according to embodiment 11 or embodiment 12, wherein grouping the plurality of identified elements comprises generating a linear text sequence of the plurality of identified elements within the unlabeled document, and determining the argmax classification of each element in the generated linear text sequence, Grouping all of the maximum contiguous subsequences of elements sharing an argmax classification in the generated linear text sequence into each group, and A method comprising.

[0131] 14. The method according to any one of embodiments 11 to 13, Further comprising assigning respective line identifiers to each of the elements within the group based on the relative order of the elements within the group.

[0132] 15. The method according to any one of embodiments 9 to 14, The one or more fields and one or more elements in the labeled document are presented in an information structure, and further comprising outputting the identified one or more elements and their corresponding fields in the same information structure.

[0133] 16. The method according to any one of embodiments 9 to 15, Before determining one or more anchors, defining the one or more fields included in the labeled document, and obtaining a schema that defines whether elements corresponding to each field in the labeled document are to be displayed in a table.

[0134] 17. The method according to any one of the preceding embodiments, Determining an anchor comprises Extracting all words in the labeled document and the unlabeled document, Ordering the words included in the labeled document into a first linear text sequence to generate a first linear text sequence, and deleting all words corresponding to labeled elements, Ordering the words included in the unlabeled document into a second linear text sequence to generate a second linear text sequence, Identifying a match between the first linear text sequence and the second linear text sequence, and Specifying the matched word as an anchor, A method including this.

[0135] 18. The method according to embodiment 17, wherein the match between the first linear text sequence and the second linear text sequence exceeds a predetermined similarity threshold and a predetermined alignment threshold.

[0136] 19. The method according to any one of embodiments 17 to 18, wherein the extraction of the words in the labeled document and the unlabeled document is based on optical character recognition (OCR).

[0137] 20. The method according to any one of embodiments 17 to 19, using a reading order algorithm to order the words included in the labeled document and the unlabeled document.

[0138] 21. The method according to any one of embodiments 17 to 20, wherein each of the one or more anchors is defined as a correspondence between the center of each matched word in the labeled document and the center of each matched word in the unlabeled document.

[0139] 22. The method according to any one of embodiments 17 to 21, wherein identifying the match between the first linear text sequence and the second linear text sequence is based on an alignment algorithm.

[0140] 23. The method according to embodiment 22, wherein the alignment algorithm is similar to or equivalent to the Needleman-Wunsch algorithm.

[0141] 24. The method according to any one of the preceding embodiments, wherein determining an anchor comprises identifying visual features present in both the labeled document and the unlabeled document, and designating the identified visual features as the anchor, the method comprising.

[0142] 25. The method according to embodiment 24, wherein identifying the visual features is based on an image alignment algorithm, the method.

[0143] 26. The method according to embodiment 25, wherein the image alignment algorithm is based on at least one of a keypoint detector, a local invariant descriptor, and a keypoint matching process, the method.

[0144] 27. The method according to any one of the preceding embodiments, wherein determining an anchor comprises placing a plurality of points around the boundary of the unlabeled document and placing the same number of points around the boundary of the labeled document, and identifying a match between the points of the labeled document and the points of the unlabeled document, and designating the matched points as the anchor, the method comprising.

[0145] 28. The method according to any one of the preceding embodiments, wherein the method further comprises receiving user input for assigning one or more labels to the elements of the labeled document before determining one or more anchors.

[0146] 29. The method according to any one of the preceding embodiments, wherein before determining one or more anchors, the method further comprises obtaining a plurality of labeled document candidates, Determining the labeled document from the obtained labeled document candidates based on the similarity measurement values between each of the plurality of labeled document candidates and the unlabeled document; A method further comprising.

[0147] 30. The method according to any one of the preceding embodiments, wherein the kernel machine includes at least one of a support vector machine, SVM, regularized kernel discriminant analysis, RKDA, Gaussian process, GP.

[0148] 31. The method according to any one of the preceding embodiments, wherein the labeled document is in the form of a PDF document or an image.

[0149] 32. A system for performing data capture on an unlabeled document based on a labeled document including one or more elements each labeled for a field, a determination unit configured to determine one or more anchors having a pair of corresponding points that match between the labeled document and the unlabeled document; a generation unit configured to generate an entire kernel using the determined one or more anchors; a specific unit configured to identify one or more elements to be captured in the unlabeled document including information to be captured using the entire kernel in the kernel machine, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements included in the labeled document; A system comprising.

[0150] 33. A computer-readable storage medium, which, when executed by a computer, includes instructions for causing the computer to execute the steps of the method according to any one of Embodiments 1 to 31.

Claims

1. A computer implementation method (100) for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the method Step (S110): A step of determining one or more anchors, wherein each anchor has a pair of location points of matching words or visual features in the labeled document and the unlabeled document. Step (S120) to generate an overall kernel using one or more anchors determined above, Step (S130): Identifying one or more elements to be captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements contained in the labeled document. A method of having.

2. The step of generating the overall kernel (S120) is, A step of generating a convolution kernel for each of the one or more anchors mentioned above. A step of generating the overall kernel by adding one or more of the aforementioned convolution kernels, The method according to claim 1, comprising having

3. The method according to claim 2, wherein each convolution kernel measures the similarity of words within and between the labeled and unlabeled documents, the similarity being a function of the distance from each point of the anchor to each word and the similarity of the relative position of the anchor to each point.

4. The method according to claim 1, wherein the labeled document further comprises the one or more fields indicated by the one or more labels.

5. The one or more fields and one or more elements within the labeled document are represented by an information structure. The method further comprises the step of outputting the identified one or more elements and their corresponding fields in the same information structure. The method according to claim 4.

6. The method further includes a step of obtaining a schema, which defines one or more fields contained in the labeled document and defines whether elements corresponding to each field in the labeled document are presented in a table, prior to the step of determining the one or more anchors (S110). The method according to claim 4.

7. The method further comprises the step of determining, for each of the one or more identified elements, the field to which the element corresponds, wherein the determination uses the kernel machine. The method according to claim 1.

8. The step of identifying multiple elements in the unlabeled document and determining the field to which each identified element corresponds includes the step of grouping the multiple identified elements into one or more groups, where each group corresponds to a field. The method according to claim 7.

9. The step of grouping the aforementioned multiple identified elements is: A step of generating a linear text sequence of the plurality of identified elements in the unlabeled document, A step of determining the argmax classification of each element in the generated linear text sequence, The step of grouping all of the longest continuous subsequences of elements that share the argmax classification in the generated linear text sequence into groups, Having, The method according to claim 8.

10. The one or more fields and one or more elements within the labeled document are represented by an information structure. The method further comprises the step of outputting the identified one or more elements and their corresponding fields in the same information structure. The method according to claim 7.

11. The method further includes a step of obtaining a schema, which defines one or more fields contained in the labeled document and defines whether elements corresponding to each field in the labeled document are presented in a table, prior to the step of determining the one or more anchors (S110). The method according to claim 7.

12. The step of determining the anchor (S110) is, A step of extracting all words from the labeled document and the unlabeled document, A step of generating a first linear text sequence by rearranging the words contained in the labeled document into a linear text sequence and deleting all words corresponding to the labeled elements. A step of generating a second linear text sequence by rearranging the words contained in the aforementioned unlabeled document into a linear text sequence, Steps to identify a match between the first linear text sequence and the second linear text sequence, The step of designating the matching word as the anchor, Having, The method according to claim 1.

13. The step of determining the anchor (S110) is, A step of identifying visual features present in both the labeled document and the unlabeled document, The step of designating the identified visual feature as the anchor, Having, The method according to claim 1.

14. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 13.

15. The step of determining the anchor (S110) is, The steps include: placing multiple points around the boundary of the unlabeled document and placing the same number of points around the boundary of the labeled document; A step of identifying a match between a point on the labeled document and a point on the unlabeled document. The step of designating the aforementioned matching point as the anchor, Having, The method according to claim 1.

16. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 15.

17. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 1.

18. The above method further includes, before the step of determining one or more anchors (S110), Steps to retrieve multiple labeled document candidates, From the acquired labeled document candidates, a step of determining the labeled document based on the similarity measurement value between each of the plurality of labeled document candidates and the unlabeled document, Having, The method according to claim 13.

19. A system (200) that performs data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the system A decision unit (210) configured to determine one or more anchors, wherein the anchors have a pair of location points of matching words or visual features in the labeled document and the unlabeled document. A generation unit (220) configured to generate an overall kernel using one or more of the determined anchors, An identification unit (230) that identifies one or more elements captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements captured correspond in a field to the one or more labeled elements contained in the labeled document, A system (200) equipped with the following features.

20. A non-temporary computer-readable storage medium, which, when executed by a computer, includes instructions causing the computer to perform a method (100) of performing data capture in an unlabeled document based on a labeled document, wherein the labeled document includes one or more elements, each labeled for a field, and the method Step (S110): A step of determining one or more anchors, wherein each anchor has a pair of location points of matching words or visual features in the labeled document and the unlabeled document. Step (S120) to generate an overall kernel using one or more anchors determined above, Step (S130): Identifying one or more elements to be captured in the unlabeled document using a kernel machine trained with matrix entries of at least a subset of the overall kernel, wherein the one or more elements to be captured correspond in a field to the one or more labeled elements contained in the labeled document. A computer-readable storage medium having [a certain characteristic].

Images