Method and system for performing data capture

Abstract

A method (100) is provided for performing data capture in an untagged document based on a tagged document, where the tagged document contains one or more elements, each element being tagged with respect to a field. The method comprises: determining (S110) one or more anchors, where the anchors comprise point pairs that match between the tagged document and the untagged document; generating (S120) a total kernel using the determined one or more anchors; and identifying (S130), in a core machine, one or more elements to be captured in the untagged document using the total core, where the one or more elements to be captured correspond in field to the one or more tagged elements contained in the tagged document.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for performing data capture. More specifically, the present disclosure relates to systems and computer-implemented methods for performing data capture in one or more unlabeled documents based on one or more labeled documents. Background Art

[0002] Documents of various types are widely used to collect and record information for many purposes and fields, such as medicine, business, education, and government. Due to the widespread use of computers and communication networks today, these documents are now often created and provided electronically, enabling them to be generated and shared digitally. These documents often contain data in a structured or semi-structured format. Structured documents may have embedded codes that enable information to be arranged in a specified format. Semi-structured documents may be documents such as bills, bank statements, utility bills, passports, etc., which do not follow the strict layout followed by structured documents, and they are not bound to specified data fields like structured documents.

[0003] In order to utilize information from these electronic documents to perform desired operations (e.g., business-related operations), the information needs to be properly extracted. However, at least for semi-structured electronic documents, it may be difficult to reliably parse and extract the required information because these documents typically do not have a well-defined layout. Summary of the invention

[0004] Data capture is the task of extracting relevant information from a document. The One-Time Data Capture (OSDC) technology described in this disclosure is designed to extract information from structured and semi-structured documents. The data to be captured may include the values ​​of key-value pairs in the document, such as bill numbers, total amounts, and / or values ​​in a table in the document. In the context of this disclosure, key-value pairs may be referred to as "field-element pairs," where keys may be referred to as "fields" and values ​​may be referred to as "elements." In its simplest form, the OSDC technology may work on a pair of documents: one is labeled and one is unlabeled. In this case, the task is to extract the same type of information from an unlabeled document as the labeled information in a labeled document.

[0005] Embodiments described herein are directed to methods and systems for performing data capture in unlabeled documents based on labeled documents.

[0006] According to one aspect of the present disclosure, there is provided a computer-implemented method for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document contains one or more elements, each element being labeled with respect to a field. The method comprises: determining one or more anchors, wherein an anchor comprises a point pair matched between the labeled document and the unlabeled document; generating a total kernel using the determined one or more anchors; and using the total kernel in a kernel machine, identifying one or more elements to be captured in the unlabeled document, wherein the one or more elements to be captured correspond in terms of fields to the one or more labeled elements contained in the labeled document.

[0007] According to another aspect of the present disclosure, a system for performing data capture in an unlabeled document based on a labeled document is provided, wherein the labeled document contains one or more elements, each of which is labeled with respect to a field. The system includes: a determination unit configured to determine one or more anchors, wherein the anchors include corresponding point pairs matched between the labeled document and the unlabeled document; a generation unit configured to generate a total core using the determined one or more anchors; and an identification unit configured to use the total core in a core machine to identify one or more elements to be captured in the unlabeled document that contain information to be captured, wherein the one or more elements to be captured correspond to the one or more labeled elements contained in the labeled document in terms of fields.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium comprising instructions is provided. When the instructions are executed by a computer, the computer is caused to perform the steps of the method as described in the present disclosure.

[0009] Optional features are set out in the attached dependent claims.

[0010] These and other aspects of the present disclosure will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. The following description, while indicating various embodiments of the present disclosure and many of its specific details, is given by way of illustration and not limitation. Many substitutions, modifications, additions, or rearrangements may be made within the scope of the present disclosure, and the present disclosure includes all such substitutions, modifications, additions, or rearrangements. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 is a flow chart of a computer-implemented method for performing data capture according to an embodiment of the present disclosure; Figure 2 is a block diagram of a system for performing data capture according to an embodiment of the present disclosure; Figure 3A and Figure 3B is a diagram showing an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure; Figure 4 It shows confirmation Figure 3A and 3B A diagram of an exemplary process of anchoring between labeled documents and unlabeled documents; Figure 5 It shows that according to Figure 4 A diagram of anchors determined by the exemplary process shown; Figure 6 is a diagram showing an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure; Fig. 7A It is shown based on Figure 6 A graph of anchors determined by labeled and unlabeled documents; Figure 7B It is shown based on Figure 6 A graph of another anchor determined by labeled documents and unlabeled documents; Figure 8 It shows that according to Figure 6 A graph of anchors determined by the document and another unlabeled document; and Fig. 9 is a diagram illustrating a topology for performing data capture according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, many specific details are set forth to provide a thorough understanding of the embodiments. However, it is apparent to one of ordinary skill in the art that the specific details are not required to practice the embodiments. In other examples, well-known materials or methods are not described in detail to avoid obfuscating the present disclosure.

[0013] Figure 1 1 is a flowchart of a computer-implemented method 100 for performing data capture according to an embodiment of the present disclosure. In more detail, the illustrated method 100 is used to perform data capture in an unlabeled document based on a labeled document. The labeled document contains one or more elements, each element being labeled with respect to a field. Each element may be represented by a word, a string of words, or a number, or a combination thereof. In some embodiments, the labeled document may also include one or more fields indicated by one or more tags. The labeled document and / or the unlabeled document may be a PDF document or an image. In some implementations, the method may be performed by a client computing device, a network, a server, or a combination thereof, such as Fig. 9 , to execute with reference to the client computer device 905, network 910, and server 920 shown in FIG.

[0014] Method 100 begins at step S110, where one or more anchors are determined. Anchors include point pairs that match between labeled documents and unlabeled documents. In some implementations, a new set of one or more anchors is determined for each new document pair (where the pair includes an unlabeled document and a labeled document).

[0015] In some embodiments, determining the anchor at step S110 may include receiving user input selecting a point pair in an unlabeled document and a labeled document. Alternatively, in some embodiments, determining the anchor at step S110 may include the following steps: - extracting (e.g., using OCR) all words in the labeled document and the unlabeled document; - generating a first linear text sequence by ordering the words contained in the marked document into a linear text sequence (and optionally removing any words corresponding to already marked elements); - generating a second linear text sequence by sorting the words contained in the unlabeled document into a linear text sequence; - identifying matches between the first and second linear text sequences (if words corresponding to marked elements have not been removed at this stage, these words are ignored in the match identification step); and - Designate the matched word as the anchor.

[0016] In this case, the match involves the inherent similarity of the two words, as well as their alignment (i.e., the similarity in the linear order of the respective words in the respective sequences). In these embodiments, the match between the first and second linear text sequences can be a match that exceeds a predetermined threshold in both inherent similarity (e.g., visual features) and alignment (i.e., how similar they are in terms of their respective linear order in the respective text sequences). In other words, a match can be two words that look the same or similar and have the same or similar linear order in the respective text sequences. As an example of visual similarity, the words "bam" and "barn" can be determined to exceed a predetermined visual similarity threshold due to their visual similarity (e.g., where the threshold takes into account optical character recognition (OCR) errors such as "m" and "rn"). As an example of alignment, the 8th word in the first text sequence will be more aligned with the 8th word in the second text sequence than with the 7th word in the sequence text sequence. In some embodiments, the identification of a match (more specifically, the determination of whether two words exceed an alignment threshold) can be based on an alignment algorithm, such as an algorithm similar to or equivalent to the Needleman-Wunsch algorithm.

[0017] Alternatively or additionally, in these embodiments, ordering of words contained in tagged and untagged documents may be performed using a reading order algorithm (ie, any algorithm that attempts to order words as a human would read them).

[0018] Alternatively or additionally, in these embodiments, each of the one or more anchors may be defined as a correspondence between a center of a corresponding matching word in a labeled document and a center of a corresponding matching word in an unlabeled document.

[0019] In some embodiments, determining the anchor at step S110 may include identifying visual features present in both the labeled document and the unlabeled document (e.g., using an image alignment algorithm), and designating the identified visual features as anchors. The image alignment algorithm may be based on at least one of the following algorithms: a key point detector (e.g., Difference of Gaussians, Harris, Good Features to Track (GFTT), a local invariant descriptor (e.g., Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB)), etc.) and a key point matching process (e.g., Random Sampling Consensus (RANSAC)).

[0020] In some embodiments, determining the anchor at step S110 may include placing a plurality of points around a boundary of the unlabeled document and placing the same number of points around a boundary of the labeled document, identifying matches between points in the labeled document and points in the unlabeled document, and designating the matched points as anchors.

[0021] Although the one or more anchors are determined for a pair of documents (i.e., a labeled and unlabeled document) in the present embodiment, it should be understood that the above process for determining anchors can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). For example, an anchor may include multiple points that match (i.e., exceed a visual similarity threshold and an alignment threshold), each of the multiple points being in a different document.

[0022] Furthermore, although in the present embodiment it is described that an anchor may include a matching single word (which may or may not consist of or include numbers), in other embodiments, an anchor may include a matching phrase, in addition to or as a replacement for a word, e.g., the word "total amount" is processed as a single phrase rather than two separate words, and thus the corresponding anchor corresponds to the phrase "total amount" that matches in the document pair.

[0023] return Figure 1, at step S120, a total kernel is generated using the determined one or more anchors. In some embodiments, generating the total kernel at step S120 may include generating a convolution kernel for each of the one or more anchors determined at step S110, and generating the total kernel by summing the one or more convolution kernels. The total kernel may be a weighted sum of the one or more convolution kernels.

[0024] Each convolution kernel may involve all word pairs within and between labeled and unlabeled documents. In other embodiments where there are more than two documents, data capture is performed in the more than two documents, and each convolution kernel may involve all word pairs within two or more documents and all word pairs across all two or more documents. Each convolution kernel may measure the similarity of words within and between labeled and unlabeled documents (or across all documents in which more than two documents are involved), the similarity being a function of the similarity of the distance of the corresponding words from the corresponding point of the anchor and the relative position of the corresponding point relative to the anchor.

[0025] To explain the convolution kernel in more detail, we define a document d as a sequence of words, where n d is the number of words in the document.

[0026] If the document is tagged, you can also provide the tags for each word, where a label can belong to any of K categories (each category corresponds to a field): l i ∈{1,…,K}, 1≤i≤n d

[0027] Furthermore, each word can have a position on the page of the document. The position w of the i-th word in the d-th document is d，i Designated as x d，i .

[0028] Spanning two documents 1 and d 2 The anchor a can be represented by a position pair, one position in each document, a = (a 1， a 2 ), where a 1 is in 1 The position in the 2 is in 2 The position in the marked document can be defined as d = d 1 , and the unlabeled document can be defined as d = d 2 .

[0029] For word pairs (w d，i , w d′，j) can be defined relative to anchor a as other radial basis function (RBF) kernel k off and k rel The key input to the kernel is the relative position of the word to the anchor, defined as: z=w d，i -a d z′=w d′，j -a d′ The kernel can be defined as: k(w d，i , w d′，j ; a) = k off (z)k off (z′)k rel (z,z′) k off (z) = k RBF (||zA off || 2 ) k rel (z, z′) = k RBF (||(zz′)A rel || 2 ) In the above equation, r is the distance (ie, the L2-norm of the vector ||(x-x')||).

[0030] Since the convolution kernel can act on any word pair, regardless of whether they are in the same document, note that it is possible for d = d′ or d ≠ d′.

[0031] Linear transformation (matrix) A off and A rel is used as the inverse length scale. In one implementation, these parameters are parameterized diagonally, where σ is the length scale: This allows the kernel to have a different influence in each direction in the document. For example, the value of σ can be set so that the influence on width is more flexible than the influence on height in the document.

[0032] The reason for the above kernel equation is the offset kernel koff The effect of the anchor is local. krel Quantify the similarity between the spatial relationship between the anchor and each word. That is, when individual words are in the same position relative to the anchor, they are similar. The product of the kernel can be taken so that individual words are considered similar only when they are close to the anchor and their positions relative to the anchor are similar.

[0033] Although it is described above that each convolution kernel may involve word pairs, in alternative embodiments, the convolution kernels may involve phrase pairs (e.g., each phrase includes multiple words) instead of, or in addition to, single word pairs.

[0034] As described above, in some embodiments, the process for determining anchors can be applied to more than two documents (e.g., more than one labeled document and one unlabeled document), where each anchor can contain multiple points that match, each of which exists in a different document. In these embodiments, an anchor can be defined as a=(a 1 , a 2 , ..., a N ), where N-1 is the number of labeled documents that have been selected for data capture (thus N is the total number of documents). In addition, in these embodiments, each convolution kernel may involve all word pairs within and across all documents, and the total kernel is the sum of all convolution kernels. This larger total kernel (when compared to the case where only two documents are used) can provide more information to the kernel machine at step S130 for identifying one or more elements to be captured. Each convolution kernel defined with respect to a set of documents can be represented as a symmetric positive semi-definite matrix having a dimension corresponding to the number of words in all documents in the corresponding set of documents. Therefore, in this case, each matrix entry in the matrix corresponds to a corresponding similarity between specific word pairs in the words in each document in the set of documents. Similarly, since the total kernel is generated by summing the convolution kernels, the total kernel can also be represented as a symmetric positive semi-definite matrix, where each matrix entry in the matrix corresponds to a corresponding similarity between specific word pairs in the words in each document in the set of documents.

[0035] return Figure 1, at step S130, the total kernel is used in the kernel machine to identify one or more elements to be captured in the unlabeled document. The one or more elements to be captured correspond to the one or more labeled elements contained in the labeled document in terms of fields. The kernel machine may include at least one of the following: a support vector machine (SVM), a regularized kernel discriminant analysis (RKDA), and a Gaussian process (GP). The kernel machine may be trained based on at least one of the one or more elements and one or more fields indicated by the corresponding tags in the labeled document. In some cases, the kernel machine may be trained based on all of the one or more elements and one or more fields indicated by the corresponding tags in the labeled document so as to maximize the number of samples used for training. In some embodiments, the kernel machine may be trained using a subset of matrix entries for the total kernel, wherein each matrix entry in the subset of matrix entries for the total kernel corresponds to a word pair within the labeled document. The identified elements may be elements corresponding to words having a high (predicted) similarity / confidence score determined by the kernel machine. In these embodiments, the method may further include outputting a similarity / confidence score determined by the kernel machine.

[0036] although Figure 1 , but in some embodiments, method 100 may further include, after identifying the one or more elements at step S130, receiving at least one of user input approving or rejecting the identified result and receiving user input to modify the identified result. In these embodiments, if user input approving the result is received, method 100 may further include outputting the result (e.g., via a display), or performing further processing of the result (e.g., field determination and / or grouping in the manner described below). If user input to modify the identified result is received, method 100 may further include performing the modification based on the user input and / or storing the modified result as at least a portion of a new marked document.

[0037] In some embodiments, the method may further include determining, for each of the identified one or more elements, a field corresponding to the corresponding element. This determination operation may also be performed in the core machine using the total core. It will be appreciated that in some embodiments, the operation of identifying one or more elements at step S130 and the operation of determining the field for each of the identified one or more elements may be performed simultaneously as a single step / process in the core machine.

[0038] In some embodiments where there are multiple identified elements, performing field determination for the identified elements may include grouping the multiple identified elements into one or more groups, where each group corresponds to a field. Such grouping may be based on a word grouping algorithm. More specifically, grouping the multiple identified elements may include: - generating a linear text sequence of the plurality of identified elements (e.g., by determining the reading order of the plurality of identified elements in an untagged document using a reading order algorithm); - determining (e.g., using a kernel machine) a classification for each element in the generated linear text sequence; and -Group each maximal contiguous subsequence of elements sharing a classification in the generated linear text sequence into corresponding groups.

[0039] The classification determined for each element may be an argmax classification.Determining the classification may be based on a plurality of possible classifications, where each possible classification corresponds to a field in a labeled document or corresponds to an indication that the element is not associated with any field in a labeled document.

[0040] In embodiments where the plurality of identified elements are grouped (e.g., using a grouping algorithm as described above), the method may further include assigning a corresponding row identifier to each element in the group based on the relative order of the elements in the group. This allows elements in a tabular format to be output as such (i.e., elements with the same row identifier are placed in the same row).

[0041] In some embodiments, the one or more fields and one or more elements in the marked document may be presented in an information structure. For example, in a bill, the one or more fields may be presented in the header of a table, and the one or more elements may be presented under the header of the table, with each column corresponding to a field and each row corresponding to a purchased item. In these embodiments, method 100 may also include outputting the identified one or more elements and their corresponding fields in the same (or similar) information structure. Alternatively, method 100 may also include outputting only the one or more elements (regardless of whether the fields and / or elements are presented in the same / similar or different information structures).

[0042] The method may also include assigning a new tag to each of the identified one or more elements to indicate the field to which the respective element corresponds.

[0043] In some embodiments, the method may further include placing each of the one or more identified elements into a normalized form. For example, any element corresponding to date information may be returned in the format of DD / MM / YYYY. This operation may be based on field-agnostic rules (i.e., one rule for all fields), such as a rule for removing any leading or trailing punctuation in the corresponding text string of the identified element. Alternatively, the operation may be based on one or more field-specific rules, wherein the field-specific rules may be based on metadata of tags in the tagged document.

[0044] It should be understood that in some embodiments, the use of tags in the marked document is sufficient to indicate which element(s) are the elements of interest, and therefore also indicate which corresponding element(s) in the unmarked document should be identified and captured. However, in some embodiments, the method may also include, before determining one or more anchors at step S110, obtaining a schema that defines whether the one or more fields contained in the marked document and the elements corresponding to each field in the marked document are presented in a table. The schema can provide more explicit information about what data should be captured from the unmarked document. This schema can be used to determine an anchor, for example, at S110 of method 100, so that each of the one or more fields defined by the schema can be used as a point for an anchor.

[0045] although Figure 1 , but in some embodiments, method 100 may further include receiving user input that assigns one or more tags to elements of the tagged document before determining one or more anchors at step S110. In some embodiments, receiving user input for assigning tags may include: receiving a user selection of a field in the tagged document, and receiving one or more user selections of one or more elements corresponding to the selected field. As an example, the user selection of an element may be implemented by allowing a user to draw a rectangular box around an element on the document, for example, via a user interface such as a touch screen. The field corresponding to the tag (which may be represented by a rectangular box) may be indicated by means of a visual indicator. For example, each rectangular box may have a color indicating the corresponding field of the element. In this case, the rectangular boxes of elements corresponding to the same first field will have the same first color, and the rectangular boxes of elements corresponding to the same second field will have the same second color, and so on. In addition, in some embodiments, each tag may contain metadata. For example, this may be metadata indicating whether the field is in the form of a number and / or metadata indicating whether the element corresponding to the field is presented in a tabular format, etc.

[0046] In some embodiments, method 100 may further include obtaining a plurality of candidate labeled documents before determining one or more anchors, and receiving user input selecting a labeled document from the plurality of candidate labeled documents. Alternatively, method 100 may further include obtaining a plurality of candidate labeled documents before determining one or more anchors, and determining a labeled document from the plurality of candidate labeled documents obtained based on a similarity measure between each of the plurality of candidate labeled documents and an unlabeled document. In some embodiments, determining a labeled document from a plurality of candidate labeled documents may be based on a similarity function and a best match algorithm. A similarity function may be a function that is capable of processing a pair of documents (i.e., in this case, a corresponding candidate labeled document and an unlabeled document) and returning a similarity measure based on the processing. An example of a similarity measure may be a score indicating the similarity of two documents.

[0047] In some embodiments, the similarity function may be based on a bag-of-words model that compares document pairs by the number of common words across documents (which may or may not include repeated words). For example, if a document consists of multiple tokens "aaabc" and another document consists of multiple tokens "aaccd", the similarity function may return a similarity score of 3 because the document pair shares three tokens: two 'a's and one 'c'.

[0048] Once all similarity scores between all candidate labeled document pairs and unlabeled document pairs have been determined in this manner, a best match algorithm can then be used to select a "best match" based on the similarity measure between each of the plurality of candidate labeled documents and the unlabeled document. For example, a best match algorithm can be used to select the document pair with the highest similarity score.

[0049] In some cases, a similarity score threshold can be used to determine that none of the plurality of candidate labeled documents are suitable for performing data capture according to the methods described herein. For example, if the similarity score threshold is set to 4, and if none of the respective similarity scores of the document pairs (each pair comprising a candidate labeled document and an unlabeled document) is equal to or greater than 4, then the method may also include providing an output to indicate that none of the respective (current) candidate labeled documents can be used for data capture. The method may also include obtaining a set of a plurality of new candidate labeled documents, and determining the labeled document based on a similarity measure between each of the plurality of new candidate labeled documents and the unlabeled document.

[0050] Alternatively or additionally, in some cases, a similarity score threshold can be used to determine that multiple candidate labeled documents are suitable for performing data capture according to the methods described herein. For example, if the similarity score threshold is set to 4, and if more than one document pair (each pair containing a candidate labeled document and an unlabeled document) has a similarity score equal to or greater than 4, the step of determining an anchor at step S110 can be based on multiple labeled documents (each labeled document corresponding to a similarity score above the threshold) and unlabeled documents. As described above, in this case, each anchor can include multiple points of matching, each of the multiple points being in a different document. Furthermore, in this case, the step of generating a total kernel at step S120 can be based on the multiple labeled documents and unlabeled documents. In other words, each convolution kernel can involve the multiple labeled documents (each labeled document corresponding to a similarity score above the threshold) and unlabeled documents.

[0051] Figure 2 2 is a block diagram of a system for performing data capture according to an embodiment of the present disclosure. In more detail, system 200 is used to perform data capture in unlabeled documents based on labeled documents. The labeled document contains one or more elements, each element is labeled with respect to a field. Each element can be represented by a word, a string of words, or a number, or a combination thereof. In some embodiments, the labeled document may also include one or more fields indicated by one or more tags. The labeled document and / or the unlabeled document may be a PDF document or an image.

[0052] like Figure 2 As shown, the system 200 includes a determination unit 210, a generation unit 220, and an identification unit 230. The determination unit 210 is configured to determine one or more anchors. As described above, the anchors include point pairs that match between labeled documents and unlabeled documents. In some implementations, the determination unit 210 can be configured to determine a new set of one or more anchors for each new document pair (wherein the pair includes an unlabeled document and a labeled document).

[0053] In some embodiments, the determination unit 210 may be configured to determine the anchor by receiving user input selecting a point pair in an unlabeled document and a labeled document. Alternatively, in some embodiments, the determination unit 210 may be configured to determine the anchor by the following steps: - extract all words in the labeled and unlabeled documents (e.g. using OCR); - generating a first linear text sequence by ordering the words contained in the labeled document into a linear text sequence (and optionally removing any words corresponding to labeled elements); - generating a second linear text sequence by sorting the words contained in the unlabeled document into a linear text sequence; - identifying matches between the first and second linear text sequences (if words corresponding to marked elements are not removed at this stage, these words are ignored in the match identification step); and - Designate the matched word as an anchor.

[0054] Matching in this context (e.g., visually) involves the inherent similarity of the two words, as well as the similarity or match in terms of the linear order of the corresponding words in the corresponding sequences. In these embodiments, the match between the first and second linear text sequences can be a match that exceeds a predetermined threshold in terms of both inherent similarity and alignment similarity. In addition, the determination unit 210 can be configured to identify a match (more specifically, determine whether the two words exceed an alignment similarity threshold) based on an alignment algorithm (e.g., an algorithm similar to or equivalent to the Needleman-Wunsch algorithm).

[0055] In these embodiments, the determining unit 210 may be configured to sort the words contained in the marked and unmarked documents using a reading order algorithm. In addition, in these embodiments, each of the one or more anchors may be defined as a correspondence between the center of a corresponding matching word in the marked document and the center of a corresponding matching word in the unmarked document.

[0056] In some embodiments, the determination unit 210 can be configured to determine the anchor by identifying visual features present in both the labeled document and the unlabeled document (e.g., using an image alignment algorithm) and designating the identified visual features as anchors. The image alignment algorithm can be an algorithm based on at least one of the following: a key point detector (e.g., Gaussian difference, Harris, good features to track (GFTT)), a local invariant descriptor (e.g., scale-invariant feature transform (SIFT), accelerated robust features (SURF), and oriented FAST and rotation BRIEF (ORB), etc.) and a key point matching process (e.g., random sampling consensus (RANSAC)).

[0057] In some embodiments, the determination unit 210 can be configured to determine the anchor by: placing a plurality of points around the boundary of the unlabeled document and placing the same number of points around the boundary of the labeled document; identifying a match between points in the labeled document and points in the unlabeled document; and designating the matched points as anchors.

[0058] Although the present embodiment describes determining one or more anchors for a document pair (i.e., a labeled and unlabeled document), it should be understood that the above process for determining the anchor by the determination unit 210 can be applied to more than two documents (e.g., multiple labeled documents and / or multiple unlabeled documents). In addition, although the present embodiment describes that the anchor can include a matched single word, in other embodiments, the anchor can include a matched phrase instead of a word, for example, the word "totalamount (total amount)" is treated as a single phrase instead of two separate words.

[0059] The generation unit 220 is configured to generate a total kernel using the determined one or more anchors. In some embodiments, the generation unit 210 may be configured to generate the total kernel by the following steps: generating a convolution kernel for each of the one or more anchors, and generating the total kernel by summing the one or more convolution kernels. The total kernel may be a weighted sum of the one or more convolution kernels.

[0060] Each convolution kernel may involve all word pairs within and between labeled and unlabeled documents. In some embodiments where there are more than two documents on which data capture is based, each convolution kernel may involve all word pairs within and across all documents. Each convolution kernel may measure the similarity of words within and between labeled and unlabeled documents (or across all documents involving more than two documents), where the similarity is a function of the similarity of the distance of the corresponding words from the corresponding point of the anchor and the relative position of the corresponding point relative to the anchor. The mathematical details of the convolution kernels are referenced above. Figure 1 Therefore, for the sake of brevity, it is not repeated here. It should be understood that these mathematical equations can be applied to this embodiment. In addition, as mentioned above, Figure 1 As explained, the convolution kernel and the total kernel defined with respect to the set of documents can each be represented as a symmetric positive semi-definite matrix having a dimension corresponding to the number of words in all documents in the set of documents, and each matrix entry in the matrix corresponds to a corresponding similarity between a specific pair of words in the documents in the set of documents.

[0061] Although it is described above that each convolution kernel may involve word pairs, in alternative embodiments, the convolution kernels may involve phrase pairs (e.g., each phrase pair consists of multiple words) instead of or in addition to single word pairs.

[0062] The recognition unit 230 is configured to use the total kernel in the kernel machine to identify one or more elements to be captured in the unlabeled document. The one or more elements to be captured correspond to the one or more labeled elements contained in the labeled document in terms of fields. The kernel machine may include at least one of the following: support vector machine (SVM), regularized kernel discriminant analysis (RKDA), and Gaussian process (GP). The kernel machine can be trained based on one or more elements and one or more fields indicated by corresponding tags in the labeled document. In some cases, the kernel machine can be trained based on all of one or more elements and one or more fields indicated by corresponding (one or more) tags in the labeled document to maximize the number of samples used for training. In some embodiments, a subset of matrix entries for the total kernel can be used to train the kernel machine, wherein each matrix entry in the subset of matrix entries for the total kernel corresponds to a word pair within the labeled document. The identified elements may be elements corresponding to words with high (predicted) similarity / confidence scores determined by the kernel machine. In these embodiments, the system 200 may include an output unit configured to output (one or more) similarity / confidence scores determined by the kernel machine.

[0063] although Figure 2 Not shown, but in some embodiments, the system 200 may further include a receiving unit configured to receive at least one of user input approving or rejecting the result of the recognition and user input modifying the result of the recognition after the recognition unit 230 recognizes the one or more elements. In addition, the system 200 may include an output unit configured to output the result (e.g., via a display) if user input approving the result is received, and / or an execution unit configured to perform further processing of the result (e.g., field determination and / or grouping, so the execution unit may be a determination unit). If user input to modify the result of the recognition is received, the execution unit may be configured to perform the modification based on the user input, and / or the storage unit in the system 200 may be configured to store the modified result as at least a portion of a new marked document.

[0064] In some embodiments, the determination unit 210 may also be configured to determine, for each of the identified one or more elements, the field to which the corresponding element corresponds. This determination may be performed using a total core in a core machine (e.g., a core machine used by the identification unit 230 to identify the element(s). It will be appreciated that in some embodiments, the determination unit 210 may be configured to simultaneously perform the identification of the one or more elements and the field determination for each of the identified one or more elements as a single step / process in the core machine.

[0065] although Figure 2, but in some embodiments where there are multiple identified elements, the determination unit 210 can be configured to determine fields for the multiple identified elements by grouping the multiple identified elements into one or more groups. In these embodiments, each group can correspond to a field. The grouping can be based on a word grouping algorithm. More specifically, the determination unit 210 can be configured to group the multiple identified elements by: - generating a linear text sequence of the plurality of identified elements in an unlabeled document; - determining (e.g., using a kernel machine) a classification for each element in the generated linear text sequence; and -Group each maximal contiguous subsequence of elements sharing a classification in the generated linear text sequence into corresponding groups.

[0066] The classification determined for each element may be an argmax classification. The determination unit 210 may determine the classification based on a plurality of possible classifications, wherein each possible classification corresponds to a field in a tagged document or corresponds to an indication that the element is not associated with any field in a tagged document.

[0067] Furthermore, in these embodiments, the system 200 may further include an assigning unit configured to assign a corresponding row identifier to each element in the group based on the corresponding order of the elements in the group.

[0068] In some embodiments, the one or more fields and the one or more elements in the marked document may be presented in an information structure. In these embodiments, the system 200 may further include an output unit configured to output the identified one or more elements and their corresponding fields in the same (or similar) information structure. Alternatively, the output unit may be configured to output only the one or more elements (regardless of whether the fields and / or elements are presented in the same / similar or different information structures).

[0069] The system 200 may further include an allocating unit configured to allocate a new label to each of the identified one or more elements to indicate a field to which the corresponding element corresponds.

[0070] In some embodiments, the determining unit 210 may also be configured to place each of the one or more identified elements into a normalized form. This operation may be based on field-agnostic rules or based on one or more field-specific rules.

[0071] It should be understood that in some embodiments, the use of tags in the marked document is sufficient to indicate which element(s) are elements of interest, and therefore also indicate which corresponding element(s) in the unmarked document should be identified and captured. However, in some embodiments, the method may also include an acquisition unit configured to acquire a pattern that defines the one or more fields contained in the marked document and whether the elements corresponding to each field in the marked document are presented in a table before the determination unit 210 determines one or more anchors. The pattern can provide more obvious information about what data should be captured from the unmarked document. This pattern can be used, for example, by the determination unit 210 when it determines the anchor, so that each of the one or more fields defined by the pattern can be used as a point of the anchor.

[0072] In some embodiments, the system 200 may also include a receiving unit configured to receive user input that assigns one or more tags to elements of the tagged document before the determining unit 210 determines one or more anchors. In some implementations, the receiving unit may be configured to receive user input for assigning tags by: receiving a user selection of a field in the tagged document, and receiving one or more user selections of one or more elements corresponding to the selected field. As an example, user selection of an element may be implemented by allowing a user to draw a rectangular box around an element on the document, for example, via a user interface. The field corresponding to the tag (which may be represented by a rectangular box) may be indicated by means of a visual indicator. For example, each rectangular box may have a color indicating the corresponding field of the element. In addition, in some embodiments, each tag may contain metadata (e.g., metadata indicating whether the field is in the form of a number, metadata indicating whether the element corresponding to the field is in a tabular format, etc.).

[0073] In some embodiments, the system 200 may further include an acquisition unit configured to acquire multiple candidate labeled documents. In these embodiments, the system 200 may further include a receiving unit configured to receive user input for selecting a labeled document from the multiple candidate labeled documents. Alternatively, in these embodiments, the determination unit 210 may further be configured to determine the labeled document from the acquired multiple candidate labeled documents based on a similarity measure between each document in the multiple candidate labeled documents and the unlabeled document. More specifically, the determination unit 210 may be configured to determine the labeled document based on a similarity function and a best matching algorithm. The above reference Figure 1 The similarity function and the best matching algorithm are described and explained in detail, so the technical functionality of these components will not be repeated here, but will be understood to apply equally as well. The operations of the acquisition unit, the receiving unit, and the determination unit described in these embodiments may be performed before the determination unit 210 determines the one or more anchors.

[0074] Figure 3A and Figure 3B is a diagram showing an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure. Figure 3A and 3B As shown, there are provided marked documents 310 and unmarked documents 320. In this embodiment, the marked documents 310 and unmarked documents 320 are bills (ie, semi-structured documents).

[0075] In the marked document 310, the information is presented in a field-element pair format or in a table format. For example, "Bill #" and "00123456" are presented as a field-element pair, "Bill Date" and "September 2, 2021" are presented as a field-element pair, and "Total" and "4596" are presented as a field-element pair. In these examples, "Bill #", "Bill Date", and "Total" are considered fields, while "00123456", "September 2, 2021", and "4596" are considered corresponding elements.

[0076] In this example, the rest of the information is presented in a table format, where an element corresponds to one of the fields "Description", "Quantity", "Unit Price", and "Amount". In more detail, the elements "Canon 5D", "Memory Card", and "Lens" correspond to "Description", which are descriptions of the purchased items. The elements "1", "2", and "1" correspond to "Quantity", which are numbers indicating the quantity of each purchased item. The elements "2499", "149", and "1799" correspond to "Unit Price", which are numbers indicating the unit price of each purchased item. The elements "2499", "298", and "1799" correspond to "Amount", which are numbers indicating the total amount (price) of each purchased item.

[0077] like Figure 3A As can be seen in the figure, all elements are labeled (indicated in the figure by boxes surrounding each element, which boxes have corresponding reference numbers 311, 312, 313A-C, 314A-C, 315A-C, 316A-C, and 317). This can be achieved by receiving user input indicating which words and / or phrases in the document (or which parts of the image pixels) are to be considered elements.

[0078] A label can indicate the field to which an element corresponds. In this example, the label for "00123456" (i.e., label 311) can indicate that the corresponding element corresponds to the field "Bill #". Similarly, the label for "September 2, 2021" (i.e., label 312) can indicate that the corresponding element corresponds to the field "Bill Date". Labels 313A-C can indicate that the elements "Canon 5D", "Memory Card", and "Lens" correspond to the field "Description", labels 314A-C can indicate that the elements "1", "2", and "1" correspond to the field "Quantity", labels 315A-C can indicate that the elements "2499", "149", and "1799" correspond to the field "Amount", and label 317 can indicate that the element "4596" corresponds to the field "Total".

[0079] Reference now Figure 3A In the unlabeled document 320 in FIG. 1 , it can be seen that the bill follows a similar information structure as the labeled document 310. Specifically, the unlabeled document 320 includes the same fields: "Bill #", "Bill Date", "Description", "Quantity", "Unit Price", "Amount", and "Total". By applying the above reference Figure 1 The computer-implemented method 100 described above or using the Figure 2 The system 200 is capable of identifying and outputting elements in unlabeled documents.

[0080] Figure 3BAn example of this output is shown in , where the unlabeled document 320 is now provided with labels 321, 322, 323, 324, 325, 326, and 327, each corresponding to an identified element. In this case, the individual identified elements are "00123457", "September 3, 2021", "Sony ZV-1", "1", "699", "699", and "699". These identified elements correspond to the elements that have been labeled in the labeled document 310 in terms of their fields. Specifically, element "00123457" corresponds to element "00123456" in terms of its field ("Billing #"), element "September 3, 2021" corresponds to element "September 2, 2021" in terms of its field ("Billing Date"), element "Sony ZV-1" corresponds to elements "Canon 5D", "Memory Card" and "Lens" in terms of its field ("Description"), and element "1" corresponds to elements "1", "2" and "1" in terms of its field ("Quantity") (marked as 314A). -C), the unit price "699" (labeled 325) corresponds to the elements "2499", "149", "1799" (labeled 315A-C) in terms of its field ("unit price"), the element "699" (labeled 326) corresponds to the elements "2499", "298" and "1799" (labeled 316A-C) in terms of its field ("amount"), and the element "699" (labeled 326) corresponds to the element "4596" in terms of its field ("total").

[0081] Figure 4 It shows confirmation Figure 3A and 3B A diagram of an exemplary process of anchoring between labeled and unlabeled documents. Figure 4 An upper linear text sequence and a lower linear text sequence are shown. The upper linear text sequence can be generated by extracting all words in the marked document 310 and sorting the extracted words (e.g., by their reading order) into a linear text sequence. Similarly, the lower linear text sequence can be generated by extracting all words in the unmarked document 320 and sorting the extracted words (e.g., by their reading order) into a linear text sequence.

[0082] In this example, any words in the upper linear text sequence that correspond to the identified elements (e.g., "00123456", "September 2, 2021", "Canon 5D", etc.) are ignored in the process of determining the anchor. Alternatively, in other embodiments or examples, words corresponding to the identified elements can be removed from the linear text sequence in the process of determining the anchor.

[0083] Once the linear text sequences corresponding to the labeled and unlabeled documents 310, 320 have been generated, matches can be identified between the two linear text sequences. As indicated by the arrows between the upper and lower linear text sequences, ten matches have been identified in this example (from left to right): "Bill", "#", "Bill", "Date", "Description", "Quantity", "Units", "Price", "Amount", and "Total". Note that, at least in this example, each individual block of text can be considered a "word". Thus, "Bill" and "#" are considered two (separate) words, and similarly, "Units" and "Price" are considered two words, even though "UnitPrice" is collectively considered to be one field.

[0084] Each match that has been identified is then designated as an anchor, thereby providing ten anchors corresponding to the words "bill", "#", "bill", "date", "description", "quantity", "unit", "price", "amount", and "total" in this example. Although the matches identified in this example are all exact matches, it will be understood that in some other examples or embodiments, the matches may not be 100% matches (e.g., due to OCR errors).

[0085] Figure 5 It shows that according to Figure 4 FIG. 1 is a diagram of an anchor determined by an exemplary process shown. To illustrate one aspect of the present disclosure, Figure 5 Only if the document 510 is pointed to (which is related to Figure 3A and 3B ) and unlabeled documents 520 (which are identical to Figure 3A and 3B The anchor is focused on as shown by the upward arched arrow of the word "quantity" in the marked document 320 of FIG. Figure 4 The process described identifies one of the matches.

[0086] As above reference Figure 1 and 2 As described, in some embodiments, the generation of the total kernel includes generating a convolution kernel for each of the one or more anchors, each convolution kernel involving all pairs of words and / or phrases within and between labeled documents and unlabeled documents. More specifically, each convolution kernel can measure the similarity of words and / or phrases within and between labeled documents and unlabeled documents, and the similarity is a function of the similarity of the distance of the corresponding words and / or phrases from the corresponding point of the anchor and the relative position relative to the corresponding point of the anchor. Therefore, in this example, the convolution kernel of the anchor "number" can measure the similarity of words and / or phrases within the labeled document 510, within the unlabeled document 520, and between the labeled document 510 and the unlabeled document 520 relative to the anchor.

[0087] As an example of how to measure the similarity of words, Figure 5 Focus on the similarity between "1" in the labeled document 510 and "1" in the unlabeled document 520 (as indicated by the lower arrow pointing to the two words). This similarity is a function of the distance between "1" and "quantity" in the labeled document 510 (as indicated by the short arrow in the document), the distance between "1" and "quantity" in the unlabeled document 520 (as indicated by the short arrow in the document), and the similarity of the relative position of the word "1" with respect to the word "quantity" in each document.

[0088] As described above with reference to multiple mathematical equations, at least in some implementations, words and / or phrases are considered similar only when they are close to the anchor and their positions relative to the anchor are similar (by taking the product of the offset kernel and the relative kernel). In this case, the similarity measured by the convolution kernel will be high due to the proximity of "1" and "quantity" in the labeled document 510, the proximity of "1" and "quantity" in the unlabeled document 520, and the similarity of the relative positions of the word "1" relative to the word "quantity" in each document (such as two "1"s are located below the word "quantity"). In contrast, the measured similarity between the word "4596" in the labeled document 510 and the word "00123457" in the unlabeled document 520 relative to the same anchor will be lower due to the fact that they are both farther away from the word "quantity" in the corresponding documents and their relative positions relative to "quantity" are different.

[0089] Convolution kernels can be generated for other anchors in a similar manner, where each convolution kernel involves all word and / or field pairs within and between labeled and unlabeled documents relative to the corresponding anchor. Once all convolution kernels have been generated, they can be summed to generate a total kernel for the purpose of identifying elements from unlabeled documents 520.

[0090] Figure 6 is a diagram showing an exemplary labeled document and an exemplary unlabeled document according to an embodiment of the present disclosure. Figure 6 As shown, there are provided marked documents 610 and unmarked documents 620. In this embodiment, the marked documents and the unmarked documents are bills (ie, semi-structured documents).

[0091] In the marked document 610, some information is presented in a field-element pair format. For example, "Bill Number" and "237481709" are presented as a field-element pair, "Total Amount Payable" and "£1901.98" are presented as a field-element pair, and "Due Date" and "May 16, 95" are presented as a field-element pair. In these examples, "Bill Number", "Total Amount Payable" and "Due Date" are considered fields, while "237481709", "£1901.98" and "May 16, 95" are considered corresponding elements.

[0092] Some other information in the marked document 610 is presented in a table format, where the elements correspond to one of the fields "description", "price", "discount", "VAT", and "amount". For example, the elements "energy bar", "100% juice apple grape", and "natural mineral water" correspond to "description", which are descriptions of the purchased goods.

[0093] Furthermore, in this example, some information is presented as elements (without explicit corresponding fields). For example, the element "Bridge Interiors, 56 Bridge Lane, Ilkley LS29 9EU" is an address, so it is understood to correspond to the "address" field. Although this field is not explicitly presented in the marked-up document, the tag 612 assigned to this element may indicate that the element corresponds to the field "address".

[0094] like Figure 6 As can be seen, only some of the various elements in the tagged document 610 are labeled (indicated in the figure by boxes surrounding each element with corresponding reference numerals 611, 612, 613, 614 and 615), which can be achieved by receiving user input indicating which words and / or phrases in the document (or which portions of image pixels) are to be considered elements. In this example, not all elements in the tagged document 610 are assigned labels, which may be because some elements (unlabeled) are not of interest and therefore it is not desired or necessary to capture elements corresponding to these elements in terms of fields. For example, even though "Place purchase order" and "855496" are presented as a field-element pair, "855496" is not assigned a label. As another example, no element corresponding to "price" is labeled in the tagged document 610.

[0095] In this example, the label may indicate the field to which the element corresponds. For example, as described above, label 612 may indicate that the corresponding element corresponds to the field "address". Similarly, the label of "237481709" (i.e., label 611) may indicate that the corresponding element corresponds to the field "bill number"; the label of "£1901.98" (i.e., label 613) may indicate that the corresponding element corresponds to the field "total amount payable"; the label of "May 16, 95" (i.e., label 614) may indicate that the corresponding element corresponds to the field "due date"; and the labels 615 of the various elements under "description" may indicate that these elements correspond to the field "description".

[0096] Reference now Figure 6 In the unlabeled document 620 in FIG. 1 , it can be seen that the bill follows a similar information structure as the labeled document. Specifically, the unlabeled document 620 includes the same fields, such as "bill number", "amount due", "due date", and "description". By applying the above reference Figure 1 The computer-implemented method 100 described above or using the Figure 2 The system 200 can identify and output elements in an unlabeled document.

[0097] As an example of the anchor determination aspect of the data capture process, Fig. 7A Shown based on Figure 6 Text anchors determined by labeled and unlabeled documents. Fig. 7A The marked document 710 in corresponds to Figure 6 The marked document 610 corresponds to Figure 6 The text anchor in this example is indicated by an arrow across the two documents, pointing to the word "description" 711 in the marked document 710 and the word "description" 721 in the unmarked document 721. This anchor can be determined in multiple steps: (1) extract all words in the labeled document 710 and the unlabeled document 720 (e.g., using OCR); (2) generating a first linear text sequence by sorting the words contained in the marked document 710 into a linear text sequence; (3) generating a second linear text sequence by sorting the words contained in the unlabeled document 720 into a linear text sequence; (4) identifying a match between the first and second linear text sequences (ignoring words corresponding to labeled elements, such as "energy bar"); and (5) Designate the matched word as an anchor.

[0098] Once this anchor ("description") is determined, a convolution kernel can be generated for the anchor, where the convolution kernel involves all pairs of words and / or phrases within and between the labeled document 710 and the unlabeled document 720. The convolution kernel measures the similarity of words and / or phrases within and between the labeled document and the unlabeled document, where the similarity is a function of the distance of the corresponding words from the corresponding point of the anchor and the similarity of the relative positions relative to the corresponding point of the anchor. For example, for these two documents, the phrase "energy bar" 712 and the word "sourdough bread flour" 722 in the labeled document will have a high measured similarity (relative to the anchor "description") because they are both in close proximity to the word "description" in their respective documents and their relative positions relative to the word "description" are similar (i.e., "energy bar" and "sourdough bread flour" are both directly below "description" in their respective documents).

[0099] Convolution kernels can be generated for other anchors in a similar manner, where each convolution kernel involves all word and / or phrase pairs within and between labeled and unlabeled documents relative to the corresponding anchor. Once all convolution kernels have been generated, they can be summed to generate a total kernel for the purpose of identifying elements from unlabeled documents 720.

[0100] As another example of the anchor determination aspect of the data capture process, Figure 7B A visual anchor determined based on a marked document 710 and an unmarked document 720 is shown. The visual anchor in this example is indicated by an arrow across the two documents, which points to a horizontal separator 713 in the marked document 710 and a horizontal separator 723 in the unmarked document 720. This anchor can be determined by identifying visual features present in both the marked document and the unmarked document and specifying the identified visual features as anchors. In this example, the identified visual features are the horizontal separators in the two documents. The identification of visual features can be based on an image alignment algorithm, such as an algorithm based on SIFT.

[0101] Once this visual anchor is determined, a convolution kernel can be generated for the anchor, where the convolution kernel involves all pairs of words and / or phrases within or between the labeled document 710 and the unlabeled document 720. The convolution kernel measures the similarity of words and / or phrases within and between the labeled and unlabeled documents as a function of the similarity of the distance of the corresponding words from the corresponding point of the anchor and the relative position of the corresponding point relative to the anchor.

[0102] For example, the phrase "100% juice apple grape" in the labeled document 710 and the phrase "protein bar cookies & cream flavor" in the unlabeled document 720 will have a high measured similarity (relative to the visual anchor) due to the relatively close proximity of these phrases to the horizontal divider in their respective documents and their relative positions relative to the horizontal divider are similar. In contrast, the phrase "pepper mayonnaise" in the labeled document 710 and the phrase "onion rings" in the unlabeled document have a lower measured similarity (relative to the visual anchor) due to the fact that at least the phrase "pepper mayonnaise" is located farther away from the horizontal divider and the relative positions of these phrases ("pepper mayonnaise" vs. "onion rings") are substantially different.

[0103] Figure 8 A marked document 810, a first unmarked document 820, and a second unmarked document 830 are shown. The marked document 810 corresponds to Figure 6 The marked document 610 shown, and the first unmarked document 820 corresponds to Figure 6 The unlabeled document 620 is shown. The second unlabeled document 630 is also a bill with a similar information structure as the other two documents. In this example, anchors have been determined for the labeled document 810 and the unlabeled document 820, which are indicated by arrows pointing to box 816 (in the labeled document 810) and box 826 (in the first labeled document 820) and by the anchor symbols in both documents. This anchor can be used as a reference when determining an anchor for a new unlabeled document (e.g., a document belonging to the same batch of unlabeled documents as 820), which in this case is the second unlabeled document 830. Figure 8 As can be seen in FIG. 8 , a new anchor indicated by the arrows pointing to box 816 (in the marked document 810 ) and box 836 (in the second unmarked document 830 ) and indicated by the anchor symbols in both documents can be determined in this way.

[0104] Fig. 9 9 is a diagram illustrating a topology for performing data capture according to an embodiment of the present disclosure. Topology 900 may include a client computing device 905 and a server 920 configured to be communicatively coupled via a network 910.

[0105] The network 910 may be a wired or wireless network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a near field communication (NFC) network, Bluetooth, infrared, radio frequency, a cellular network, or another type of network. It will be understood that the network 910 may be a combination of multiple different types of wired or wireless networks.

[0106] Each client computing device 905 may be a smartphone, tablet computer, laptop computer, personal data assistant, or any other type of mobile device having a hardware processor configured to process instructions and connected to one or more portions of network 910. Each client computing device 905 may have a graphical user interface configured to allow a user to interact with the processor of the client computing device 905.

[0107] Server 920 may include a physical computing device residing at a specific location, or may be deployed in a cloud computing network environment. In the present disclosure, "cloud computing" may be defined as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services), which may be quickly provisioned via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly. A cloud model may consist of various features (e.g., on-demand self-service, wide area network access, resource pools, rapid elasticity, measured services, etc.), service models (e.g., software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0108] It should be understood that the description is by way of example only; variations and modifications may be made to the described embodiments without departing from the scope of the present disclosure as defined in the claims.

[0109] Although the present technology has been described in detail for purposes of illustration based on what are presently considered to be the most practical and preferred implementations, it should be understood that such detail is solely for that purpose and that the present technology is not limited to the disclosed implementations, but rather is intended to cover modifications and equivalent arrangements within the scope of the appended claims. For example, it should be understood that the present technology contemplates that, to the extent possible, one or more features of any embodiment may be combined with one or more features of any other embodiment.

[0110] Throughout the specification, references to "one embodiment," "an embodiment," or "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present disclosure. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing throughout this specification do not necessarily all refer to the same embodiment or example. In addition, in one or more embodiments or examples, the particular features, structures, or characteristics may be combined in any suitable combination and / or sub-combination. In addition, it should be understood that the figures provided herein are for the purpose of explanation to persons of ordinary skill in the art, and that the figures are not necessarily drawn to scale.

[0111] The embodiments of the present disclosure may be implemented as a device, a method or a computer program product. For example, in some embodiments, a computer program product including instructions may be provided, and when the program is executed by a processor, the instructions cause the processor to perform a Figure 1 As another example, in some embodiments, a computer-readable storage medium may be provided that includes instructions that, when executed by a computer, cause the computer to perform Figure 1 Thus, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be collectively referred to herein as a "module" or "system." Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0112] Although described in conjunction with an exemplary computing system environment, embodiments of the present disclosure may be used in numerous other general or special computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be applicable to aspects of the present disclosure include, but are not limited to, mobile computing devices, personal computers (e.g., desktop computers), server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0113] Any combination of one or more computer-usable or computer-readable media may be utilized. For example, a computer-readable medium may include one or more of a portable computer disk, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or flash memory) device, a portable compact disk read-only memory (CDROM), an optical storage device, and a magnetic storage device. The computer program code for performing the operations of the embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0114] Flowcharts and block diagrams illustrate possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure, functions, and operations. In this regard, each box in a flow chart or block diagram may represent a module, segment, or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It will also be noted that each box in a block diagram and / or flow chart and a combination of boxes in a block diagram and / or flow chart may be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action. These computer program instructions may also be stored in a computer-readable medium, which may guide a computer or other programmable data processing device to work in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including an instruction device that implements the function / action specified in the flow chart and / or block diagram. The following is a list of embodiments of the present disclosure: 1. A computer-implemented method for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document contains one or more elements, each element being labeled with respect to a field, the method comprising: determining one or more anchors, wherein the anchors comprise point pairs matched between the labeled document and the unlabeled document; generating a total core using the determined one or more anchors; and The overall kernel is used in a kernel machine to identify one or more elements to be captured in the unlabeled document, wherein the one or more elements to be captured correspond in terms of fields to the one or more labeled elements contained in the labeled document. 2. The method according to embodiment 1, wherein generating the total core comprises: generating a convolution kernel for each of the one or more anchors; and The total kernel is generated by summing the one or more convolution kernels. 3. A method according to embodiment 2, wherein the total kernel is a weighted sum of the one or more convolution kernels. 4. A method according to embodiment 2 or embodiment 3, wherein each convolution kernel involves all word pairs within and between the labeled document and the unlabeled document. 5. A method according to any one of embodiments 2 to 4, wherein each convolution kernel measures the similarity of words within and between the labeled document and the unlabeled document, wherein the similarity is a function of the similarity of the distance of the corresponding words from the corresponding point of the anchor and the relative position of the corresponding point relative to the anchor. 6. The method according to any one of embodiments 2 to 5, wherein the marked document is defined as d=d 1, the unlabeled document is defined as d = d 2 , d is a document, and each document is defined as a sequence of words n d is the number of words in the corresponding document, and Among them, each convolution kernel is defined as k(w d，i , w d′，j ; a) = k off (z)k off (z′)k rel (z, z′), each word pair is represented as (w d，i , w d′，j ),as well as where a is the corresponding anchor and is defined as a=(a 1 , a 2 ), a 1 Yes 1 The position of the corresponding point of the anchor in, and a 2 Yes 2 The positions of the corresponding points of the anchors in , and where k off (z) = k RBF (||zA off || 2 ),k rel (z, z′) = k RBF (||(zz′)A rel || 2 ),as well as in and and where r is the distance and σ is the length scale. 7. The method according to embodiment 6, wherein d=d′ and / or d≠d′. 8. The method according to any of the preceding embodiments further comprises, after identifying the one or more elements in the unlabeled document, performing at least one of the following: receiving user input approving or rejecting a result of the identification; and User input is received to modify a result of the recognition. 9. A method according to any one of the preceding embodiments, wherein the marked document further includes one or more fields indicated by one or more tags. 10. The method according to any one of the preceding embodiments further comprises determining, for each element of the one or more identified elements, a field corresponding to the corresponding element, wherein the determining uses the total core in a core machine. 11. A method according to embodiment 10, wherein multiple elements are identified in the unlabeled document, and wherein determining the field corresponding to each of the one or more identified elements includes grouping the multiple identified elements into one or more groups, each group corresponding to a field. 12. The method of embodiment 11, wherein grouping the plurality of identified elements is based on a word grouping algorithm. 13. The method of embodiment 11 or embodiment 12, wherein grouping the plurality of identified elements comprises: generating a linear text sequence of the plurality of identified elements in the unlabeled document; determining the argmax classification of each element in the generated linear text sequence; and Each maximal contiguous subsequence of elements sharing the argmax classification in the generated linear text sequence is grouped into corresponding groups. 14. The method of any one of embodiments 11 to 13 further comprises assigning a corresponding row identifier to each element in the group based on the relative order of the elements in the group. 15. A method according to any one of embodiments 9 to 14, wherein the one or more fields and one or more elements in the marked document are presented in an information structure, and the method also includes outputting the identified one or more elements and their corresponding fields in the same information structure. 16. The method according to any one of embodiments 9 to 15 also includes, before determining one or more anchors, obtaining a schema that defines the one or more fields contained in the marked document and whether the elements corresponding to each field in the marked document are presented in a table. 17. The method of any preceding embodiment, wherein determining an anchor comprises: Extracting all words from the labeled document and the unlabeled document; generating a first linear text sequence by ordering the words contained in the labeled document into a linear text sequence and removing any words corresponding to labeled elements; generating a second linear text sequence by sorting words contained in the unlabeled document into a linear text sequence; identifying a match between the first and second linear text sequences; and The matched word is designated as the anchor. 18. A method according to embodiment 17, wherein the match between the first and second linear text sequences is a match that exceeds a predetermined similarity threshold and a predetermined alignment threshold. 19. The method according to Embodiment 17 to Embodiment 18, wherein the extraction of words from the marked document and the unmarked document is based on optical character recognition (OCR). 20. The method of any one of embodiments 17 to 19, wherein sorting the words contained in the tagged and untagged documents is performed using a reading order algorithm. 21. A method according to any one of embodiments 17 to 20, wherein each of the one or more anchors is defined as a correspondence between the center of a corresponding matching word in the marked document and the center of a corresponding matching word in the unmarked document. 22. A method according to any one of embodiments 17 to 21, wherein identifying a match between the first and second linear text sequences is based on an alignment algorithm. 23. The method of embodiment 22, wherein the alignment algorithm is similar to or identical to the Needleman-Wunsch algorithm. 24. The method of any preceding embodiment, wherein determining an anchor comprises: identifying visual features present in both the labeled document and the unlabeled document; and The identified visual feature is designated as the anchor. 25. The method of embodiment 24, wherein identifying visual features is based on an image alignment algorithm. 26. The method of embodiment 25, wherein the image alignment algorithm is based on at least one of: a key point detector, a local invariant descriptor, and a key point matching process. 27. The method of any preceding embodiment, wherein determining an anchor comprises: placing a plurality of points around a boundary of the unlabeled document and placing the same number of points around a boundary of the labeled document; identifying matches between points in the labeled document and points in the unlabeled document; and The matched point is designated as the anchor. 28. The method according to any of the preceding embodiments further includes receiving user input to assign the one or more tags to elements of the tagged document before determining one or more anchors. 29. The method of any of the preceding embodiments, further comprising, before determining one or more anchors: Obtaining a plurality of candidate labeled documents; and The labeled document is determined from the obtained plurality of candidate labeled documents based on a similarity measure between each of the plurality of candidate labeled documents and the unlabeled document. 30. A method according to any one of the preceding embodiments, wherein the kernel machine comprises at least one of the following: a support vector machine SVM, a regularized kernel discriminant analysis RKDA, and a Gaussian process GP. 31. A method according to any of the preceding embodiments, wherein the marked document is in the form of a PDF document or an image. 32. A system for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document contains one or more elements, each element being labeled with respect to a field, the system comprising: a determining unit configured to determine one or more anchors, wherein an anchor comprises a pair of corresponding points matched between the labeled document and the unlabeled document; a generating unit configured to generate a total core using the determined one or more anchors; and An identification unit configured to use the overall core in a core machine to identify one or more elements to be captured in the unlabeled document that contain information to be captured, wherein the one or more elements to be captured correspond in terms of fields to the one or more labeled elements contained in the labeled document. 33. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform the steps of the method of any one of embodiments 1 to 31.

Claims

1. A computer-implemented method (100) for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document contains one or more elements, each element being labeled with respect to a field, the method comprising: determining (S110) one or more anchors, wherein an anchor comprises a point pair matched between the labeled document and the unlabeled document; generating ( S120 ) a total core using the determined one or more anchors; and Using the total core in a core machine, one or more elements to be captured in the unlabeled document are identified (S130), wherein the one or more elements to be captured correspond in terms of fields to the one or more labeled elements contained in the labeled document.

2. The method according to claim 1, wherein generating (S120) the total core comprises: generating a convolution kernel for each of the one or more anchors; as well as The total kernel is generated by summing the one or more convolution kernels.

3. The method according to claim 2, wherein: Each convolution kernel measures the similarity of words within and between the labeled and unlabeled documents, wherein the similarity is a function of the similarity of the distance of the corresponding words from the corresponding point of the anchor and the relative position of the corresponding point relative to the anchor.

4. A method according to any one of the preceding claims, wherein: The tagged document also includes one or more fields indicated by one or more tags.

5. The method according to any of the preceding claims, further comprising determining, for each of the identified one or more elements, a field to which the corresponding element corresponds, wherein the determining uses the total core in a core machine.

6. The method according to claim 5, wherein: A plurality of elements are identified in the untagged document, and wherein determining a field to which each of the one or more identified elements corresponds includes grouping the plurality of identified elements into one or more groups, each group corresponding to a field.

7. The method according to claim 6, wherein: Grouping the plurality of identified elements comprises: generating a linear text sequence of the plurality of identified elements in the unlabeled document; determining the argmax classification of each element in the generated linear text sequence; and Each maximal contiguous subsequence of elements sharing the argmax classification in the generated linear text sequence is grouped into corresponding groups.

8. The method according to any one of claims 4 to 7, wherein: The one or more fields and the one or more elements in the marked-up document are presented in an information structure, and the method further comprises outputting the identified one or more elements and their corresponding fields in the same information structure.

9. The method according to any one of claims 4 to 8 further includes obtaining a pattern before determining (S110) one or more anchors, wherein the pattern defines the one or more fields contained in the marked document and whether the elements corresponding to each field in the marked document are presented in a table.

10. A method according to any one of the preceding claims, wherein: Determining (S110) an anchor includes: Extracting all words from the labeled document and the unlabeled document; generating a first linear text sequence by ordering the words contained in the marked document into a linear text sequence and removing any words corresponding to marked elements; generating a second linear text sequence by sorting words contained in the unlabeled document into a linear text sequence; identifying a match between the first linear text sequence and the second linear text sequence; and The matched word is designated as the anchor.

11. A method according to any one of the preceding claims, wherein: Determining (S110) an anchor includes: identifying visual features present in both the labeled document and the unlabeled document; and The identified visual feature is designated as the anchor.

12. A method according to any one of the preceding claims, wherein: Determining (S110) an anchor includes: placing a plurality of points around a boundary of the unlabeled document and placing the same number of points around a boundary of the labeled document; identifying matches between points in the labeled document and points in the unlabeled document; and The matched point is designated as the anchor.

13. The method according to any of the preceding claims, further comprising, before determining (S110) one or more anchors: Obtaining a plurality of candidate labeled documents; and The labeled document is determined from the obtained plurality of candidate labeled documents based on a similarity measure between each of the plurality of candidate labeled documents and the unlabeled document.

14. A system (200) for performing data capture in an unlabeled document based on a labeled document, wherein the labeled document contains one or more elements, each element being labeled with respect to a field, the system comprising: A determination unit (210) configured to determine one or more anchors, wherein an anchor comprises a pair of corresponding points matched between the labeled document and the unlabeled document; a generating unit (220) configured to generate a total core using the determined one or more anchors; as well as An identification unit (230) is configured to use the overall core in a core machine to identify one or more elements to be captured in the unlabeled document that contain information to be captured, wherein the one or more elements to be captured correspond in terms of fields to the one or more labeled elements contained in the labeled document.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1 to 13.

Images