System and method for field extraction from unlabeled data

Abstract

To provide a field extraction system that does not require field-level annotation for training.SOLUTION: A training process is bootstrapped by mining a pseudo-label from unlabeled forms using simple rules. Then, a transformer-based structure is used to model interaction between text tokens in an input form and predict a field tag for each token accordingly. The pseudo-label is used to supervise transformer training. Since the pseudo-labels are noisy, a refinement module including a series of branches is used to refine the pseudo-labels. Each of the refinement branches performs field tagging, and generates refined labels. At each stage, a branch is optimized by labels ensembled from all previous branches to reduce label noise.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] This application is a nonprovisional adaptation of U.S. Provisional Application No. 63 / 189,579, both filed May 17, 2021, and claims priority to U.S. Nonprovisional Application Nos. 17 / 484,618 and 17 / 484,623, which in turn claim priority under 35 U.S.C. § 119.

[0002] All of the above applications are expressly incorporated herein by reference in their entirety.

[0003] The present embodiments relate generally to machine learning systems and computer vision, and more particularly to mechanisms for extracting fields from forms with unlabeled data. [Background technology]

[0004] Form-like documents, such as invoices, payslips, and medical information forms, are commonly used in everyday business workflows. Extracting fields from various forms can often be a challenging task. For example, document layout and text representation can vary even for the same form type. Forms are issued by different vendors; for example, invoices from different companies may have significantly different designs, and payslips from different systems (e.g., ADP and Workday) may have different text representations for similar information. Traditionally, extracting information from such form documents required significant human effort. For example, human workers are typically given a list of expected form fields, such as purchase_order, invoice_number, and total_amount, and then extract the corresponding values ​​based on their understanding of the form.

[0005] Therefore, there is a need for an efficient system for information extraction from form documents. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 2 is a schematic diagram illustrating an example of information extraction from an invoice according to embodiments described herein.

[0007] [Figure 2] FIG. 1 is a simplified diagram illustrating an overall self-supervised training framework of a field extraction system according to embodiments described herein.

[0008] [Figure 3] FIG. 3 is a block diagram illustrating an example framework for refining the field extraction framework described in FIG. 2 with a pseudo label ensemble (PLE), according to embodiments described herein.

[0009] [Figure 4] FIG. 1 is a simplified diagram of a computing device implementing a field extraction framework according to some embodiments described herein.

[0010] [Figure 5] FIG. 1 is a simplified diagram of a method for field extraction from a form with unlabeled data via a field extraction model, according to some embodiments.

[0011] [Figure 6] FIG. 1 is a schematic diagram of a method for label refinement in field extraction from forms with unlabeled data via a field extraction model, according to some embodiments.

[0012] [Figure 7] 1 is a data table providing an example key list and date types for a training dataset of unlabeled form data, according to some embodiments.

[0013] [Figure 8A]1 illustrates an exemplary label-less form, according to some embodiments. [Figure 8B] 1 illustrates an exemplary label-less form, according to some embodiments.

[0014] [Figure 9] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 10] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 11] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 12] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 13] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 14] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 15] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments. [Figure 16] 1-6 provide exemplary results of data experiments of the field extraction model described in FIGS. 1-6, according to some embodiments.

[0015] In the figures, elements with the same designation have the same or similar function. DETAILED DESCRIPTION OF THE INVENTION

[0016] Machine learning systems are widely used in computer vision, e.g., pattern recognition, object identification, etc. Several recent machine learning methods formulate form field extraction as field-value pairing or field tagging. For example, some existing systems employ representation learning methods that take field-value candidates as input and apply a high pairing score for positive field-value pairs and a low score for negative pairs using metric learning techniques. Other systems use pre-trained transformers that take both text and its location as input. However, these existing methods generally require a large number of field-level annotations for training. Obtaining field-level annotations for forms is very costly, labor-intensive, and sometimes even impossible because (1) forms typically contain sensitive information, so there is limited public data available for training purposes, and (2) hiring external annotators is also infeasible due to the risk of exposing personal information.

[0017] Considering the need for an efficient system for extracting information from form documents, embodiments describe a field extraction system that does not require field-level annotations for training. Specifically, the training process is bootstrapped by mining pseudo-labels from unlabeled forms using simple rules. A transformer-based structure is then used to model interactions between text tokens in the input form and predict field tags for each token accordingly. The pseudo-labels are used to supervise the transformer training. Because the pseudo-labels are noisy, a refinement module containing a series of branches is used to refine the pseudo-labels. Each refinement branch tags a field and generates a refined label. At each stage, the branch is optimized by ensembling labels from all previous branches to reduce label noise.

[0018] For example, a field extraction system may be trained based on self-supervised pseudo-labels from unlabeled data. Specifically, the field extraction system detects a set of words and their positions within a form and identifies field values ​​based on geometric rules between the words. For example, fields and field values ​​are typically horizontally aligned and separated by colons. The identified field values ​​may then be used as pseudo-labels to train a transformer network that encodes the detected words and positions for classification.

[0019] In some embodiments, multiple pseudo-label ensemble (PLE) branches may be used to refine the pseudo-labels for training. Specifically, the PLE branches are operated in parallel to generate predicted classifications from the coded representations of detected words and positions. For each branch, a loss component is computed by comparing the refined labels for that branch with the predicted labels generated by the "previous" PLE as pseudo-labels. The loss components across the PLE branches are then summed to jointly update the PLE.

[0020] As used herein, the term "network" may include any artificial intelligence network or system, neural network or system, and / or any hardware or software-based framework including any training or learning model implemented therein or with it.

[0021] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0022] FIG. 1 is a schematic diagram 100 illustrating an example of information extraction from an invoice, according to an embodiment described herein. Traditionally, in form processing, a human worker is typically given a list of expected form fields, such as purchase_order, invoice_number, and total_amount, and the goal is to extract corresponding values ​​based on their understanding of the form. Keys, such as INVOICE#, PO Number, and Total, refer to specific textual representations of fields within a form and are important indicators for locating values. Keys are generally the most important feature for locating values. Therefore, field extraction systems aim to automatically extract field values ​​from irrelevant information within a form, which is essential for improving processing efficiency and reducing human effort.

[0023] As shown in FIG. 100, the form contains various phrases, such as "invoice#," "1234," "PO Number," and "000001." The field extraction system may identify "PO Number" 102 as a location identification key and then determine whether any of the values ​​"1234" 104, "00000001" 103, or "100.00" 105 match the location identification key. Such a match may be determined based on a geometric relationship between the location identification key 102 and the values ​​103-105. For example, a rule-based algorithm may be applied to determine that the value "0000001" 103 is likely to correspond to the location of the location identification key 102 because the value 103 has a vertically aligned position with the location identification key 102.

[0024] Unlike traditional methods that access large-scale labeled forms, rule-based methods may be used to generate noisy pseudo-labels (e.g., fields and values) from unlabeled data. Rule-based algorithms are built on the following observations: (1) A field value (e.g., 103 in FIG. 1) is usually shown in a form along with some key (e.g., 102 in FIG. 1), which is a specific textual representation of the field. (2) The keys and their corresponding values ​​have a strong geometric relationship (as shown in Figure 1, the keys are mostly vertically or horizontally next to their values). (3) Form layouts vary greatly, but there are usually some key texts that are frequently used across different form instances (e.g., the key texts for the field purchase_order are "PO Number", "PO#", etc.). (4) Field values ​​are always associated with some date type (for example, the data type of the value of "invoice_date" is date, and the data type of the value of "total_amount" is money amount or number).

[0025] Therefore, a rule-based method may be used to generate useful pseudo-labels for each field of interest from a large form. As shown in Figure 1, key location 102 is first performed based on string matching between the text in the form and the field's possible key string. Values ​​103-105 are then inferred based on the data type of the text and its geometric relationship to the location key 102.

[0026] 2 is a simplified diagram illustrating an overall self-supervised training framework 200 of a field extraction system according to an embodiment described herein. The framework 200 includes an optical character recognition module 205, a transformer network 210, and a classifier 220. Unlabeled forms 202, e.g., checks, invoices, pay slips, etc., are trained using a predefined list {fd1, fd2, ..., fd N}. Given a form as input, a general OCR detection and recognition module 205 is applied to the unlabeled form 202 to find the bounding box {b1, b2, ..., bM}, a set of words with positions represented as {w1, w2, …, w M Therefore, the goal of the field extraction method is to extract the field fd if the field information exists in the input form. i The target value v that matches i Let us consider a large number of word candidates {w1, w2, …, w M} automatically.

[0027] Then, the pairs of words and bounding box positions {w i ,b i} may be input to the transformer encoder 210 and encoded into a feature representation. i ,b i} may also be sent to a pseudo-label inference module 215 configured to perform key location to identify the location of a key corresponding to each predefined field and value inference to determine the corresponding field value of the located key.

[0028] For example, the keys and values ​​may contain multiple words, and the pairs of words and bounding box positions {w i ,b i Upon receiving the}, the pseudo-label inference module 215 uses the DBSCAN algorithm (Ester et al., 1996) to group nearby recognized words based on their positions and generate phrase candidates.

number

number

[0029] Each field of interest fd i Here is a list of frequently used keys for

number

number

number

number

number

[0030] The key is then located by finding the candidate with the largest key score, as follows:

number

[0031] The pseudo label inference module 215 may then determine the value (or one or more values, if applicable) of the location key. Specifically, it infers the value according to the following two criteria: First, the data types must be in line with the field. Second, their positions must be well-matched with the location key. For each field, a list of eligible data types may be pre-determined. For example, for the data field "invoice number", the data types may include string or integer. A pre-trained BERT-based model may be used to predict the data type of each phrase candidate, and candidates with the correct data type may be selected.

number

[0032] In one embodiment, a value score is calculated for each eligible candidate.

number

number

number

number

number

number

number

number

number

number

number

[0033] In one embodiment, the output of the pseudo-label inference module 215, e.g., the field estimates as pseudo-labels, may be used as a standalone field extraction output. In another embodiment, the field estimates may be used as pseudo-labels for bootstrap training to further improve field extraction performance. Specifically, to predict the target label of a word, it is necessary to learn the meaning of this word and its interaction with the surrounding context. Transformer-based architectures (e.g., LayoutLM, as described in Xu et al., 2020) may be used to learn word representations due to their superior ability to model contextual information. In addition to semantic representations, the word position and the general layout of the input form are also important and can be used to capture the discriminative features of words. The Transformer Encoder 210 generates a representation of the input pair {w i ,b i}, features may be extracted.

number

[0034] The classifier 220 for token classification may receive input of the encoded feature representations from the transformer encoder 210, which generates a predicted field containing the background for each token from the original unlabeled form 202. Specifically, the classifier 220 generates a field space ({background, fd1, fd2, ..., fd N}) to obtain the field prediction score s k The predicted field scores from classifier 220 and the pseudo labels generated from pseudo label inference 215 may then be compared in loss module 230 to generate training targets. The training targets may be further utilized to update transformer 210 and classifier 220 via a backpropagation path (shown by the dashed line).

[0035] In one embodiment, multiple Progressive Pseudo-Label Ensembles (PLEs) may be employed for bootstrap training, as further described in Figure 3.

[0036] 3 is a block diagram illustrating an example framework for refining the field extraction framework described in FIG. 2 with PLE, according to embodiments described herein. As depicted in FIG. 2, the transformer 210 receives input 302 of words extracted from unlabeled form 202 and the positions (w1, b1), (w2, b2), ..., (w M ,b M ) and based on that, initial word-level field labels (also known as bootstrap labels)

number

number

[0037] However, using only noisy bootstrap labels as the ground truth for training can compromise the performance of the model. After the transformer 210, we employ a refinement module 304 that includes multiple PLEs, each acting as a classification branch. Specifically, in each branch j, the PLEs independently perform field classification and generate pseudo-labels based on their predictions.

number

[0038] For example, in branch k, the refined label is calculated by: (1) for each word, argmax(sk c ) by predicting field labels

number

number

number

number

number

number

number

number

number

[0039] Therefore, the final loss is the sum of all losses,

number

[0040] In this way, progressive refinement of labels reduces label noise. However, using only refined labels at each stage limits performance improvement because although labels become more accurate after refinement, some unreliable values ​​are filtered out, resulting in a decrease in recall. To mitigate this issue, each branch is enriched with ensemble labels from all previous stages. Ensembled labels not only maintain a better balance between precision and recall, but are also more diverse and can serve as a regularizer for model optimization. During inference, the average predicted score from all branches may be used. A similar procedure may be applied to obtain the final field value when generating refined labels.

[0041] Computer environment 4 is a simplified diagram of a computing device 400 implementing a field extraction framework according to some embodiments described herein. As shown in FIG. 4, computing device 400 includes a processor 410 coupled to a memory 420. The operation of computing device 400 is controlled by processor 410. Also, while computing device 400 is shown with only one processor 410, it is understood that processor 410 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits, graphics processing units (GPUs), etc. within computing device 400. Computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0042] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or computer.

[0043] The processor 410 and / or memory 420 may be located in any suitable physical location. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same board, the same package (e.g., a system-in-package), the same chip (e.g., a system-on-chip), etc. In some embodiments, the processor 410 and / or memory 420 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located in one or more data centers and / or cloud computing facilities.

[0044] In some examples, memory 420 may include a non-transitory, tangible, machine-readable medium containing executable code that, when operated by one or more processors (e.g., processor 410), may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 420 includes instructions for extraction 430, which may be used to implement and / or emulate systems and models and / or to implement any of the methods described further herein. In some examples, field extraction module 430 may receive input 440, such as an image instance of an unlabeled form, via data interface 415. Data interface 415 may be either a user interface that receives an image instance of a form uploaded by a user or a communications interface that may receive or retrieve a previously stored image instance of a form from a database. Field extraction module 430 may generate output 450, such as extracted fields of input 440.

[0045] In some embodiments, the field extraction module 430 may further include a pseudo-label inference module 431 and a PLE module 432. The pseudo-label inference module 431 uses a rule-based method for mining noisy pseudo-labels from forms, for example, as described in FIG. 2. The PLE module 432 (similar to the refinement module 304 of FIG. 3) may use an input of a set of tokens extracted from the form and an output of predicted fields including the background of each token to learn a data-driven model using the field estimates as pseudo-labels during training, implemented as a token classification task. Further details of the PLE module 432 are described below in connection with FIG. 3.

[0046] Field Extraction Workflow FIG. 5 is a simplified diagram of a method 500 for field extraction from a form with unlabeled data via a field extraction model, according to some embodiments. One or more of the processes of method 500 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes. In some embodiments, method 500 corresponds to the operation of field extraction module 430 ( FIG. 4 ) for performing a method of field extraction or training a field extraction model. As illustrated, method 500 includes several enumerated steps, although aspects of method 500 may include additional steps before, after, and between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

[0047] In step 502, an unlabeled form including a plurality of fields and a plurality of field values ​​is received via a data interface (e.g., 415 in FIG. 4). For example, the unlabeled form may take a form similar to that shown in FIGS. 8A-8B.

[0048] In step 504, a set of words and a set of locations are found in unlabeled form for the set of words. For example, the words and locations may be found by OCR module 205 of FIG.

[0049] In step 506, a field value for the field is identified from a set of words and a set of positions based at least in part on a geometric relationship between the set of words. For example, a field value may be identified by applying a first rule that one or more words in the form of a key are related to the field name of the field. As another example, a field value may be identified by applying a second rule that a pair of horizontally or vertically aligned words is a key for the field and field value. As another example, a field value may be identified by applying a third rule that a word from a set of words that matches a predefined key text is a key for the field.

[0050] In one implementation, a key location corresponding to a field is determined. For example, a set of phrase candidates is determined from a set of words, and a corresponding set of phrase locations is determined from the set of locations by grouping nearby recognized words. For each phrase candidate, a key score is computed that indicates the likelihood that the respective phrase candidate is a key for the field. The key score is computed based on the string distance between each phrase candidate and a predefined key (e.g., see Equation (1)). Then, a key is determined for the field based on the maximum key score among the set of phrase candidates (e.g., see Equation (2)).

[0051] Specifically, to compute the key score, a neural model may be used to predict the respective data type for each phrase candidate. Then, a subset of phrase candidates having data types matching the predefined data type of the field is determined. For each phrase candidate in the subset, a value score is computed indicating the likelihood that the respective phrase candidate is a field value of the field. The value score is computed based on the key score of the location key corresponding to the field and a geometric relationship criterion between each phrase candidate and the location key (e.g., Equation (3)). The geometric relationship criterion is computed based on, for example, the string distance and the angle between each phrase candidate and the location key (e.g., Equation (4)). Then, the field value is determined based on the maximum score among the subset of phrase candidates.

[0052] In step 508, an encoder (eg, transformer encoder 210 of FIG. 2) may encode a first word and first position pair corresponding to the field value into a first representation.

[0053] In step 510, a classifier (eg, classifier 220 of FIG. 2) may generate a field classification distribution from the first representation.

[0054] In step 512, a first loss target is computed by comparing the field classification distribution with the field values ​​as pseudo labels.

[0055] In step 514, the encoder is updated based on the first loss target via backpropagation.

[0056] FIG. 6 is a schematic diagram of a method 600 for label refinement in field extraction from forms with unlabeled data via a field extraction model, according to some embodiments. One or more of the processes of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes. In some embodiments, method 600 corresponds to the operation of field extraction module 430 ( FIG. 4 ) to perform a method for training a field extraction or field extraction model. As illustrated, method 600 includes several enumerated steps, although aspects of method 600 may include additional steps before, after, and between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

[0057] In step 602, an unlabeled form including a plurality of fields and a plurality of field values ​​is received via a data interface (e.g., 415 in FIG. 4). For example, the unlabeled form may take a form similar to that shown in FIGS. 8A-8B.

[0058] A first word and a first location of the first word are detected in the unlabeled form in step 604. For example, the word and location may be detected by the OCR module 205 of FIG.

[0059] In step 606, an encoder (eg, transformer encoder 210 of FIG. 2) encodes the first word and first position pair into a first representation (eg, equation (6)).

[0060] In step 608, multiple progressive label ensemble (PLE) branches (e.g., see 304 a-n in FIG. 3 ) generate multiple predicted labels in parallel based on the first representation. Each of the multiple PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. The predicted label in one PLE branch is generated by projecting the first representation onto a set of field prediction scores through one or more fully connected layers and generating a predicted score based on the maximum field prediction score among the set of words. When the maximum field prediction score is greater than a predetermined threshold, the word corresponding to the maximum field prediction score from the set of words is selected for a field from the multiple fields.

[0061] In step 610, the loss component in one PLE branch is computed by comparing the predicted label in one PLE branch with the predicted label from the previous PLE branch as a pseudo label.

[0062] In step 612, a loss target is computed as the sum of loss components across multiple PLE branches (eg, equation (7)).

[0063] In step 614, the multiple PLE branches are updated based on the first loss target via backpropagation. In one embodiment, the first PLE branch from the multiple PLE branches uses the identified field value of the field as the first pseudo label. The combined loss target is computed by adding the first loss target computed in step 512 of FIG. 5 and the loss target. The encoder and the multiple PLE branches are then jointly updated based on the combined loss target.

[0064] Exemplary Performance An exemplary training dataset may include actual invoices collected from various vendors. For example, the training set includes 7,664 unlabeled invoice forms across 2,711 templates. For example, the validation set includes 348 labeled invoices across 222 templates. The test set includes 339 labeled invoices across 222 templates. Each template has up to five images in each set. Seven frequently used fields are considered, including invoice_number, purchase_order, invoice_date, due_date, amount_due, total_amount, and total_tax.

[0065] For the Tobacco test set, 350 invoices are collected from the publicly available Tobacco Collections of Industry Documents Library 2. While the validation and test sets of the internal IN-Invoice dataset have similar statistical distributions of fields, the publicly available Tobacco test set differs. For example, invoices in the Tobacco set (shown in Figure 8A) may have lower resolution and cluttered backgrounds compared to other invoices in the training dataset (shown in Figure 8B).

[0066] The end-to-end macro-averaged F1 score on the fields is used as the metric for evaluating the models. Specifically, we count true positives, false positives, and false negatives using exact string matching between predicted and ground truth values. For each field, precision, recall, and F1 score are obtained. The reported scores are averaged over five runs to reduce the impact of randomness.

[0067] Since there are no existing methods for performing field extraction using only unlabeled data, the following baselines are constructed to validate our method: Bootstrap Labels (B-Labels): Using initial pseudo-labels inferred using the proposed simple rules, field extraction can be performed directly without training data. Transformer Training with B-Labels: Because the Transformer is used as the backbone for extracting word features, the Transformer model is trained using B-Labels as a baseline for evaluating (1) the data-driven model of the pipeline and (2) the performance gain from the refinement module. Both the content of the text and its location are important for field prediction. An example of a Transformer backbone is LayoutLM, which takes both text and location as input. In addition, two popular Transformer models, BERT and RoBERTa, which take only text as input, are used.

[0068] An OCR engine is used to detect words and their locations and rank them in reading order. Exemplary key lists and date types for each dataset are shown in Table 1 of Figure 7. The key lists and data types are quite broad. α is set to 4.0 in Equation (4). To further eliminate false positives, a value candidate is removed if the location key is not within the adjacent zone. Specifically, the adjacent zone around the value candidate extends to the left side of the image, with four candidate heights above it and one candidate height below it. The number of refinement branches k = 3 in all experiments. When the number of stages is greater than 1, one hidden FC layer is added, 768 units deep, before classification. β in Equation (7) is set to 1.0 for all invoice experiments, except for β = 5.0 for the BERT-based refinement in Table 4 of Figure 11, due to its good performance on the validation set. For both the field extraction model described herein and the baseline, the model with the best F1 score is selected on the validation set. To prevent overfitting, a two-stage training strategy is adopted: the pseudo-labels are used to train the first branch of the model, which is then fixed with the feature extractor during refinement. The batch size is set to 8, and the learning rate is 5e. 5 Use the Adam optimizer.

[0069] The proposed model was then validated using the IN-Invoice dataset, which contains large-scale unlabeled training data and sufficient validation / test data, which better fits the experimental setting. The proposed training method was first validated using LayoutLM as the backbone. The comparison results are shown in Table 2 in Figure 9 and Table 3 in Figure 10. The bootstrap label (B-Labels) baseline achieved F1 scores of 43.8% and 44.1% on the validation and test sets, respectively, indicating that B-Labels has reasonable accuracy rates but is still noisy. When training the LayoutLM transformer using B-Labels, significant performance improvements of approximately 15% on the validation set and approximately 17% on the test set are obtained. Adding the PLE refinement module significantly improves model precision by approximately 6% on the validation set and approximately 7% on the test set, but slightly decreases recall by approximately 2.5% on the validation set and approximately 3% on the test set. This suggests that refined labels become increasingly reliable in later stages, leading to higher model precision. However, the refinement stage also removes low-confidence false negatives, resulting in a decrease in recall. Overall, the PLE refinement module further improves performance, resulting in a 3% gain in F1 score.

[0070] Because both the text and its position are important for the task, LayoutLM is used as the default feature backbone. Furthermore, to understand the impact of different Transformer models as backbones, we evaluated two additional models, BERT and RoBERTa, which use only text as input. The comparative results are shown in Table 4 of Figure 11 and Table 5 of Figure 12. We observed that directly training BERT and RoBERTa using the B-label and PLE refinement modules consistently improves the baseline results for various Transformer choices with different numbers of parameters (base or large). However, LayoutLM yields much higher results compared to the other two backbones, indicating that the position of the text is crucial for achieving good performance on the task.

[0071] The proposed model was then tested using the introduced Tobacco test set (Table 6 in Figure 13). The simple rule-based method obtained an F1 score of 25.1%, which is reasonable but much lower than the results for the internal IN-Invoice dataset. This is because the Tobacco test set is visually noisy, resulting in more text recognition errors. The LayoutLM baseline is significantly improved when using B-labels. Furthermore, the PLE refinement module further improves the F1 score by approximately 2%. The results suggest that the proposed method is well suited to diverse scenarios. Figures 8A-8B show that the proposed method achieves good performance despite the sample invoices being highly diverse across different templates, cluttered backgrounds, and low resolution.

[0072] Ablation studies are further conducted on the invoice dataset with the LayoutLM-based backbone. Impact of the number of stages: The proposed model is refined in k stages, with k=3 fixed in all experiments. The number of stages is varied and evaluated. Figure 15 shows that as the number of stages k increases, the model generally performs better on both the validation and test sets. The performance of using multiple stages is always higher than that of the single-stage model (Transformer baseline). The model performance reaches its best when k=3. As shown in Figure 16, during model refinement, precision improves but recall decreases. The best balance between precision and recall is obtained when k=3. When k>3, recall decreases more than the improvement in precision, and a deterioration in F1 score is observed.

[0073] Effect of refined labels (R-labels): To analyze the effect of this design, all refined labels are removed in the final loss, and the three branches are trained independently using only B-labels, and the predictions are ensembled during inference. As shown in Table 7 of Figure 14, removing refined labels reduces the F1 scores on the validation and test sets by 2.2% and 2.6%, respectively.

[0074] Effect of regularization using B-labels. At each stage, B-labels are used as a type of regularization to prevent the model from overfitting to the overfitted refined labels. Utilizing B-labels in the refinement stage by setting β = 0 in equation (7). As shown in Table 7 of Figure 14, the model performance drops by about 2% in F1 score without this regularization.

[0075] Effect of the two-stage training strategy: To avoid overfitting to noisy labels, we adopt a two-stage training strategy, where the first branch is trained using the B-label and is fixed during refinement. This effect is analyzed by training the model in a single step. As shown in Table 7 of Figure 14, single-step training leads to a decrease in F1 score of 1.8% and 1.4% on the validation and test sets, respectively.

[0076] Some examples of computing devices, such as computing device 400, may include non-transitory, tangible, machine-readable media that includes executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform the processes of method 400. Some common forms of machine-readable media that may include the processes of method 400 are, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media adapted to be read by a processor or computer.

[0077] This description and the accompanying drawings, which illustrate aspects, embodiments, implementations, or applications of the invention, should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Like numbers in two or more figures represent the same or similar elements.

[0078] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous details are set forth to provide a thorough understanding of the embodiments. It will be apparent to one of ordinary skill in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, but not limiting. Those skilled in the art may recognize other elements not specifically described herein that are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless otherwise specifically described or unless one or more features render the embodiment non-functional.

[0079] This application is further described with reference to the attached document at page 9 of Appendix I entitled "Field Extraction from Forms with Unlabeled Data," which is considered part of this disclosure and is incorporated by reference in its entirety.

[0080] While exemplary embodiments have been shown and described, a wide range of modifications, variations, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention is to be limited only by the claims that follow, and it is appropriate that such claims be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

[0081] This application also discloses the following: [Appendix 1] 1. A method for field extraction from a form with unlabeled data via a field extraction model, comprising: receiving, via a data interface, an unlabeled form including a plurality of fields and a plurality of field values; detecting, by a processor, a set of words and a set of locations for said set of words in an unlabeled form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; encoding, by an encoder, a first word and first position pair corresponding to the field value into a first representation; generating, by a classifier, a field classification distribution from the first representation; Computing a first loss target by comparing the field classification distribution with field values ​​as pseudo labels; and updating the encoder based on the first loss target via backpropagation. [Appendix 2] 2. The method of claim 1, wherein identifying the field value of the field includes applying a first rule that one or more words in a form of a key are related to a field name of the field. [Appendix 3] 3. The method of claim 2, wherein identifying the field value of the field includes applying a second rule that a pair of horizontally or vertically aligned words is a key to the field and the field value. [Appendix 4] 4. The method of claim 3, wherein identifying the field value of the field includes applying a third rule that a word from the set of words that matches a predefined key text is a key for the field. [Appendix 5] determining a set of phrase candidates from the set of words and a set of corresponding phrase positions from the set of positions by grouping nearby recognized words; computing, for each phrase candidate, a key score indicating the likelihood that the respective phrase candidate is a key for the field; 2. The method of claim 1, further comprising: determining the key for the field based on a maximum key score among the set of phrase candidates. [Appendix 6] 6. The method of claim 5, wherein the key score is computed based on a string distance between each of the phrase candidates and a predefined key. [Appendix 7] predicting a respective data type for each phrase candidate via a neural model; determining a subset of phrase candidates having data types that match a predefined data type of the field; For each phrase candidate in the subset, computing a value score indicating the likelihood that the respective phrase candidate is the field value of the field; 6. The method of claim 5, further comprising determining the field value based on a maximum key score among the subset of phrase candidates. [Appendix 8] 8. The method of claim 7, wherein the value score is computed based on a key score of a location key corresponding to the field and a geometric relationship criterion between the respective phrase candidate and the location key. [Appendix 9] 9. The method of claim 8, wherein the geometric relationship criterion is computed based on a string distance and an angle between each phrase candidate and the location key. [Appendix 10] generating, by a plurality of progressive label ensemble (PLE) branches, a plurality of predicted labels based on the first representation, respectively; computing a loss component in one PLE branch by comparing the predicted label in the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; 2. The method of claim 1, wherein a first PLE branch from the plurality of PLE branches receives the identified field value of the field as a first pseudo label. [Appendix 11] 1. A system for field extraction from forms with unlabeled data via a field extraction model, comprising: a data interface for receiving an unlabeled form including a plurality of fields and a plurality of field values; a memory for storing a plurality of processor-executable instructions; a processor that executes the processor-executable instructions to perform operations, the operations comprising: Detecting a set of words and a set of locations for the set of words in an unlabeled form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; encoding, by an encoder, a first word and first position pair corresponding to the field value into a first representation; generating, by a classifier, a field classification distribution from the first representation; Computing a first loss target by comparing the field classification distribution with field values ​​as pseudo labels; and updating the encoder based on the first loss target via backpropagation. [Appendix 12] 12. The system of claim 11, wherein identifying the field value of the field includes applying a first rule that one or more words in a form of a key are related to a field name of the field. [Appendix 13] 13. The system of claim 12, wherein identifying the field value for the field includes applying a second rule that a pair of horizontally or vertically aligned words is a key to the field and the field value. [Appendix 14] 14. The system of claim 13, wherein identifying the field value of the field includes applying a third rule that a word from the set of words that matches a predefined key text is a key for the field. [Appendix 15] The operation is determining a set of phrase candidates from the set of words and a set of corresponding phrase positions from the set of positions by grouping nearby recognized words; computing, for each phrase candidate, a key score indicating the likelihood that the respective phrase candidate is a key for the field; 12. The system of claim 11, further comprising: determining the key for the field based on a maximum key score among the set of phrase candidates. [Appendix 16] 16. The system of claim 15, wherein the key score is computed based on a string distance between each phrase candidate and a predefined key. [Appendix 17] The operation is predicting a respective data type for each phrase candidate via a neural model; determining a subset of phrase candidates having data types that match a predefined data type of the field; computing, for each phrase candidate in the subset, a value score indicating the likelihood that the respective phrase candidate is the field value of the field; 16. The system of claim 15, further comprising: determining the field value based on a maximum key score among the subset of phrase candidates. [Appendix 18] 18. The system of claim 17, wherein the value score is computed based on a key score of a location key corresponding to the field and a geometric relationship criterion between the respective phrase candidate and the location key. [Appendix 19] 19. The system of claim 18, wherein the geometric relationship criterion is computed based on a string distance and an angle between the respective phrase candidate and the location key. [Appendix 20] The operation is generating a plurality of predicted labels in parallel, respectively, based on the first representation by a plurality of progressive label ensemble (PLE) branches; computing a loss component in one PLE branch by comparing the predicted label in the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; 2. The method of claim 1, wherein a first PLE branch from the plurality of PLE branches receives the identified field value of the field as a first pseudo label. [Appendix 21] 1. A method for field extraction from a form with unlabeled data via a field extraction model, comprising: receiving, via a data interface, an unlabeled form including a plurality of fields and a plurality of field values; Detecting, by a processor, a first word and a first location of the first word in the unlabeled form; encoding, by an encoder, the pair of the first word and the first location into a first representation; generating a plurality of predicted labels in parallel, respectively, based on the first representation by a plurality of progressive label ensemble (PLE) branches; computing a loss component in one PLE branch by comparing the predicted label in the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; computing a loss target as a sum of loss components across the plurality of PLE branches; updating the plurality of PLE branches via backpropagation based on the loss target. [Appendix 22] 22. The method of claim 21, wherein each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. [Appendix 23] The predicted label in the one PLE branch is projecting the first representation through one or more fully connected layers into a set of field prediction scores; and generating the prediction score based on a maximum field prediction score among a set of words. [Appendix 24] 24. The method of claim 23, further comprising, for a field among the plurality of fields, selecting a word from the set of words corresponding to the largest field prediction score when the largest field prediction score is greater than a predefined threshold. [Appendix 25] detecting, by a processor, a set of words and a set of locations for said set of words in an unlabeled form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; generating, by a classifier, a field classification distribution from the first representation; 22. The method of claim 21, further comprising computing a first loss target by comparing the field classification distribution to the field values ​​as pseudo labels. [Appendix 26] 26. The method of claim 25, wherein a first PLE branch from the plurality of PLE branches uses the identified field value of the field as a first pseudo label. [Appendix 27] computing a combined loss target by adding the loss target and the first loss target; 26. The method of claim 25, further comprising: jointly updating the encoder and the multiple PLE branches via backpropagation based on the joint loss target. [Appendix 28] and updating the encoder based on the first loss target via backpropagation. [Appendix 29] 29. The method of claim 28, further comprising, after updating the encoder, updating the plurality of PLE branches via backpropagation based on the loss target while fixing parameters of the encoder. [Appendix 30] 1. A system for field extraction from forms with unlabeled data via a field extraction model, comprising: a data interface for receiving an unlabeled form including a plurality of fields and a plurality of field values; a memory for storing a plurality of processor-executable instructions; a processor that executes the processor-executable instructions to perform operations, the operations comprising: Detecting a first word and a first location of the first word in an unlabeled form; encoding, by an encoder, the pair of the first word and the first location into a first representation; generating a plurality of predicted labels in parallel, respectively, based on the first representation by a plurality of progressive label ensemble (PLE) branches; Computing a loss component in one PLE branch by comparing the predicted label in the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; Computing a loss target as a sum of loss components across multiple PLE branches; and updating the plurality of PLE branches via backpropagation based on the loss target. [Appendix 31] 31. The system of claim 30, wherein each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. [Appendix 32] The predicted label in the one PLE branch is projecting the first representation through one or more fully connected layers into a set of field prediction scores; and generating the predicted label based on a maximum field prediction score among a set of words. [Appendix 33] The operation is 33. The system of claim 32, further comprising: for a field among the plurality of fields, selecting a word from the set of words corresponding to the largest field prediction score when the largest field prediction score is greater than a predefined threshold. [Appendix 34] The operation is detecting, by a processor, a set of words and a set of locations for said set of words in an unlabeled form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; generating, by a classifier, a field classification distribution from the first representation; 31. The system of claim 30, further comprising: computing a first loss target by comparing the field classification distribution to field values ​​as pseudo labels. [Appendix 35] 35. The system of claim 34, wherein a first PLE branch from the plurality of PLE branches uses the identified field value of the field as a first pseudo label. [Appendix 36] The operation is computing a combined loss target by adding the loss target and the first loss target; 35. The system of claim 34, further comprising: jointly updating the encoder and the plurality of PLE branches via backpropagation based on the joint loss target. [Appendix 37] The operation is and updating the encoder based on the first loss target via backpropagation. [Appendix 38] The operation is 38. The system of claim 37, further comprising, after updating the encoder, updating the plurality of PLE branches via backpropagation based on the loss target while fixing parameters of the encoder. [Appendix 39] 1. A non-transitory storage processor-readable medium storing processor-executable instructions for field extraction from a form having unlabeled data via a field extraction model, the instructions being executed by a processor to perform operations, the operations including: receiving, via a data interface, an unlabeled form including a plurality of fields and a plurality of field values; locating, by a processor, a first word and a first location of the first word in the unlabeled form; encoding, by an encoder, the pair of the first word and the first location into a first representation; generating a plurality of predicted labels in parallel, respectively, based on the first representation by a plurality of progressive label ensemble (PLE) branches; Computing a loss component in one PLE branch by comparing the predicted label in one PLE branch with the predicted label from the previous PLE branch as a pseudo label; Computing a loss target as a sum of loss components across multiple PLE branches; and updating the plurality of PLE branches via backpropagation based on the loss target. [Appendix 40] each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation; The predicted label in the one PLE branch is projecting the first representation through one or more fully connected layers into a set of field prediction scores; and generating the prediction score based on a maximum field prediction score among a set of words.

Claims

1. 1. A computer vision method for field extraction from images of unlabeled forms via a field extraction model, comprising: receiving, via a data interface, an image of an unlabeled form including a plurality of fields and a plurality of field values; Detecting a first word and a first location of the first word in the image of the unlabeled form with an optical character recognition (OCR) engine; encoding, by an encoder implemented on one or more processors, the first word and first location pair into a first representation; generating, by a plurality of Progressive Label Ensemble (PLE) models arranged in parallel and connected to the encoder, a plurality of predicted labels based at least in part on the first representation, respectively, wherein a first predicted label is generated by a first PLE model of the plurality of PLE models from a second predicted label generated by a second PLE model of the plurality of PLE models; Computing, in the first PLE model, a corresponding loss component by comparing a first field classification score generated by the first PLE model with a subset of predicted labels as pseudo labels generated from a subset of the plurality of PLE models other than the first PLE model; training the plurality of PLE models using a loss target as a sum of loss components across the plurality of PLE models; and predicting, by the field extraction model comprising the trained plurality of PLE models, keys and corresponding field values ​​for the unlabeled form image based on an average of field classification scores produced by the trained plurality of PLE models.

2. The method of claim 1 , wherein each of the plurality of PLE models includes a respective classifier that generates a respective predicted label based on the first representation.

3. The first predicted label is projecting the first representation into a set of field prediction scores via one or more fully connected layers; and generating the first predicted label based on a maximum field prediction score among a set of words.

4. 4. The method of claim 3, further comprising: for a field from the plurality of fields, when the maximum field prediction score is greater than a predefined threshold, selecting a word from the set of words corresponding to the maximum field prediction score.

5. detecting, by a processor, a set of words and a set of locations within the image of the unlabeled form for the set of words; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; generating, by a classifier, a field classification distribution from the first representation; The method of claim 1 , further comprising: computing a first loss target by comparing the field classification distribution to the field values ​​as pseudo labels.

6. The method of claim 5 , wherein the first PLE model from the plurality of PLE models uses the identified field value of the field as a first pseudo label.

7. computing a combined loss target by adding the loss target and the first loss target; The method of claim 5 , further comprising: jointly updating the encoder and the plurality of PLE models via backpropagation based on the joint loss target.

8. The method of claim 5 , further comprising updating the encoder based on the first loss target via backpropagation.

9. The method of claim 8 , further comprising, after updating the encoder, updating the plurality of PLE models via backpropagation based on the loss target while keeping parameters of the encoder fixed.

10. 1. A system for field extraction from images of unlabeled forms via a field extraction model, comprising: a data interface for receiving an image of an unlabeled form including a plurality of fields and a plurality of field values; a memory for storing a plurality of processor-executable instructions; a processor that executes the processor-executable instructions to perform operations, the operations comprising: Detecting a first word and a first location of the first word in the image of the unlabeled form with an optical character recognition (OCR) engine; encoding, by an encoder, the pair of the first word and the first position into a first representation; generating, by a plurality of Progressive Label Ensemble (PLE) models arranged in parallel and connected to the encoder, a plurality of predicted labels based at least in part on the first representation, respectively, wherein a first predicted label is generated by a first PLE model of the plurality of PLE models from a second predicted label generated by a second PLE model of the plurality of PLE models; Computing, in the first PLE model, a corresponding loss component by comparing a first field classification score generated by the first PLE model with a subset of predicted labels as pseudo labels generated from a subset of the plurality of PLE models other than the first PLE model; training the plurality of PLE models using a loss target as a sum of loss components across the plurality of PLE models; predicting, by the field extraction model including the trained plurality of PLE models, keys and corresponding field values ​​for the unlabeled form image based on an average of field classification scores produced by the trained plurality of PLE models.

Images