Virtual confrontation training of document information extraction model

Abstract

The present disclosure relates to computer-implemented methods, software, and systems for extracting information from documents based on training techniques to generate a document base model for initializing a document information extraction model that is trimmed to business document details. A document information extraction model is initialized based on weights provided from a first pre-trained model. Fine tuning of the document information extraction model is performed based on the business document as a marker of the second training data. The tagged business document is tagged and evaluated according to virtual adversarial training (VAT). Based on the performed trim, a classifier for classification of the information extraction is generated.

Description

Technical Field

[0001] The present disclosure relates to computer-implemented methods, software, and systems for data processing. Background Art

[0002] Information extraction methods can be used in different natural language processing based applications. These methods can be used to identify information and provide natural language understanding. Information extraction models can be learned through a combination of pre-training, fine-tuning, and contextual learning. Pre-training is the initial stage of model learning. During pre-training, the model is exposed to a large amount of unlabeled text data, where pre-training aims to capture the underlying patterns, structures, and semantic knowledge present in the text corpus. Fine-tuning is the process of further training the model on a specific task or in a specific domain. Fine-tuning can improve the performance of a model pre-trained on a specific task or domain to better fit the data. Summary of the invention

[0003] Embodiments of the present disclosure are generally directed to a computer-implemented method for performing selection of a predictive model from a set of candidate predictive models based on an assessment of risk of bias.

[0004] In one aspect, an example method may include operations such as: obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction; pre-training the first model based on a dynamic window adjustable for word token counts for each of the plurality of unlabeled documents, wherein the pre-training includes: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate corresponding information relevance of the word tokens; and providing the pre-trained first model for initializing a second document information extraction model to be trained based on labeled documents as second training data.

[0005] In a second aspect, an example method may include operations such as: pre-training a domain-specific neural network extraction model based on first training data, wherein the training data includes unlabeled documents, wherein the pre-training is performed based on informative word masking; performing fine-tuning of the pre-trained domain-specific neural network extraction model based on labeled business documents as second training data, wherein the labeled business documents are processed according to a virtual adversarial training (VAT) technique; and generating a classifier associated with the domain-specific neural network extraction model based on the performed fine-tuning.

[0006] In a third aspect, an example method may include operations such as: initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data; performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and generating a classifier for classification for information extraction based on the performed fine-tuning.

[0007] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the methods provided herein.

[0008] The present disclosure also provides a system for implementing the method provided herein. The system includes one or more processors, and a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to the implementation of the method provided herein.

[0009] It should be understood that the method according to the present disclosure may include any combination of aspects and features described herein. That is, the method according to the present disclosure is not limited to the combination of aspects and features specifically described herein, but also includes any combination of aspects and features provided.

[0010] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram depicting an example computer-implemented system that may perform embodiments of the present disclosure in accordance with embodiments of the present disclosure.

[0012] Figure 2 is a block diagram depicting an example of a business document and extracted information from the business document including header information and line item information.

[0013] Figure 3 is a block diagram presenting an example of a process for pre-training and fine-tuning a model for extracting information from documents according to an embodiment of the present disclosure.

[0014] Figure 4 is a flow chart representing a process for training and executing a model to extract information from a document according to an embodiment of the present disclosure.

[0015] Figure 5 is a flowchart showing a process of fine-tuning a training model to extract information from a document based on a virtual adversarial technique according to an embodiment of the present disclosure.

[0016] Figure 6 is a block diagram representing steps of embedding applied in virtual adversarial training for training a model for extracting information from business documents according to an embodiment of the present disclosure.

[0017] Figure 7 is a block diagram representing a process of performing virtual adversarial training in the context of business document information extraction according to an embodiment of the present disclosure.

[0018] Figure 8 is a schematic diagram of an example computer system that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] This disclosure describes various tools and techniques for extracting information from business documents based on training techniques to generate a document-based model customized to the specifics of the business documents through pre-training and fine-tuning.

[0020] In some instances, a web application or service may provide a service for extracting information of logical entities identified in a scanned document (such as a business document). Logical entities in a business document may include header information and / or line item information. For example, header information may include data such as document date, name (e.g., sender name), sender bank account, sender tax ID, price amount or total amount, and other entity-related data. Line item information may be data provided in a table in a document and included in a field. Line item information may include line item text (e.g., description of a product, product name, etc.), line item quantity (e.g., unit of product), and / or line item quantity (e.g., price, number of products sold, etc.).

[0021] A business document may be a document associated with a particular business task, such as an order, purchase, request, or other event or action. For example, information may be extracted from a business document, such as an invoice, purchase order, order confirmation, or payment proposal. In some instances, the extraction of information may be based on logic at a trained model that, when provided with a scanned business document, may automatically classify identified data from the document.

[0022] In some embodiments, a machine learning model can be trained to automatically extract information from business documents and provide the extracted information to an end user or other software application or service (e.g., as input for initiating another process or task). For example, if an invoice is provided in a scanned format for use in preparing a tax return (e.g., by another application service or process), a document information extraction model can be trained to process the invoice based on the model and provide output from the processing to trigger a tax return generation process. Different configurations for using the trained model can be configured in the software system and / or application to provide documents as input and obtain data as output, which can be further used as input in other processes (e.g., triggering the initiation of a process at another system based on the accumulated data from the processed business documents).

[0023] In some embodiments, a service that provides information extraction capabilities based on a training model can be integrated into a software solution to convert business documents into database entries. In some embodiments, the creation of such a database entry can trigger a business process such as payment or other transaction execution.

[0024] Training a model for document information extraction can be associated with processing a large number of documents that may have to be manually annotated for training. In some instances, document annotation is a labor-intensive process that may be associated with significant resource and time requirements, and the results may be error-prone. In some instances, annotated datasets may not be available for every type of business document. Therefore, providing a trained model that can effectively process domain-specific scanned documents (e.g., business documents) can provide advantages for processing documents in a faster and reliable manner to obtain accurate results.

[0025] In some instances, a solution for training a model is provided according to an embodiment of the present disclosure, which relies on a pre-training technique applied to unlabeled documents, which is also fine-tuned using a domain-specific and annotated smaller training set. The pre-training technique may include training based on a masking technique that masks words based on a probability rate determined for a word token, which is based on the relevance of the word token to the text in the document. In this way, these techniques allow training to focus on words with higher relevance to the text in the document, and the training process is optimized. Pre-training can be performed based on a dynamic window for evaluating the text in the document, so that the evaluation of the document can be adjusted to the size of each document, rather than fixed. By utilizing a dynamic window, the word distribution of the language corpus of the document is modeled, and masking during training can be performed based on the determined informativeness of the words in the document, treating each document as a whole. The trained model can be fine-tuned by applying virtual adversarial training, in which adversarial perturbations are applied to the concatenated text and spatial embeddings. Perturbations can be added to the input training data (e.g., as small noise) to be processed by the model, and used to train the model to still correctly recognize and classify the input data.

[0026] Figure 1 1 is a block diagram depicting an example computer-implemented system 100 that can perform embodiments of the present disclosure in accordance with embodiments of the present disclosure. In the depicted example, the example system 100 includes a client device 102, a client device 104, a network 110, a cloud environment 106, and a cloud environment 108. The cloud environment 106 may include one or more server devices and databases (e.g., processors, memory). In the depicted example, a user 114 interacts with the client device 102, and a user 116 interacts with the client device 104.

[0027] In some examples, client device 102 and / or client device 104 can communicate with cloud environment 106 and / or cloud environment 108 via network 110. Client device 102 may include any suitable type of computing device, such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular phone, a network device, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of any two or more of these devices or other data processing devices. In some implementations, network 110 may include a large computer network that connects any number of communication devices, mobile computing devices, fixed computing devices, and server systems, such as a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., a public switched telephone network (PSTN)), or a suitable combination thereof.

[0028] In some implementations, the cloud environment 106 includes at least one server and at least one data store 120. Figure 1 In the example of , cloud environment 106 is intended to represent various forms of servers, including but not limited to web servers, application servers, proxy servers, network servers and / or server pools. Typically, the server system accepts requests for application services and provides such services to any number of client devices (e.g., client devices 102 on network 110).

[0029] According to embodiments of the present disclosure, and as described above, cloud environment 106 can host applications and databases running on a host infrastructure. In some instances, cloud environment 106 can include multiple cluster nodes, which can represent physical or virtual machines, which can represent a web node cluster. Hosted applications and / or services can run on VMs hosted on cloud infrastructure.

[0030] In some examples, cloud environment 106 and / or cloud environment 108 may provide infrastructure for performing pre-training and adversarial training to classify documents, such as with respect to Figure 2 , 3 , 4 and 5 further described.

[0031] In some examples, cloud environment 106 and / or cloud environment 108 may provide infrastructure to perform training of a document information extraction model according to embodiments of the present disclosure.

[0032] Figure 2 is a block diagram depicting an example of a business document 200 and corresponding extracted information therefrom, where the extracted information includes header information and line item information.

[0033] Business document 200 is an invoice, which includes information about purchasing a certain amount of product "apple" at a certain price, here, 1400 pieces for a total price of $1,684. Figure 2 Other items of the invoice in the example document 200 are omitted in the representation. When such a document is provided as a scanned document, the text in the document can be recognized by using an optical character recognition (OCR) application to obtain the text on the document and two-dimensional spatial information identifying the position of the extracted text on the document.

[0034] exist Figure 2In the example of , document 200 is a business document including logical entities, including title information and line item information. In some instances, such information can be extracted from the document based on a training model according to an embodiment of the present disclosure (e.g., a neural network model trained using pre-training techniques and virtual adversarial training). The extracted information can be used to create an entry in a database storage, or can be provided as input to be stored in another application or system that can store data according to a different data model.

[0035] Logical entities in a business document may include header information 210 and line item information 220. The header information 210 includes information about the sender name, supplier ID, tax ID, invoice number, currency of the invoice, total amount of the invoice, and taxes applied. This information is located at word blocks on the business document and is spatially distant from each other. The line item information 220 of a business document is information located in a row (single line) on the document and includes information about the subject of the invoice, here the purchase of apples. The line item information 220 can be extracted from multiple fields on the same row identified at the invoice. In the example invoice 200, this is information about the quantity of apples purchased at a specific price.

[0036] Figure 3 is a block diagram presenting an example of a process for pre-training 300 and fine-tuning 340 a model for extracting information from business documents according to an embodiment of the present disclosure.

[0037] In some instances, a model can be trained to classify data extracted from business documents by first performing pre-training based on unlabeled training data (i.e., business documents) to learn the structure of the data, wherein the pre-trained model can be fine-tuned using self-supervised learning techniques based on the training data, which is labeled and relies on representations of the documents determined based on the pre-training.

[0038] During the pre-training phase (300), the model is trained on unlabeled business documents 310 as training data. The business documents can be processed for text recognition as they can be scanned documents. Text recognition can be used to extract text from each document, which is then used during training. Based on the training, representations 320 are learned by solving a pretext task 330 and by applying masked language modeling (MLM). The pre-training phase can be performed based on MLM techniques to achieve bidirectional learning from text from training data by masking or hiding words or other defined parts of text in the document, and predicting the masked words using other words on either side of the masked words (bidirectionally). The MLM technique can be applied so that a certain percentage of words in the document (which can be defined as text embeddings) can be masked, and the model can predict those masked words based on the context of the business document.

[0039] In some examples, the pretext task 330 can be defined as a loss minimization problem for learning a pre-trained model, where the problem is parameterized by θ based on considering training business documents presented as sequences of word tokens and spatial tokens. M ' and word token Y L 'With a sequence of masked working tokens y and masked positions M'. The model can be presented as follows and can be defined as minimizing the cross entropy loss function, as presented below at formula (1):

[0040]

[0041] However, randomly masking tokens during pre-training as in MLM may have certain disadvantages. For example, some word tokens may be too easy to predict if masked; therefore, including those as masking elements may lead to inefficient loss determination during pre-training. Therefore, masking all words at the same rate without considering their informativeness in the context of the text (i.e., the content of the scanned document) may reduce the usefulness of the final model's extraction accuracy. Furthermore, if masking is applied randomly to a set of consecutive words, training may focus on the interpretation of the word based only on the surrounding words, and ignore the entire sequence representation.

[0042] In some implementations, to address the limitation of randomly masking tokens during pre-training, the informativeness of words can be determined, and masking can be applied based on individual masking ratios corresponding to a metric that defines the relevance of a given word. Thus, word tokens with higher masking ratios can be masked more frequently during training than words with lower masking ratios. Figure 4 The pre-training process is described in further detail in , which includes taking into account the informativeness of words and the entire size of the document as a sequence of tokens.

[0043] During fine-tuning 340, the representation 320 of the pre-trained model is used to initialize the representation 350 to be used for fine-tuning based on training using labeled documents as training data. By initializing the representation 350 with the representation 320 at the fine-tuning 340 step, the informative identification of words within the document determined during pre-training can be transferred to the fine-tuning stage, thereby supporting improved accuracy, while also reducing resource expenditure by reducing training time and computing resources used to perform data processing. In addition, the representation 350 is initialized based on the structure identified for the document at the pre-training step 300 to maintain the architecture of the training execution to match the structure of the training, thereby generating a representation of the model - here, the representations 320 and 350. Pre-training and fine-tuning as described in the present disclosure can include substantially similar training steps with respect to the encoding step, but can differ in decoding. In some instances, during pre-training, words are classified based on all words in the vocabulary, while fine-tuning classifies words based on predefined labels because the input is labeled (e.g., the label of the sender of the invoice document is used as input). The input documents for training at the fine-tuning 340 stage are annotated or otherwise labeled, and the representation 350 is trained to generate a fine-tuned representation after being initialized according to the virtual adversarial technique. The fine-tuned representation can then be used to generate a classifier 360. Pre-training 300 allows the representation 320 to be generated and input into the training at the fine-tuning step 340, where the learning process is faster and produces accurate results for classification in a specific field of business documents because the training is performed based on a training set with labeled training data. In some instances, the training data used for fine-tuning 340 can be significantly smaller than the data used for pre-training 300, which can improve the efficiency of training execution.

[0044] Based on the generated classifier 360, when a new business document is provided, the generated classifier 360 can automatically identify the categories in the document. For example, the classifier 360 can be used to identify the categories in the document from an invoice (such as from Figure 2 The invoice 200) automatically identifies the sender address (370).

[0045] Figure 4 4 is a flow chart of an example process 400 for self-supervised pre-training of a model based on informative word masking according to an embodiment of the present disclosure. Pre-training can be performed as a pre-step to a training step for generating a model, wherein the generated model is trained to perform information extraction from business documents, such as, for example, about Figure 3 Subsequent training after pre-training can be performed to fine-tune a model generated using labeled data specific to the context of information extraction.

[0046] At 410, a first document base model is pre-trained based on unlabeled documents used as training data (e.g., business documents or other scanned documents with specific structure and title and line item information). The pre-training at 410 can be performed in the same manner as Figure 3 The pre-training 300 of is performed in a substantially similar manner. In some instances, the pre-training 410 can be performed as self-supervised training. Self-supervised pre-training can learn the common structure of the underlying training data by defining an excuse task that can produce a meaningful and descriptive representation of the unlabeled data. The excuse task is designed to generate labels for the unlabeled data.

[0047] Pre-training 410 includes obtaining (420) training data including unlabeled business documents. For example, obtaining 420 can be performed by uploading a scanned document at a storage space associated with the process execution. The obtained document can be a scanned document that can be processed by an optical character recognition (OCR) process, such as performed at an OCR engine. In some instances, the document is obtained as a scanned document, and text in the document and two-dimensional spatial information identifying the location of the extracted text (words or numbers or a combination thereof) can be obtained. The spatial information of the text obtained can identify the location of the text on the document. The business document can have a specific structure in which data is provided at different locations and regions, which can be aligned with each other to form a complete representation of a page of a specific page format.

[0048] At 430, pre-training is performed based on evaluating the masked word tokens according to individual masking ratios. The individual masking ratios are determined based on applying a dynamic window that is adjustable to the word token counts (occurrences) for each document in the unlabeled documents. The window applied is not a fixed window and is adjustable to the length of different documents in the training data. In some instances, the documents may be scanned pages, where different documents may have different numbers of words that may be organized in different ways (with different spatial relationships) at their respective pages.

[0049] In some instances, logical entities can be extracted from text (including text in images) and spatial text information obtained from executing an OCR program. When a scanned document is obtained, the text and spatial information can be extracted by applying rules that define rule-based extraction heuristics, neural network-based extraction models (e.g., DocumentReader, Chargrid, or Charmer), or post-processing logic (such as sender address coordination and matching). In response to obtaining the text and spatial information, the information can be saved in local storage, provided for use by a user, or provided to an embedded solution of a data extraction service for further document processing.

[0050] Pre-training of the first document-based model includes evaluating word tokens in each unlabeled business document, where masking is applied according to individual masking ratios determined for the word tokens. The masking ratios can be calculated to reflect the information relevance of words in the document, where those word tokens associated with higher relevance can be masked more frequently than those word tokens associated with lower relevance. Thus, the individual masking ratios indicate the respective information relevance of the word tokens.

[0051] In some instances, pre-training includes determining masking ratios for word tokens. Figure 3 As discussed in the pre-training of 300, masking every word in a document at the same rate during training may be associated with disadvantages in training performance (e.g., inefficiencies in training), and may therefore reduce the accuracy of the trained model. This disadvantage of randomly masking tokens during pre-training can be overcome by considering the informativeness of certain words and applying masking rates to words based on this consideration of an informativeness metric.

[0052] In some examples, the probability that words can appear together in a set window of words can be calculated. For example, a point-wise mutual information (PMI) metric can be calculated as a statistical measure to determine the probability that two words appear together in a predefined window divided by the probability that those words (w1 and w2) appear independently. The PMI metric can be calculated according to the following formula (2):

[0053]

[0054] In some embodiments, the PMI metric calculated according to formula (2) can be used to determine how "surprising" the co-occurrence of two words in a sequence is. p(w1, w2) represents the probability that two words appear together in a sequence, while p(w1) or p(w2) represents the probability that a word appears in a sequence.

[0055] In some embodiments, a measure of the information relevance of words in a defined sequence can be calculated based on a PMI metric. In some instances, the measure of the information relevance of a word can be calculated by summing the PMI scores, which are calculated for a combination of all masked and unmasked words in the sequence (e.g., all words in a document forming a page). In some instances, the PMI scores can be calculated using a dynamic window size corresponding to the length of a document page, and the PMI scores can be used to define the probability of masking word tokens during pre-training of a document-based model.

[0056] In some examples, when pre-training the first document base model, a sequence of word tokens used for training can be defined according to a dynamic window that is adjustable to the different lengths of different business documents used as training data for the pre-trained first document base model. During training, masking is applied, which is performed according to an individual masking ratio determined for each word token.

[0057] In some instances, a process for obtaining an individual masking ratio for each word token can be deployed and executed to provide the masking ratio as an input for determining masking probabilities for words during pre-training at 430. Based on such individual masking ratios, masking of some words can be reduced compared to other words. In particular, the masking probabilities of words that are less important to the information extraction task (e.g., words with relatively low information relevance measures) can be reduced. For example, stop words can be considered to be relatively less important words for providing information relevance, and therefore can be masked less frequently during training without negatively affecting the quality of training to infer the labels of word tokens.

[0058] In some embodiments, a process for obtaining an individual masking ratio for each word in a sequence of word tokens defined according to a dynamic window adjustable to the size of the training business document may be calculated by the following steps, including:

[0059] (i.) calculating a matrix having PMI scores for all words in a dataset including word tokens defined based on OCR-generated business documents as training data;

[0060] (ii.) for each word token sequence in the document, mask a specific portion (e.g., 15%) of the word tokens n times, where n corresponds to the number of sample sequences used during training;

[0061] (iii.) for each masked sequence, calculate the informativeness score by summing the PMI scores calculated between all masked and unmasked words;

[0062] (iv.) sorting the masking sequences in descending order according to their information scores;

[0063] (v.) taking the most informative masked sequence (e.g., with the highest ranking) and counting the masked tokens; and

[0064] (vi.) Divide the masked counts by the token frequency in the dataset.

[0065] Based on the execution of the above steps, the results achieved in the last step can be used to assign individual masking rates to each word token. In some instances, in order to make a masking decision for each sequence, a sample number of sequences with different masked words can be used when determining the most informative masked sequence. For example, n can be selected to be equal to 20 sample sequences, where the number of sample sequences determined can be determined taking into account the computational cost of performing the sorting and determining the masking rates while executing the algorithm to produce accurate results.

[0066] The PMI matrix calculated in step (i.) takes into account the special word distribution in the document compared to natural language. Instead of having a fixed window to count the co-occurrence of two words on the sentence or paragraph level, a dynamic window is applied to the evaluation. The dynamic window is adjustable to the number of tokens on the current document page. Thus, the word distribution of the document specific language corpus (specific to business documents) is modeled, and the masking decision is adapted to the information content of the words on the business documents. Having more intelligent masking decisions during model pre-training leads to higher accuracy of extracted entities and improves model convergence behavior. This can also lead to shorter training time and less computational cost.

[0067] Using individual masking rates can support improved pre-training and a reduction in the number of processing times for masking sequences including masking words that do not have high information relevance. Deploying this technique as a pre-processing step for training supports masking during training that can be performed faster and with fewer samples. Masking of each word token does not have to have the same masking rate, but instead relies on the information relevance of the words to optimize the determination of masking and training as a whole, while still achieving accurate training in a cost-effective manner without having to iterate through all possible masking combinations.

[0068] In some instances, by deploying this technique for calculating the masking ratio, the masking probability of words that are less important to the information extraction task (such as stop words) is reduced. For example, experiments show that although the word "limitation" has a high frequency, the masking ratio of this word token is less than 10% because it does not have high relevance at the document level. On the other hand, increasing the masking probability of words such as "phone" can be a logical decision because these tokens are often associated with extracted document entities (such as the buyer's phone number) and such entities have high relevance at the document level.

[0069] Based on applying the MLM pre-training process at 430, the first document base model is trained to reduce the loss in determining masked word tokens from unlabeled documents. At 440, the parameters of the learned first document base model are provided for initializing the second model. The second model is a model for document information extraction, which can be initialized and then trained based on labeled business documents as second training data.

[0070] At 450, a second document information extraction model is initialized based on the pre-trained first document base model, and the second document base model is trained by fine-tuning according to an adversarial training technique. Figure 6 and Figure 7 Performed as described.

[0071] At 460, based on the training, classifiers are generated on top of the second document information extraction model. In some instances, the second document information extraction model can be considered a domain-specific neural network extraction model because it is relevant to the details of business documents that can have different structures and word densities on document pages.

[0072] At 470, in response to receiving a new scanned business document, the generated classifier may be invoked to extract information from the new scanned business document and provide data about data objects identified in the document, such as a sender address from an invoice document.

[0073] In some instances, while masking as described above is determined at the word token level, masking may be performed on sub-word tokens. In some instances, masking performed at the sub-word token level may be defined for specific types of entities in a document and may not depend on the type of the entire document. For example, sub-word tokens may help with special entities such as date and time data or numbers (e.g., invoice numbers). The preprocessing step for determining individual masking rates may be performed at the sub-word level, and the masking rates for sub-words may be used to perform training based on unlabeled business documents.

[0074] In some instances, when word tokens are masked, the informativeness score of the sequence can be calculated. In some instances, spatial information can also be considered. For example, the spatial distance of words can be used to determine the amount of information of words in a document. For example, when the sum of PMI scores is performed in the above step (iii.), a distance-based weight can be applied. For example, the distance-based weight can be calculated based on any Minkowski distance (such as Euclidean distance and Manhattan distance) or similar or derived distance metrics. In some instances, the a posteriori knowledge of the word tokens into the information score can also be considered when determining the masking score, such as whether the word is already part of an entity extracted by a different extraction model. For example, key-value pairs can be defined and assigned higher information scores. In one example, the solution can assign key-value pairs with higher information scores to the information scores of other word tokens that do not belong to the predefined pair, such as "Invoice#:" and "123234". In some other examples, the informativeness score of the predefined key-value pairs can be defined as a higher score by adding a predefined "add" value (e.g., a percentage or other value on a given scale).

[0075] Figure 5 is a flow chart showing a process 500 of fine-tuning a training model to extract information from a document based on a virtual adversarial technique according to an embodiment of the present disclosure.

[0076] At 510, a document information extraction model is initialized based on a pre-trained model, for example, Figure 4 The initialization may be based on weights provided from a first (pre-trained) model, the first (pre-trained) model being pre-trained on training data comprising unlabeled documents as first training data.

[0077] At 520, fine-tuning of the information extraction model is performed based on training based on labeled documents (e.g., business documents or other scanned documents having a specific structure and including header and line item data) as second training data. The labeled business documents are labeled and evaluated based on virtual adversarial training.

[0078] At 530 , classifiers are generated on top of the second document information extraction model based on the performed fine-tuning.

[0079] At 540, in response to receiving the scanned business document for information extraction, the generated classifier can be called to obtain information of the data objects identified in the scanned business document. For example, the name and address of the sender of the invoice can be automatically determined.

[0080] Figure 6is a block diagram representing a process 600 for applying virtual adversarial training (VAT) to train a model for extracting information from business documents, according to an embodiment of the present disclosure.

[0081] In some instances, for example, as Figure 3 As discussed in fine-tuning 340 , a pre-trained model may be fine-tuned, such as by Figure 3 and Figure 4 The model provided by the explained pre-training. Fine-tuning can be performed based on initializing the model according to the parameters provided by the pre-trained model and then training the model based on labeled data and according to the virtual adversarial techniques described in this disclosure.

[0082] In some instances, VAT can be applied in the context of training in the business document domain and based on input from a pre-trained model (for initialization). The pre-trained model can be trained on unlabeled data that is masked based on a masking probability that corresponds to the information relevance of the masked word tokens over a dynamic window. Therefore, VAT can be applied in a more efficient manner by starting training from a better positioned starting point for accurately identifying the structure of business documents. By applying the VAT technique, VAT aims to improve the robustness of the model by adding small perturbations to the input to have the same output distribution between task-specific examples and adversarial examples.

[0083] In some instances, training data for VAT can be obtained, and input embedding 610 can be defined for input business documents marked as training data. Input embedding can be defined to include text (e.g., word tokens) and spatial embedding. Text and spatial embeddings with tokens can be obtained from the OCR engine based on the recognition performed on the scanned document provided as marked training data. The text embedding and the spatial embedding are cascaded to provide a cascaded embedding 620, which will be used to generate predictions during training, which can be compared to fine-tune the model to perform accurate information extraction from business documents. Layer normalization can be applied on these embeddings as a cascade to generate normalized embedding 630. The cascaded embedding 620 is also perturbed with a VAT target, and the labels of the models trained for both the original embedding and the perturbed embedding are predicted. Based on the normalized embedding 630, a model prediction 650 is generated, and based on the adversarial perturbation 640, a model prediction 660 is generated. Predictions 650 and 660 are compared according to the Kullback-Leibler (KL) divergence 670. KL divergence 670 can be used as a comparison measure of the probability distributions of two predictions ( 650 and 660 ) ​​from two models trained based on normalized embedding 630 and adversarial perturbation 640 .

[0084] In some examples, the adversarial perturbation 640 is calculated by maximizing the KL divergence between the regular model output distribution q(y|x) and the adversarial output distribution p(y|x+r,θ), as in the following equation (3):

[0085] (3)L adv (x,θ)=l s [q(y|x),p(y|x+r,θ)],

[0086] where y is the model output, x is the model input, r is the adversarial perturbation, and l s is a symmetric version of the KL divergence with two distributions P and Q, where:

[0087] (4) l s (P,Q)=D KL (P||Q)+D KL (Q||P)

[0088] Thus, the perturbation is not random or Gaussian noise, but is calculated to ensure that it is small enough not to change the inherent meaning of the input by maximizing the KL divergence. In some instances, the noise added for the adversarial perturbation 640 can be directed to areas of the model where the model is most sensitive or uncertain to expose vulnerabilities during training. Compared to standard adversarial training that selects the direction that deviates most from the correct classification, process 600 applies a label-independent VAT technique and therefore only "virtually" adversarial.

[0089] In some instances, the output as the prediction from the adversarial example is compared with the original input (just normalized) based on the KL-divergence metric that defines the VAT loss.

[0090] Figure 7 700 is a block diagram of a process 700 for performing virtual adversarial training in the context of business document information extraction according to an embodiment of the present disclosure. Process 700 may be performed in a VAT technique implementation, such as with respect to Figure 5 and Figure 6 as described.

[0091] The VAT technique relies on using labeled input business documents processed twice by the same model, e.g. a model initialized based on parameters provided from a pre-training process, e.g., as described about Figure 3 and Figure 4 In some instances, the training labeled data may include a document 701 that is a business document and is associated with a labeled version (705) of the document 701 and an adversarial example 710 that is a version of the document. In some instances, the obtained document 701 may be pre-labeled and obtained as a labeled document 705.

[0092] The processing of the labeled input document (such as document 705) is performed twice. First, the processing is based on the text and spatial embeddings extracted from the original business document, and then, the processing is based on the perturbed embeddings. This processing corresponds to the processing of the labeled document 705 and the adversarial example 710 (with perturbations). Figure 7 The adversarial training shown includes a process of providing adversarial examples as instances that are introduced into the document information extraction model and are marked as threats because they are designed to cause the model to make incorrect predictions. Therefore, the classification of the model is tested based on such adversarial examples, and the performance of the model is evaluated to improve the predictive ability of the model. The output distributions resulting from the predictions based on the processing of the input "as is" and the perturbed input can be presented as shown in 720 and 725, respectively. At 730, the prediction can be performed using the method described in relation to Figure 6 The KL divergence in question is used to compare the distribution of the results, i.e., correctly classified and incorrectly classified. The comparison of the distributions can be a scalar number, which can be viewed as a new component of the training loss function in the context of VAT training as a fine-tuning process of a pre-trained model. Thus, the KL divergence 730 can be used together with the cross entropy loss 740 to determine the final loss 750 from the execution of the processing. It can be as described with respect to Figure 3 As discussed in more detail, the cross entropy loss 740 can be calculated as described with respect to Figure 3 As discussed in more detail and based on the true labeled data 735 and the data obtained from training based on the labeled documents 705, the cross entropy loss 740 is calculated.

[0093] In some instances, the resulting scalar obtained from the comparison (as output from 730) can be added to the conventional cross entropy loss (as determined at 740) to determine the final loss 750. The cross entropy loss 740 can be determined from a conventional classification task based on the original input training data (including document 705) processed by the training model to obtain the resulting distribution at 720 and the true label of the input data (e.g., labeled document 705).

[0094] Applying VAT techniques can provide several benefits. The approach based on applying adversarial training to fine-tune a pre-trained model is conceptually model-agnostic, i.e., it can be deployed to different neural network architectures and supports robust and accurate results.

[0095] In some embodiments, perturbations that can be defined for embeddings (e.g., text embeddings) can be entity-aware. In this way, perturbations can be designed to specifically target entities of a document, such as a name, date, or address, by using the predictions of an existing model. In some instances, constraints can be applied to perturbations of those entities to avoid the perturbations becoming invalid or meaningless.

[0096] According to the present invention, the VAT technique is applied to the field of two-dimensional (2D) document information extraction and adapted to the task of identifying the labeling of masked tokens based on the processing of the original output and the adversarial output. To this end, adversarial perturbations are applied to the cascade of text embeddings and spatial embeddings, and thus, perturbations are applied to the document representation. In some instances, the resulting fine-tuning performance may exceed previous methods in terms of extraction accuracy and training stability.

[0097] Figure 8 800 is a schematic diagram of an example computer system 800 that can be used to perform embodiments of the present disclosure. For example, the computer system 800 can be included in any or all of the server components discussed herein. The computer system 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. The components 810, 820, 830, and 840 are interconnected using a system bus 850. The processor 810 is capable of processing instructions for execution within the computer system 800. In some embodiments, the processor 810 is a single-threaded processor. In some embodiments, the processor 810 is a multi-threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830 to display graphical information of a user interface on the input / output device 840.

[0098] The memory 820 stores information within the computer system 800. In some embodiments, the memory 820 is a computer-readable medium. In some embodiments, the memory 820 is a volatile memory unit. In some embodiments, the memory 820 is a non-volatile memory unit. The storage device 830 can provide mass storage for the system 800. In some embodiments, the storage device 830 is a computer-readable medium. In some embodiments, the storage device 830 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device. The input / output device 840 provides input / output operations for the computer system 800. In some embodiments, the input / output device 840 includes a keyboard and / or a pointing device. In some embodiments, the input / output device 840 includes a display unit for displaying a graphical user interface.

[0099] The features described may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The apparatus may be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device for execution by a programmable processor), and the method operations may be performed by a programmable processor executing a program of instructions to perform the functions of the described implementation by operating on input data and generating output. The features described may advantageously be implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform some activity or produce some result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or another unit suitable for use in a computing environment.

[0100] As examples, suitable processors for executing instruction programs include both general and special purpose microprocessors, as well as the sole processor of any type of computer or one of multiple processors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer may include a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer may also include one or more mass storage devices for storing data files, or be operably coupled to communicate with one or more mass storage devices; such devices include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including: for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0101] To provide for interaction with a user, these features may be implemented on a computer having a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer.

[0102] These features can be implemented in a computer system that includes a back-end component (such as a data server), or includes a middleware component (such as an application server or an Internet server), or includes a front-end component (such as a client computer with a graphical user interface or an Internet browser), or any combination thereof. The components of the system can be connected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include, for example, LANs, WANs, and the computers and networks that form the Internet.

[0103] A computer system may include clients and servers. The client and server are generally remote from each other and typically interact through a network (such as the described network). The relationship between the client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0104] In addition, the logic flows depicted in the accompanying drawings do not require the particular order or sequential order shown to achieve the desired results. In addition, other operations can be provided, or operations can be removed from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0105] A number of embodiments of the present disclosure have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0106] In view of the implementation of the above-mentioned subject matter, the present application discloses the following list of examples, wherein one feature of a separate example or more than one feature combination of the examples and optionally combined with one or more features of one or more additional examples are additional examples that also fall within the disclosure of the present application.

[0107] Example

[0108] Although the application is defined in the accompanying claims, it should be understood that the invention may also (alternatively) be defined according to the following examples:

[0109] Self-guided pre-training of document information extraction models via informative word masking

[0110] Example 1. A computer-implemented method comprising:

[0111] Obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction;

[0112] pre-training the first model based on a dynamic window of word token counts adjustable for each of the plurality of unlabeled documents, wherein the pre-training comprises: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate respective information relevances of the word tokens; and

[0113] The pre-trained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.

[0114] Example 2. A method according to Example 1, wherein obtaining the training data includes performing optical character recognition to determine words as part of at least some of the multiple unlabeled documents.

[0115] Example 3. The method of Example 1 or Example 2, wherein pre-training the first model comprises:

[0116] tokenizing each document of the plurality of unlabeled documents into a corresponding sequence of word tokens and spatial embeddings;

[0117] calculating individual masking ratios for word tokens that identify information relevance of corresponding word tokens in the plurality of unlabeled documents; and

[0118] Representations of the first model are learned as a document base model, wherein the first model includes weights as parameters of the model defining connection strengths across different layers in the model.

[0119] Example 4. The method of Example 1, wherein the word token comprises at least one of a single word or a sub-word.

[0120] Example 5. A method according to any of the preceding examples, wherein the pre-training comprises:

[0121] defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and

[0122] An individual masking ratio is obtained for each word token in the word token sequence.

[0123] Example 6. The method of Example 5, wherein obtaining training data comprises performing text recognition on a plurality of unlabeled documents to obtain documents including the recognized text portions, and wherein obtaining the individual masking ratios comprises:

[0124] computing a matrix having pointwise mutual information (PMI) scores for the word tokens in the training data;

[0125] for each sequence of word tokens in a document, masking specific portions of the word tokens, wherein each sequence is masked n times, where n corresponds to the number of sample sequences used during the training;

[0126] For each masked sequence, the information score was calculated by summing the PMI scores calculated between all masked and unmasked words;

[0127] Sort the masked sequences according to their corresponding information scores;

[0128] selecting the masking sequence associated with the highest information score according to the ranking as the most informative masking sequence; and

[0129] Divide the masked count by the frequency of the corresponding word token in the training data.

[0130] Example 7. A method according to any of the preceding examples, comprising:

[0131] The second document information extraction model is trained based on the marked documents for document information extraction, wherein the training comprises:

[0132] The second document information extraction model is initialized based on input provided from a pre-trained first model.

[0133] Example 8. A method according to any of the preceding examples, comprising:

[0134] Adversarial training is performed on the second document information extraction model, where the second document information extraction model is initialized based on the pre-trained first model.

[0135] Example 9. A method according to any of the preceding examples, comprising:

[0136] Initializing the second model based on weights provided from the pre-trained first model;

[0137] performing fine-tuning of the pre-trained first model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and

[0138] Based on the performed fine-tuning, a classifier is generated on top of the second document information extraction model.

[0139] Example 10. The method of Example 9, wherein performing the fine-tuning comprises:

[0140] Creating text and spatial embeddings from the tagged business documents, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the tagged business documents;

[0141] concatenating the textual and spatial embeddings to form a document representation;

[0142] applying layer normalization to the concatenated textual and spatial embeddings; and

[0143] The concatenated textual and spatial embeddings are perturbed according to an objective of the VAT to predict tokens of the original and perturbed embeddings, wherein performing the VAT comprises:

[0144] processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation;

[0145] comparing a distribution of documents with the concatenated embedding and a distribution of the documents with the perturbed embedding; and

[0146] A result is obtained to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data.

[0147] Example 11. A system comprising:

[0148] one or more processors; and

[0149] One or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform any one of the methods of Examples 1-10.

[0150] Example 12. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of Examples 1 to 10.

[0151] Virtual Adversarial Training of Document Information Extraction Model

[0152] Example 1. A computer-implemented method comprising:

[0153] pre-training a domain-specific neural network extraction model based on first training data, wherein the training data comprises unlabeled documents, wherein the pre-training is performed based on informative word masking;

[0154] performing fine-tuning of a pre-trained domain-specific neural network extraction model based on labeled business documents as second training data, wherein the labeled business documents are processed according to a virtual adversarial training (VAT) technique; and

[0155] Based on the performed fine-tuning, a classifier associated with the domain-specific neural network extraction model is generated.

[0156] Example 2. A computer-implemented method comprising:

[0157] Initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data;

[0158] performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and

[0159] Based on the performed fine-tuning, a classifier for classification for information extraction is generated.

[0160] Example 3. The method according to Example 2, comprising:

[0161] In response to receiving a scanned business document for information extraction, the generated classifier is invoked to obtain information for data objects identified in the scanned business document.

[0162] Example 4. The method of Example 2 or Example 3, wherein performing the fine-tuning comprises:

[0163] Creating text and spatial embeddings from the tagged business documents, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the tagged business documents;

[0164] concatenating the textual and spatial embeddings to form a document representation;

[0165] applying layer normalization to the concatenated textual and spatial embeddings;

[0166] The concatenated textual and spatial embeddings are perturbed according to the objective of the VAT to predict tags for the original and perturbed embeddings.

[0167] Example 5. The method of Example 4, wherein performing the fine-tuning comprises:

[0168] processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation;

[0169] comparing a distribution of documents with the concatenated embedding and a distribution of the documents with the perturbed embedding; and

[0170] A result to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data as an original is obtained.

[0171] Example 6. A method according to any of the preceding examples, wherein the pre-training is applied based on a dynamic window that is adjustable to a word token size of each document of the unlabeled business documents.

[0172] Example 7. A method according to any of the preceding examples, wherein the pre-training comprises evaluating word tokens in each unlabeled business document to which masking is applied based on individual masking ratios determined for the word tokens.

[0173] Example 8. The method of Example 7, wherein the individual masking ratios indicate respective information relevance of the word tokens, and wherein pre-training comprises:

[0174] defining a sequence of word tokens for each of the unlabeled business documents, wherein the sequence has different lengths corresponding to counts of word tokens determined for each of the unlabeled business documents after performing text recognition; and

[0175] An individual masking ratio is obtained for each word token in the word token sequence.

[0176] Example 9. A system comprising:

[0177] one or more processors; and

[0178] One or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform any one of the methods of Examples 1 to 8.

[0179] Example 10. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of Examples 1 to 8.

Claims

1. A computer-implemented method comprising: Initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data; performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); as well as Based on the performed fine-tuning, a classifier for classification for information extraction is generated.

2. The method according to claim 1, comprising: In response to receiving a scanned business document for information extraction, the generated classifier is invoked to obtain information for data objects identified in the scanned business document.

3. The method according to claim 1, wherein: Performing the fine-tuning includes: creating text and spatial embeddings from the tagged business documents, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the tagged business documents; concatenating the textual and spatial embeddings to form a document representation; applying layer normalization to the concatenated textual and spatial embeddings; The concatenated textual and spatial embeddings are perturbed according to the objective of the VAT to predict labels for the original embeddings and the perturbed embeddings.

4. The method according to claim 3, wherein: Performing the fine-tuning includes: processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation; comparing a distribution of documents with the concatenated embedding and a distribution of the documents with the perturbed embedding; and A result to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data as an original is obtained.

5. The method according to claim 1, wherein: The pre-training is applied based on a dynamic window that can be adjusted to a word token size per document of the unlabeled business documents.

6. The method according to claim 1, wherein: The pre-training includes evaluating word tokens in each of the unlabeled business documents in which masking is applied according to individual masking ratios determined for the word tokens.

7. The method according to claim 6, wherein: The individual masking ratios indicate respective information relevances of the word tokens, and wherein the pre-training comprises: defining a sequence of word tokens for each of the unlabeled business documents, wherein the sequence has different lengths corresponding to counts of word tokens determined for each of the unlabeled business documents after performing text recognition; and An individual masking ratio is obtained for each word token in the word token sequence.

8. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: Initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data; performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); as well as Based on the performed fine-tuning, a classifier for classification for information extraction is generated.

9. The non-transitory computer-readable medium of claim 8, comprising: In response to receiving a scanned business document for information extraction, the generated classifier is invoked to obtain information for data objects identified in the scanned business document.

10. The non-transitory computer readable medium of claim 8, wherein: Performing the fine-tuning includes: creating text and spatial embeddings from the tagged business documents, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the tagged business documents; concatenating the textual and spatial embeddings to form a document representation; applying layer normalization to the concatenated textual and spatial embeddings; The concatenated textual and spatial embeddings are perturbed according to the objective of the VAT to predict labels for the original embeddings and the perturbed embeddings.

11. The non-transitory computer readable medium of claim 10, wherein: Performing the fine-tuning includes: processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation; comparing a distribution of documents with the concatenated embedding and a distribution of the documents with the perturbed embedding; and A result to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data as an original is obtained.

12. The non-transitory computer readable medium of claim 8, wherein: The pre-training is applied based on a dynamic window that can be adjusted to a word token size per document of the unlabeled business documents.

13. The non-transitory computer readable medium of claim 8, wherein: The pre-training includes evaluating word tokens in each of the unlabeled business documents in which masking is applied according to individual masking ratios determined for the word tokens.

14. The non-transitory computer readable medium of claim 13, wherein: The individual masking ratios indicate respective information relevances of the word tokens, and wherein the pre-training comprises: defining a sequence of word tokens for each of the unlabeled business documents, wherein the sequence has different lengths corresponding to counts of word tokens determined for each of the unlabeled business documents after performing text recognition; and An individual masking ratio is obtained for each word token in the word token sequence.

15. A computer-implemented system comprising: one or more processors; as well as one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform operations comprising: Initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data; performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and Based on the performed fine-tuning, a classifier for classification for information extraction is generated.

16. The system of claim 15, wherein: The one or more computer-readable memories coupled to the one or more processors have instructions stored thereon, the instructions executable by the one or more processors to perform operations comprising: In response to receiving a scanned business document for information extraction, the generated classifier is invoked to obtain information for data objects identified in the scanned business document.

17. The system of claim 15, wherein: Performing the fine-tuning includes: creating text and spatial embeddings from the tagged business documents, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the tagged business documents; concatenating the textual and spatial embeddings to form a document representation; applying layer normalization to the concatenated textual and spatial embeddings; The concatenated textual and spatial embeddings are perturbed according to the objective of the VAT to predict labels for the original embeddings and the perturbed embeddings.

18. The system of claim 15, wherein: Performing the fine-tuning includes: processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation; comparing a distribution of documents with the concatenated embedding and a distribution of the documents with the perturbed embedding; and A result to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data as an original is obtained.

19. The system of claim 15, wherein: The pre-training is applied based on a dynamic window that can be adjusted to a word token size per document of the unlabeled business documents.

20. The system of claim 15, wherein: The pre-training includes evaluating word tokens in each of the unlabeled business documents in which masking is applied according to individual masking ratios determined for the word tokens.

Images