Self-guided pre-training of document information extraction model based on information word masking

Abstract

The present disclosure relates to computer-implemented methods, software, and systems for extracting information from business documents based on training techniques to generate a document base model by pre-training. First training data based on a plurality of unmarked documents is obtained for training a first model for document information extraction. The first model is pre-trained according to a dynamic window that is adjustable for a word token count for each of the plurality of untagged documents. The pre-training includes evaluating a word token in each untagged document of the plurality of untagged documents, wherein masking is applied according to an individual masking rate determined for the word token. The individual masking rates indicate respective information dependencies of the word tokens. A pre-trained first model is provided for initializing a second document information extraction model to be trained based on a tagged document as second training data.

Description

Technical Field

[0001] The present disclosure relates to computer-implemented methods, software, and systems for data processing. Background Art

[0002] Information extraction methods can be used in different natural language processing based applications. These methods can be used to identify information and provide natural language understanding. Information extraction models can be learned through a combination of pre-training, fine-tuning, and contextual learning. Pre-training is the initial stage of model learning. During pre-training, the model is exposed to a large amount of unlabeled text data, where pre-training aims to capture the underlying patterns, structures, and semantic knowledge present in the text corpus. Fine-tuning is the process of further training the model on a specific task or in a specific domain. Fine-tuning can improve the performance of a model pre-trained on a specific task or domain to better fit the data. Summary of the invention

[0003] Embodiments of the present disclosure are generally directed to a computer-implemented method for performing selection of a predictive model from a set of candidate predictive models based on an assessment of risk of bias.

[0004] In one aspect, an example method may include operations such as: obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction; pre-training the first model based on a dynamic window adjustable for word token counts for each of the plurality of unlabeled documents, wherein the pre-training includes: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate corresponding information relevance of the word tokens; and providing the pre-trained first model for initializing a second document information extraction model to be trained based on labeled documents as second training data.

[0005] In a second aspect, an example method may include operations such as: pre-training a domain-specific neural network extraction model based on first training data, wherein the training data includes unlabeled documents, wherein the pre-training is performed based on informative word masking; performing fine-tuning of the pre-trained domain-specific neural network extraction model based on labeled business documents as second training data, wherein the labeled business documents are processed according to a virtual adversarial training (VAT) technique; and generating a classifier associated with the domain-specific neural network extraction model based on the performed fine-tuning.

[0006] In a third aspect, an example method may include operations such as: initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data; performing fine-tuning of the second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and generating a classifier for classification for information extraction based on the performed fine-tuning.

[0007] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the methods provided herein.

[0008] The present disclosure also provides a system for implementing the method provided herein. The system includes one or more processors, and a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations according to the implementation of the method provided herein.

[0009] It should be understood that the method according to the present disclosure may include any combination of aspects and features described herein. That is, the method according to the present disclosure is not limited to the combination of aspects and features specifically described herein, but also includes any combination of aspects and features provided.

[0010] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram depicting an example computer-implemented system capable of performing embodiments of the present disclosure in accordance with embodiments of the present disclosure.

[0012] Figure 2 is a block diagram depicting an example of a business document and extracted information from the business document including header information and line item information.

[0013] Figure 3 is a block diagram presenting an example of a process for pre-training and fine-tuning a model for extracting information from documents according to an embodiment of the present disclosure.

[0014] Figure 4 is a flow chart representing a process for training and executing a model to extract information from a document according to an embodiment of the present disclosure.

[0015] Figure 5 is a flowchart representing a process of fine-tuning a trained model to extract information from a document based on a virtual adversarial technique according to an embodiment of the present disclosure.

[0016] Figure 6 is a block diagram representing steps of embedding applied in virtual adversarial training for training a model for extracting information from business documents according to an embodiment of the present disclosure.

[0017] Figure 7 is a block diagram representing a process of performing virtual adversarial training in the context of business document information extraction according to an embodiment of the present disclosure.

[0018] Figure 8 is a schematic diagram of an example computer system that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] This disclosure describes various tools and techniques for extracting information from business documents based on training techniques to generate a document-based model customized to the specifics of the business documents through pre-training and fine-tuning.

[0020] In some instances, a web application or service can provide a service for extracting information of logical entities identified in a scanned document, such as a business document. Logical entities in a business document can include header information and / or line item information. For example, header information can include data such as document date, name (e.g., sender name), sender bank account, sender tax ID, price amount or total amount, and other entity related data. Line item information can be data provided in a table in a document and included in a field. Line item information can include line item text (e.g., description of a product, product name, etc.), line item quantity (e.g., unit of product), and / or line item quantity (e.g., price, number of products sold, etc.).

[0021] A business document can be a document associated with a particular business task, such as an order, purchase, request, or other event or action. For example, information can be extracted from a business document, such as an invoice, purchase order, order confirmation, or payment proposal. In some instances, the extraction of information can be based on logic at a trained model that, when provided with a scanned business document, can automatically classify recognized data from the document.

[0022] In some embodiments, a machine learning model can be trained to automatically extract information from business documents and provide the extracted information to an end user or other software application or service (e.g., as input for initiating another process or task). For example, if an invoice is provided in a scanned format for use in preparing a tax return (e.g., by another application service or process), a document information extraction model can be trained to process the invoice based on the model and provide output from the processing to trigger a tax return generation process. Different configurations for using the trained model can be configured in the software system and / or application to provide documents as input and obtain data as output, which can be further used as input in other processes (e.g., triggering the initiation of a process at another system based on accumulated data from processed business documents).

[0023] In some embodiments, a service that provides information extraction capabilities based on a trained model can be integrated into a software solution to convert business documents into database entries. In some embodiments, the creation of such a database entry can trigger a business process such as payment or other transaction execution.

[0024] Training a model for document information extraction can be associated with processing a large number of documents that may have to be manually annotated for training. In some instances, document annotation is a labor-intensive process that may be associated with significant resource and time requirements, and the results may be error-prone. In some instances, annotated datasets may not be available for every type of business document. Therefore, providing a trained model that can effectively process domain-specific scanned documents (e.g., business documents) can provide advantages for processing documents in a faster and more reliable manner to obtain accurate results.

[0025] In some instances, a solution for training a model is provided according to an embodiment of the present disclosure, which relies on pre-training techniques applied to unlabeled documents, which are also fine-tuned using a domain-specific and annotated smaller training set. The pre-training technique can include training based on a masking technique that masks words based on a probability rate determined for a word token, which is based on the relevance of the word token to the text in the document. In this way, these techniques allow training to focus on words that have a higher relevance to the text in the document, and the training process is optimized. Pre-training can be performed based on a dynamic window for evaluating the text in the document, so that the evaluation of the document is adjustable for the size of each document, rather than fixed. By utilizing a dynamic window, the word distribution of the language corpus of the document is modeled, and masking during training can be performed based on the determined amount of information of the words in the document, treating each document as a whole. The trained model can be fine-tuned by applying virtual adversarial row training, in which adversarial row perturbations are applied to cascaded text and spatial embeddings. Perturbations can be added to input training data (eg, as small noise) to be processed by the model and used to train the model to still correctly recognize and classify the input data.

[0026] Figure 1 1 is a block diagram depicting an example computer-implemented system 100 capable of performing embodiments of the present disclosure in accordance with embodiments of the present disclosure. In the depicted example, the example system 100 includes a client device 102, a client device 104, a network 110, a cloud environment 106, and a cloud environment 108. The cloud environment 106 may include one or more server devices and databases (e.g., processors, memory). In the depicted example, a user 114 interacts with the client device 102, and a user 116 interacts with the client device 104.

[0027] In some examples, client device 102 and / or client device 104 can communicate with cloud environment 106 and / or cloud environment 108 via network 110. Client device 102 can include any suitable type of computing device, such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular phone, a network device, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or a suitable combination of any two or more of these devices or other data processing devices. In some embodiments, network 110 can include a large computer network that connects any number of communication devices, mobile computing devices, fixed computing devices, and server systems, such as a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., a public switched telephone network (PSTN)), or a suitable combination thereof.

[0028] In some implementations, the cloud environment 106 includes at least one server and at least one data store 120. Figure 1 In the example of , cloud environment 106 is intended to represent various forms of servers, including but not limited to web servers, application servers, proxy servers, network servers and / or server pools. Typically, the server system accepts requests for application services and provides such services to any number of client devices (e.g., client devices 102 via network 110).

[0029] According to embodiments of the present disclosure, and as described above, cloud environment 106 can host applications and databases running on a host infrastructure. In some instances, cloud environment 106 can include multiple cluster nodes, which can represent physical or virtual machines that can represent a cluster of web nodes. Hosted applications and / or services can run on VMs hosted on cloud infrastructure.

[0030] In some examples, cloud environment 106 and / or cloud environment 108 can provide infrastructure for performing pre-training and adversarial training to classify documents, such as with respect to Figure 2 , 3 , 4 and 5 are further described.

[0031] In some examples, cloud environment 106 and / or cloud environment 108 can provide infrastructure to perform training of document information extraction models according to embodiments of the present disclosure.

[0032] Figure 2 is a block diagram depicting an example of a business document 200 and corresponding extracted information from the business document, where the extracted information includes header information and line item information.

[0033] Business document 200 is an invoice, which includes information about purchasing a certain quantity of product "apples" at a certain price, here, 1400 pieces for a total price of $1,684. Figure 2 Other items of the invoice in the example document 200 are omitted in the representation. When such a document is provided as a scanned document, it is possible to recognize text in the document by using an optical character recognition (OCR) application to obtain the text on the document and two-dimensional spatial information identifying the position of the extracted text on the document.

[0034] exist Figure 2In the example of , document 200 is a business document including logical entities, including title information and line item information. In some instances, such information can be extracted from the document based on a trained model according to an embodiment of the present disclosure (e.g., a neural network model trained using pre-training techniques and virtual adversarial training). The extracted information can be used to create an entry in a database storage, or can be provided as input to be stored in another application or system that can store data according to a different data model.

[0035] Logical entities in a business document can include header information 210 and line item information 220. The header information 210 includes information about the sender's name, the ID provided, the tax ID, the invoice number, the currency of the invoice, the total amount of the invoice, and the taxes applied. This information is located at word blocks on the business document and is spatially distant from each other. The line item information 220 of the business document is information located in a row (single line) on the document and includes information about the subject of the invoice, here the purchase of apples. The line item information 220 can be extracted from multiple fields on the same line identified at the invoice. In the example invoice 200, this is information about the amount of apples purchased at a specific price.

[0036] Figure 3 is a block diagram presenting an example of a process for pre-training 300 and fine-tuning 340 a model for extracting information from business documents according to an embodiment of the present disclosure.

[0037] In some instances, a model can be trained to classify data extracted from business documents by first performing pre-training based on unlabeled training data (i.e., business documents) to learn the structure of the data, wherein the pre-trained model can be fine-tuned using self-supervised learning techniques based on the training data, which is labeled and relies on representations of the documents determined based on the pre-training.

[0038] During the pre-training phase (300), the model is trained on unlabeled business documents 310 as training data. Business documents can be processed for text recognition as they can be scanned documents. Text recognition can be used to extract text from each document, which is then used during training. Based on the training, representations 320 are learned by solving a pretext task 330 and by applying masked language modeling (MLM). The pre-training phase can be performed based on MLM techniques to achieve bidirectional learning from text from training data by masking or hiding words or other defined parts of text in the document, and predicting the masked words using other words on either side of the masked words (bidirectionally). MLM techniques can be applied so that a certain percentage of words in the document (which can be defined as text embeddings) can be masked, and the model can predict those masked words based on the context of the business document.

[0039] In some examples, the pretext task 330 can be defined as a loss minimization problem for learning a pre-trained model, where the problem is parameterized by θ based on considering training business documents presented as sequences of word tokens and spatial tokens. M′ and word token Y L′ The model can be presented as follows and can be defined to minimize the cross entropy loss function as presented below at formula (1):

[0040]

[0041] However, randomly masking tokens during pre-training as in MLM may have certain disadvantages. For example, some word tokens may be too easy to predict if masked; therefore, including those as masking elements may lead to inefficient loss determination during pre-training. Therefore, masking all words at the same rate without considering how informative they are in the context of the text (i.e., the content of the scanned document) may reduce the usefulness of the final model's extraction accuracy. Furthermore, if masking is applied randomly to a set of consecutive words, training can focus on the interpretation of the word based only on the surrounding words, and ignore the entire sequence representation.

[0042] In some embodiments, to address the limitation of randomly masking tokens during pre-training, the informativeness of a word can be determined, and masking can be applied based on an individual masking ratio corresponding to a metric defining the relevance of a given word. Thus, tokens of words with higher masking ratios can be masked more frequently during training than words with lower masking ratios. Figure 4 The pre-training process is described in further detail in , which includes taking into account the informativeness of words and the entire size of the document as a sequence of tokens.

[0043] During fine-tuning 340, the representation 320 of the pre-trained model is used for initialization of a representation 350 to be used for fine-tuning based on training using labeled documents as training data. By initializing the representation 350 with the representation 320 at the fine-tuning 340 step, the identification of the amount of information of words within the document determined during pre-training can be transferred to the fine-tuning stage, thereby supporting improved accuracy, while also reducing resource expenditure by reducing training time and computing resources used to perform data processing. In addition, the representation 350 is initialized based on the structure identified for the document at the pre-training step 300 to maintain the architecture of the training execution to match the structure of the training, thereby generating a representation of the model - here, the representations 320 and 350. Pre-training and fine-tuning as described in the present disclosure can include substantially similar training steps with respect to the encoding step, but can differ in decoding. In some instances, during pre-training, words are classified based on all words in the vocabulary, while fine-tuning classifies words based on predefined labels because the input is labeled (e.g., the label of the sender of the invoice document is used as input). The input documents for training at the fine-tuning 340 stage are annotated or otherwise labeled, and the representation 350 is trained according to a virtual adversarial technique after being initialized to generate a fine-tuned representation. The fine-tuned representation can then be used to generate a classifier 360. Pre-training 300 allows the representation 320 to be generated and input into the training at the fine-tuning step 340, where the learning process is faster and produces accurate results for classification in a specific field of business documents because the training is performed based on a training set with labeled training data. In some instances, the training data used for fine-tuning 340 can be significantly smaller than the data used for pre-training 300, which can improve the efficiency of training execution.

[0044] Based on the generated classifier 360, when a new business document is provided, the generated classifier 360 can automatically identify the categories in the document. For example, the classifier 360 can be used to classify the invoice (such as from Figure 2 The invoice 200) automatically identifies the sender address (370).

[0045] Figure 4 4 is a flow chart of an example process 400 for self-supervised pre-training of a model based on informative word masking according to an embodiment of the present disclosure. Pre-training can be performed as a pre-step to a training step for generating a model, wherein the generated model is trained to perform information extraction from business documents, such as, for example, about Figure 3 Subsequent training after pre-training can be performed to fine-tune the model generated using labeled data specific to the context of information extraction.

[0046] At 410, a first document base model is pre-trained based on unlabeled documents (e.g., business documents or other scanned documents with specific structure and title and line item information) used as training data. The pre-training at 410 can be performed in the same manner as Figure 3 The pre-training 300 of is performed in a substantially similar manner. In some instances, the pre-training 410 can be performed as self-supervised training. Self-supervised pre-training can learn the common structure of the underlying training data by defining an excuse task that can produce a meaningful and descriptive representation of the unlabeled data. The excuse task is designed to generate labels for the unlabeled data.

[0047] Pre-training 410 includes obtaining (420) training data including unlabeled business documents. For example, obtaining 420 can be performed by uploading a scanned document at a storage space associated with the process execution. The obtained document can be a scanned document that can be processed by an optical character recognition (OCR) process, such as performed at an OCR engine. In some instances, the document is obtained as a scanned document, and the text in the document and two-dimensional spatial information (words or numbers or a combination thereof) identifying the location of the extracted text can be obtained. The spatial information of the text obtained can identify the location of the text on the document. The business document can have a specific structure, where data is provided at different locations and regions, which can be aligned with each other to form a complete representation of a page of a specific page format.

[0048] At 430, pre-training is performed based on evaluating masked word tokens according to individual masking ratios. The individual masking ratios are determined based on applying a dynamic window that is adjustable for word token counts (occurrences) for each of the unlabeled documents. The window applied is not a fixed window and is adjustable in length for different documents in the training data. In some instances, the documents can be scanned pages, where different documents can have different numbers of words that can be organized in different ways (with different spatial relationships) at their respective pages.

[0049] In some instances, logical entities can be extracted from text (including text in images) and spatial text information obtained from executing an OCR program. When a scanned document is obtained, the text and spatial information can be extracted by applying rules that define rule-based extraction heuristics, neural network-based extraction models (e.g., DocumentReader, Chargrid, or Charmer), or post-processing logic (such as sender address coordination and matching). In response to obtaining the text and spatial information, the information can be saved in local storage, provided for use by a user, or provided to an embedded solution of a data extraction service for further document processing.

[0050] Pre-training of the first document base model includes evaluating word tokens in each unlabeled business document, wherein masking is applied according to individual masking ratios determined for the word tokens. The masking ratios can be calculated to reflect the information relevance of the words in the document, wherein those word tokens associated with higher relevance can be masked more frequently than those word tokens associated with lower relevance. Thus, the individual masking ratios indicate the respective information relevance of the word tokens.

[0051] In some instances, pre-training includes determining masking ratios for word tokens. Figure 3 As discussed in connection with pre-training 300 of the invention, masking every word in a document at the same rate during training can be associated with disadvantages in training performance (e.g., inefficiencies in training), and thus can reduce the accuracy of the trained model. This disadvantage of randomly masking tokens during pre-training can be overcome by considering the informativeness of certain words and applying masking rates to the words based on this consideration of an informativeness metric.

[0052] In some examples, the probability that words can appear together in a set window of words can be calculated. For example, a point-wise mutual information (PMI) metric can be calculated as a statistical measure to determine the probability that two words appear together in a predefined window divided by the probability that those words (w1 and w2) appear independently. The PMI metric can be calculated according to the following formula (2):

[0053]

[0054] In some embodiments, the PMI metric calculated according to formula (2) can be used to determine how "surprising" the co-occurrence of two words in a sequence is. p(w1, w2) represents the probability of two words appearing together in a sequence, while p(w1) or p(w2) represents the probability of a word appearing in a sequence.

[0055] In some implementations, a measure of information relevance of words in a defined sequence can be computed based on a PMI metric. In some instances, a measure of information relevance of a word can be computed by summing a PMI score that is computed for a combination of all masked and unmasked words in a sequence (e.g., all words in a document that forms a page). In some instances, a PMI score can be computed using a dynamic window size corresponding to the length of a document page, and the PMI score can be used to define the probability of masking word tokens during pre-training of a document-based model.

[0056] In some examples, when pre-training the first document based model, a sequence of word tokens for training can be defined according to a dynamic window that is adjustable for different lengths of different business documents used as training data for the pre-trained first document based model. During training, masking is applied, which is performed according to an individual masking ratio determined for each word token.

[0057] In some instances, a process for obtaining individual masking rates for each word token can be deployed and executed to provide the rates as input for determining masking probabilities for words during pre-training at 430. Based on such individual masking rates, masking of some words can be reduced compared to other words. In particular, the masking probabilities of words that are less important to the information extraction task (e.g., words with relatively low information relevance measures) can be reduced. For example, stop words can be considered to be relatively less important words for providing information relevance, and thus can be masked less frequently during training without negatively affecting the quality of training to infer the labels of word tokens.

[0058] In some embodiments, a process for obtaining an individual masking ratio for each word in a word token sequence defined according to a dynamic window with adjustable size for a training business document can be calculated by the following steps, including:

[0059] (i.) computing a matrix with PMI scores for all words in the dataset, including word tokens defined based on generated business documents as training data;

[0060] (ii.) for each word token sequence in the document, mask a specific portion (e.g., 15%) of the word tokens n times, where n corresponds to the number of sample sequences used during training;

[0061] (iii.) for each masked sequence, calculate the informativeness score by summing the PMI scores calculated between all masked and unmasked words;

[0062] (iv.) sorting the masking sequences in descending order according to their informativeness scores;

[0063] (v) taking the most informative masking sequence (e.g., with the highest ranking) and counting the masked tokens; and

[0064] (vi.) Divide the masked counts by the token frequency in the dataset.

[0065] Based on the execution of the above steps, the results achieved in the last step can be used to assign individual masking rates to each word token. In some instances, in order to make a masking decision for each sequence, a sample number of sequences with different masked words can be used when determining the most informative masking sequence. For example, n can be selected to be equal to 20 sample sequences, where the number of sample sequences determined can be determined taking into account the computational cost of performing the sorting and determining the masking rates while executing the algorithm to produce accurate results.

[0066] The PMI matrix calculated in step (i.) takes into account the special word distribution in the document compared to natural language. Instead of having a fixed window to count the co-occurrence of two words on the sentence or paragraph level, a dynamic window is applied to the evaluation. The dynamic window is adjustable for the number of tokens on the current document page. Thus, the word distribution of the document specific language corpus (specific to business documents) is modeled, and the masking decision is adapted to the information content of the words on the business documents. Having more intelligent masking decisions during model pre-training leads to higher accuracy of extracted entities and improves model convergence behavior. This can also lead to shorter training time and less computational cost.

[0067] Using individual masking rates can support improved pre-training and a reduction in the number of processing times for masking sequences, including masking words that do not have high information relevance. Deploying this technique as a pre-processing step for training supports masking during training that can be performed faster and with fewer samples. Masking of each word token does not have to have the same rate, but rather relies on the information relevance of the words to optimize the determination of masking and training as a whole, while achieving accurate training in a cost-effective manner without having to iterate through all possible masking combinations.

[0068] In some instances, by deploying this technique for calculating the masking ratio, the masking probability of words that are less important to the information extraction task (such as stop words) is reduced. For example, experiments show that although the word "limit" has a high frequency, the masking ratio of this word token is less than 10% because it does not have high relevance at the document level. On the other hand, increasing the masking probability of words such as "phone" can be a logical decision because these tokens are often associated with extracted document entities (such as the purchaser's phone number) and such entities have high relevance at the document level.

[0069] Based on applying the MLM pre-training process at 430, the first document base model is trained to reduce the loss when determining masked word tokens from unlabeled documents. At 440, the parameters of the learned first document base model are provided for initializing the second model. The second model is a model for document information extraction, which can be initialized and then trained based on labeled business documents as second training data.

[0070] At 450, a second document information extraction model is initialized based on the pre-trained first document base model, and the second document base model is trained by fine-tuning according to an adversarial training technique. Adversarial training can be as described in relation to Figure 6 and Figure 7 Performed as described.

[0071] At 460, based on the training, a classifier is generated on top of the second document information extraction model. In some instances, the second document information extraction model can be considered a domain-specific neural network extraction model because it is relevant to the details of business documents that can have different structures and word densities on document pages.

[0072] At 470, in response to receiving a new scanned business document, the generated classifier can be invoked to extract information from the new scanned business document and provide data regarding data objects identified in the document, such as a sender address from an invoice document.

[0073] In some instances, while masking as described above is determined at the word token level, masking can be performed on sub-word tokens. In some cases, masking performed at the sub-word token level can be defined for specific types of entities in a document, and can be independent of the type of the entire document. For example, sub-word tokens can help with special entities such as date and time data or numbers (e.g., invoice numbers). The preprocessing step for determining individual masking rates can be performed at the sub-word level, and the masking rates for sub-words can be used to perform training based on unlabeled business documents.

[0074] In some instances, when word tokens are masked, the informativeness score of the sequence can be calculated. In some instances, spatial information can also be considered. For example, the spatial distance of words can be used to determine the amount of information of words in a document. For example, when performing the summation of PMI scores in the above step (iii.), distance-based weights can be applied. For example, distance-based weights can be calculated based on any Minkowski distance (such as Euclidean distance and Manhattan distance) or similar or derived distance metrics. In some cases, knowledge of word tags posterior to informativeness scores can also be considered when determining masking scores, such as whether a word is already part of an entity extracted by a different extraction model. For example, key-value pairs can be defined and assigned higher informativeness scores. In one example, the solution can assign key-value pairs with higher informativeness scores to the informativeness scores of other word tokens that do not belong to a predefined pair, such as "Invoice#:" and "123234". In some other examples, the informativeness score of a predefined key-value pair can be defined as a higher score by adding a predefined "add" value (e.g., a percentage or other value on a given scale).

[0075] Figure 5 is a flow chart representing a process 500 of fine-tuning a trained model to extract information from a document based on a virtual adversarial technique according to an embodiment of the present disclosure.

[0076] At 510, a document information extraction model is initialized based on a pre-trained model, for example, as described in relation to Figure 4 The initialization can be based on weights provided from a first (pre-trained) model, the first (pre-trained) model being pre-trained on training data comprising unlabeled documents as first training data.

[0077] At 520, fine-tuning of the information extraction model is performed based on training with labeled documents (e.g., business documents or other scanned documents having a specific structure and including header and line item data) as second training data. The benchmark business documents are labeled and evaluated according to the virtual adversarial training.

[0078] At 530 , a classifier is generated on top of the second document information extraction model based on the performed fine-tuning.

[0079] At 540, in response to receiving the scanned business document for information extraction, the generated classifier can be invoked to obtain information of data objects identified in the scanned business document. For example, the name and address of the sender of the invoice can be automatically determined.

[0080] Figure 6 is a block diagram representing a process 600 for applying virtual adversarial training (VAT) to train a model for extracting information from business documents, according to an embodiment of the present disclosure.

[0081] In some instances, it is possible to fine-tune a pre-trained model, such as by Figure 3 and Figure 4 The models provided by the pre-training explained, for example, as Figure 3 The fine-tuning 340 discussed above can be based on initializing the model according to the parameters provided by the pre-trained model, and then fine-tuning the trained model based on the labeled data and according to the virtual adversarial technique described in the present disclosure.

[0082] In some instances, VAT can be applied in the context of training in the business document domain and based on input from a pre-trained model (for initialization). The pre-trained model can be trained on unlabeled data that is masked based on a masking probability that corresponds to the information relevance of the masked word tokens over a dynamic window. Therefore, VAT can be applied in a more efficient manner by starting training from a better positioned starting point for accurately identifying the structure of business documents. By applying the VAT technique, VAT aims to improve the robustness of the model by adding small perturbations to the input to have the same output distribution between task-specific examples and adversarial examples.

[0083] In some instances, training data for VAT can be obtained, and input embedding 610 can be defined for input business documents marked as training data. Input embedding can be defined to include text embedding (e.g., word tokens) and spatial embedding. Text embedding and spatial embedding with tokens can be obtained from the OCR engine based on the recognition performed on the scanned document provided as labeled training data. Text embedding and spatial embedding are cascaded to provide cascaded embedding 620, which will be used to generate predictions during training, which can be compared to fine-tune the model to perform accurate information extraction from business documents. Layer normalization can be applied on these cascaded embeddings to generate normalized embedding 630. The cascaded embedding 620 is also perturbed with a VAT target, and the labels of the models trained for both the original embedding and the perturbed embedding are predicted. Based on the normalized embedding 630, a model prediction 650 is generated, and based on the adversarial perturbation 640, a model prediction 660 is generated. Predictions 650 and 660 are compared according to Kullback-Leibler (KL) divergence 670. KL divergence 670 can be used as a comparison measure of the probability distributions of two predictions (650 and 660) from two models trained based on normalized embedding 630 and adversarial perturbation 640.

[0084] In some examples, the adversarial perturbation 640 is calculated by maximizing the KL divergence between the regular model output distribution q(y|x) and the adversarial output distribution p(y|x+r,θ), as in the following equation (3):

[0085] (3) L adv (x,θ)=l s [q(y|x),p(y|x+r,θ)],

[0086] where y is the model output, x is the model input, r is the adversarial perturbation, and l s is a symmetric version of the KL-divergence with two distributions P and Q, where:

[0087] (4) l s (P,Q)=DKL (P||Q)+D KL (Q||P)

[0088] Thus, the perturbation is not random or Gaussian noise, but is calculated to ensure that it is small enough not to change the inherent meaning of the input by maximizing the KL divergence. In some cases, the noise added for the adversarial perturbation 640 can point to areas of the model where the model is most sensitive or uncertain to expose vulnerabilities during training. In contrast to standard adversarial training, in which the direction that deviates most from the correct classification is selected, the process 600 applies a label-independent VAT technique and therefore only "virtually" adversarial.

[0089] In some instances, the output as the prediction from the adversarial example is compared with the original input (normalized only) based on the KL-divergence metric that defines the VAT loss.

[0090] Figure 7 700 is a block diagram of a process 700 for performing virtual adversarial training in the context of business document information extraction according to an embodiment of the present disclosure. The process 700 can be performed in a VAT technique implementation, such as with respect to Figure 5 and Figure 6 as described.

[0091] VAT techniques rely on using labeled input business documents processed twice by the same model, e.g. a model initialized based on parameters provided from a pre-training process, e.g., as described about Figure 3 and Figure 4 In some instances, the training labeled data can include a document 701 that is a business document and is associated with a labeled version (705) of the document 701 and an adversarial example 710 that is a version of the document. In some cases, the obtained document 701 can be pre-labeled and obtained as the labeled document 705.

[0092] The processing of labeled input documents (such as document 705) is performed twice. First, the processing is based on the text and spatial embeddings extracted from the original business document, and then, the processing is based on the perturbed embeddings. This processing corresponds to the processing of labeled documents 705 and adversarial examples 710 (with perturbations). Figure 7 The adversarial training shown includes a process of providing adversarial examples as instances that are introduced into the document information extraction model and are marked as threats because they are designed to cause the model to make incorrect predictions. Therefore, the classification of the model is tested based on such adversarial examples, and the performance of the model is evaluated to improve the predictive ability of the model. The output distributions resulting from the predictions based on the processing of the input "as is" and the perturbed input can be presented as shown in 720 and 725, respectively. At 730, the output distribution can be generated using the method described in relation to Figure 6The KL divergence in question is used to compare the distribution of results, i.e., correct classifications and incorrect classifications. The comparison of the distributions can be a scalar number, which can be viewed as a new component of the training loss function in the context of VAT training as a fine-tuning process of a pre-trained model. Thus, the KL divergence 730 can be used together with the cross entropy loss 740 to determine the final loss 750 from the execution of the processing. Figure 3 The cross entropy loss 740 is discussed in more detail and is calculated based on the true label data 735 and the data obtained from training based on the labeled documents 705.

[0093] In some examples, the resulting scalar obtained from the comparison (as output from 730) can be added to the traditional cross entropy loss (as determined at 740) to determine the final loss 750. The cross entropy loss 740 can be determined from a conventional classification task based on the original input training data (including document 705) processed by the trained model to obtain the resulting distribution at 720 and the true label of the input data (e.g., labeled document 705).

[0094] Applying VAT techniques can provide several benefits. The approach based on applying adversarial training to fine-tune a pre-trained model is conceptually model-agnostic, i.e., it can be deployed to different neural network architectures and supports robust and accurate results.

[0095] In some embodiments, perturbations that can be defined for embeddings (e.g., text embeddings) can be entity-aware. In this way, perturbations can be designed to specifically target entities of a document, such as a name, date, or address, by using the predictions of an existing model. In some cases, constraints can be applied to perturbations of those entities to avoid the perturbations becoming invalid or meaningless.

[0096] According to the present invention, the VAT technique is applied to the field of two-dimensional (2D) document information extraction and adapted to the task of identifying labels of masked tokens based on the processing of original output and adversarial output. To this end, adversarial perturbations are applied to the cascade of text embedding and spatial embedding, and thus, perturbations are applied to document representations. In some instances, the resulting fine-tuning performance may exceed previous methods in terms of extraction accuracy and training stability.

[0097] Figure 88 is a schematic diagram of an example computer system 800 that can be used to perform embodiments of the present disclosure. For example, the computer system 800 can be included in any or all of the server components discussed herein. The computer system 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. The components 810, 820, 830, and 840 are interconnected using a system bus 850. The processor 810 can process instructions for execution within the computer system 800. In some embodiments, the processor 810 is a single-threaded processor. In some embodiments, the processor 810 is a multi-threaded processor. The processor 810 can process instructions stored in the memory 820 or on the storage device 830 to display graphical information of a user interface on the input / output device 840.

[0098] The memory 820 stores information within the computer system 800. In some embodiments, the memory 820 is a computer-readable medium. In some embodiments, the memory 820 is a volatile memory unit. In some embodiments, the memory 820 is a non-volatile memory unit. The storage device 830 can provide mass storage for the system 800. In some embodiments, the storage device 830 is a computer-readable medium. In some embodiments, the storage device 830 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device. The input / output device 840 provides input / output operations for the computer system 800. In some embodiments, the input / output device 840 includes a keyboard and / or a pointing device. In some embodiments, the input / output device 840 includes a display unit for displaying a graphical user interface.

[0099] The described features can be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device for execution by a programmable processor), and the method operations can be performed by a programmable processor executing a program of instructions to perform the functions of the described implementation by operating on input data and generating output. The described features can advantageously be implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform some activity or produce some result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or another unit suitable for use in a computing environment.

[0100] As examples, suitable processors for executing instruction programs include both general and special purpose microprocessors, and the sole processor or one of multiple processors of any type of computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer can include a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer can also include one or more mass storage devices for storing data files, or be operably coupled to communicate with one or more mass storage devices; such devices include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into an ASIC (application-specific integrated circuit).

[0101] To provide for interaction with a user, features can be implemented on a computer having a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer.

[0102] Features can be implemented in a computer system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server or an Internet server, or includes a front-end component, such as a client computer with a graphical user interface or an Internet browser, or any combination thereof. The components of the system can be connected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, a LAN, a WAN, and the computers and networks that form the Internet.

[0103] A computer system can include a client and a server. The client and server are generally remote from each other and generally interact through a network (such as the described network). The relationship between the client and the server arises by means of computer programs running on the respective computers and having a client-server relationship to each other.

[0104] In addition, the logic flows depicted in the accompanying drawings do not require the particular order or sequential order shown to achieve the desired results. In addition, other operations can be provided, or operations can be omitted from the described processes, and other components can be added to or removed from the described systems. Therefore, other implementations are within the scope of the appended claims.

[0105] A number of embodiments of the present disclosure have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0106] In view of the implementation of the above-mentioned subject matter, the present application discloses the following list of examples, wherein one feature of a separate example or more than one feature combination of the examples and optionally combined with one or more features of one or more additional examples are additional examples that also fall within the disclosure of the present application.

[0107] Example

[0108] Although the present application is defined in the appended claims, it is to be understood that the invention may also (alternatively) be defined according to the following examples:

[0109] Self-supervised pre-training of document information extraction models with informative word masking

[0110] Example 1. A computer-implemented method comprising:

[0111] Obtaining first training data based on a plurality of unlabeled documents for training a first document information extraction model;

[0112] pre-training a first model based on a dynamic window of word token counts adjustable for each of a plurality of unlabeled documents, wherein the pre-training includes evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate respective information relevance of the word tokens; and

[0113] A pre-trained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.

[0114] Example 2. The method of Example 1, wherein obtaining training data comprises performing optical character recognition to determine that a word is part of at least some of a plurality of unlabeled documents.

[0115] Example 3. The method according to Example 1 or Example 2, wherein pre-training the first model comprises:

[0116] tokenizing each of the plurality of unlabeled documents into a corresponding sequence of word tokens and spatial embeddings;

[0117] computing individual masking ratios for word tokens that identify information relevance of corresponding word tokens in a plurality of unlabeled documents; and

[0118] Representations of a first model are learned as a document base model, wherein the first model includes weights as parameters of the model defining connection strengths across different layers in the model.

[0119] Example 4. The method of Example 1, wherein the word token comprises at least one of a single word or a sub-word.

[0120] Example 5. The method of any of the preceding examples, wherein pre-training comprises:

[0121] defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and

[0122] Get the individual masking ratio for each word token in the word token sequence.

[0123] Example 6. The method of Example 5, wherein obtaining training data comprises performing text recognition on a plurality of unlabeled documents to obtain documents including the recognized text portions, and wherein obtaining the individual masking ratios comprises:

[0124] Compute a matrix with pointwise mutual information (PMI) scores for the word tokens in the training data;

[0125] For each sequence of word tokens in the document, masking specific portions of the word tokens, wherein each sequence is masked n times, and wherein n corresponds to the number of sample sequences used during training;

[0126] For each masked sequence, the informativeness score was calculated by summing the PMI scores calculated between all masked and unmasked words;

[0127] The masked sequences are ranked according to their corresponding informativeness scores;

[0128] selecting the masking sequence associated with the highest informativeness score according to the ranking as the most informative masking sequence; and

[0129] Divide the masked counts by the frequency of the corresponding word token in the training data.

[0130] Example 7. The method according to any of the preceding examples, comprising:

[0131] Training a second document information extraction model based on the marked document for document information extraction, wherein the training comprises:

[0132] The second document information extraction model is initialized based on the input provided from the pre-trained first model.

[0133] Example 8. The method according to any of the preceding examples, comprising:

[0134] Adversarial training is performed on a second document information extraction model, which is initialized based on the pre-trained first model.

[0135] Example 9. The method according to any of the preceding examples, comprising:

[0136] Initializing a second model based on weights provided from a pre-trained first model;

[0137] performing fine-tuning of the pre-trained first model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and

[0138] Based on the performed fine-tuning, a classifier is generated on top of the second document information extraction model.

[0139] Example 10. The method of Example 9, wherein performing fine-tuning comprises:

[0140] creating text and spatial embeddings from a marked-up business document, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the marked-up business document;

[0141] concatenating the textual and spatial embeddings to form a document representation;

[0142] Applying layer normalization on the concatenated textual and spatial embeddings; and

[0143] The concatenated textual and spatial embeddings are perturbed according to the objective of VAT to predict labels of the original embeddings and the perturbed embeddings, where performing VAT includes:

[0144] processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation;

[0145] Compare the distribution of documents with the concatenated embeddings to the distribution of documents with the perturbed embeddings; and

[0146] and

[0147] A result is obtained to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data.

[0148] Example 11. A system comprising:

[0149] one or more processors; and

[0150] One or more computer-readable memories coupled to one or more processors and storing instructions executable by the one or more processors to perform the method of any of Examples 1 to 10.

[0151] Example 12. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1 to 10.

[0152] Virtual Adversarial Training of Document Information Extraction Models

[0153] Example 1. A computer-implemented method comprising:

[0154] pre-training a domain-specific neural network extraction model based on first training data, wherein the training data comprises unlabeled documents, wherein the pre-training is performed based on informative word masking;

[0155] performing fine-tuning of a pre-trained domain-specific neural network extraction model based on labeled business documents as second training data, wherein the labeled business documents are processed according to a virtual adversarial training (VAT) technique; and

[0156] Based on the performed fine-tuning, a classifier associated with the domain-specific neural network extraction model is generated.

[0157] Example 2. A computer-implemented method comprising:

[0158] Initializing a second document information extraction model based on weights provided from a first model, the first model being pre-trained on training data including unlabeled business documents as first training data;

[0159] performing fine-tuning of a second document information extraction model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); and

[0160] Based on the performed fine-tuning, a classifier for classification for information extraction is generated.

[0161] Example 3. The method according to Example 2, comprising:

[0162] In response to receiving a scanned business document for information extraction, the generated classifier is invoked to obtain information of data objects identified in the scanned business document.

[0163] Example 4. The method of example 2 or example 3, wherein performing fine tuning comprises:

[0164] Creating text and spatial embeddings from a marked business document, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the marked business document;

[0165] Concatenate textual and spatial embeddings to form a document representation;

[0166] Applying layer normalization on the concatenated textual and spatial embeddings; and

[0167] The concatenated textual and spatial embeddings are perturbed according to the objective of VAT to predict the labels of the original and perturbed embeddings.

[0168] Example 5. The method of Example 4, wherein performing fine-tuning comprises:

[0169] processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation;

[0170] comparing the distribution of documents with the concatenated embeddings to the distribution of documents with the perturbed embeddings; and

[0171] A result is obtained to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data.

[0172] Example 6. The method of any of the preceding examples, wherein pre-training is applied based on a dynamic window of adjustable word token size for each unlabeled document of the unlabeled business documents.

[0173] Example 7. The method of any of the preceding examples, wherein pre-training comprises evaluating word tokens in each of a plurality of unlabeled business documents to which masking is applied according to individual masking ratios determined for the word tokens.

[0174] Example 8. The method of Example 7, wherein the individual masking ratios indicate respective information relevance of the word tokens, and wherein pre-training comprises:

[0175] defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled business documents after performing text recognition; and

[0176] Get the individual masking ratio for each word token in the word token sequence.

[0177] Example 9. A system comprising:

[0178] one or more processors; and

[0179] One or more computer-readable memories coupled to one or more processors and storing instructions, the instructions being executable by the one or more processors to perform the method of any one of Examples 1 to 8.

[0180] Example 10. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1 to 8.

Claims

1. A computer-implemented method comprising: Obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction; pre-training a first model based on a dynamic window of word token counts adjustable for each of a plurality of unlabeled documents, wherein the pre-training comprises: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate respective information relevance of the word tokens; and A pre-trained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.

2. The method according to claim 1, wherein: Obtaining training data includes performing optical character recognition to determine words as part of at least some of the plurality of unlabeled documents.

3. The method according to claim 1, wherein: The pre-trained first model includes: tokenizing each of the plurality of unlabeled documents into a corresponding sequence of word tokens and spatial embeddings; computing individual masking ratios for word tokens that identify information relevance of corresponding word tokens in a plurality of unlabeled documents; and Representations of a first model are learned as a document base model, wherein the first model includes weights as parameters of the model defining connection strengths across different layers in the model.

4. The method according to claim 1, wherein: A word token includes at least one of a single word or a sub-word.

5. The method according to claim 1, wherein: Pre-training includes: defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and Get the individual masking ratio for each word token in the word token sequence.

6. The method according to claim 5, wherein: Obtaining training data includes performing text recognition on a plurality of unlabeled documents to obtain documents including the recognized text portions, and wherein obtaining the individual masking ratios includes: Compute a matrix with pointwise mutual information (PMI) scores for the word tokens in the training data; For each sequence of word tokens in the document, masking specific portions of the word tokens, wherein each sequence is masked n times, and wherein n corresponds to the number of sample sequences used during training; For each masked sequence, the informativeness score was calculated by summing the PMI scores calculated between all masked and unmasked words; The masked sequences are ranked according to their corresponding informativeness scores; selecting the masking sequence associated with the highest informativeness score according to the ranking as the most informative masking sequence; and Divide the masked counts by the frequency of the corresponding word token in the training data.

7. The method according to claim 1, comprising: Training a second document information extraction model based on the marked document for document information extraction, wherein the training comprises: The second document information extraction model is initialized based on the input provided from the pre-trained first model.

8. The method according to claim 1, comprising: Adversarial training is performed on a second document information extraction model, which is initialized based on the pre-trained first model.

9. The method according to claim 1, comprising: Initializing a second model based on weights provided from a pre-trained first model; performing fine-tuning of the pre-trained first model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to virtual adversarial training (VAT); as well as Based on the performed fine-tuning, a classifier is generated on top of the second document information extraction model.

10. The method according to claim 9, wherein: Execution fine-tuning includes: creating text and spatial embeddings from a marked-up business document, wherein the embeddings are created as tokens obtained after performing optical character recognition (OCR) on the marked-up business document; concatenating the textual and spatial embeddings to form a document representation; Applying layer normalization on the concatenated textual and spatial embeddings; and The concatenated textual and spatial embeddings are perturbed according to the objective of VAT to predict labels of the original embeddings and the perturbed embeddings, where performing VAT includes: processing each labeled document twice by the model, wherein the processing comprises i) processing each document with the perturbed concatenated embedding, and ii) processing each document without the concatenated embedding without the perturbation; comparing the distribution of documents with the concatenated embeddings to the distribution of documents with the perturbed embeddings; and A result is obtained to be used as a component of a training loss value to be added to a cross entropy loss calculated based on the second training data.

11. A non-transitory computer-readable medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: Obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction; pre-training a first model based on a dynamic window of word token counts adjustable for each of a plurality of unlabeled documents, wherein the pre-training comprises: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate respective information relevance of the word tokens; and A pre-trained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.

12. The non-transitory computer readable medium of claim 11, wherein: Obtaining training data includes performing optical character recognition to determine words as part of at least some of the plurality of unlabeled documents.

13. The non-transitory computer readable medium of claim 11, wherein: The pre-trained first model includes: tokenizing each of the plurality of unlabeled documents into a corresponding sequence of word tokens and spatial embeddings; computing individual masking ratios for word tokens that identify information relevance of corresponding word tokens in a plurality of unlabeled documents; and Representations of a first model are learned as a document base model, wherein the first model includes weights as parameters of the model defining connection strengths across different layers in the model.

14. The non-transitory computer readable medium of claim 11, wherein: A word token includes at least one of a single word or a sub-word.

15. The non-transitory computer readable medium of claim 11, wherein: Pre-training includes: defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and Get the individual masking ratio for each word token in the word token sequence.

16. A computer-implemented system comprising: one or more processors; as well as one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon, the instructions executable by the one or more processors to perform operations comprising: Obtaining first training data based on a plurality of unlabeled documents for training a first model for document information extraction; pre-training a first model based on a dynamic window of word token counts adjustable for each of a plurality of unlabeled documents, wherein the pre-training comprises: evaluating word tokens in each of the plurality of unlabeled documents to which masking is applied based on individual masking ratios determined for the word tokens, and wherein the individual masking ratios indicate respective information relevance of the word tokens; and A pre-trained first model is provided for initializing a second document information extraction model to be trained based on labeled documents as second training data.

17. The system of claim 16, wherein: Obtaining training data includes performing optical character recognition to determine words as part of at least some of the plurality of unlabeled documents.

18. The system of claim 16, wherein: The pre-trained first model includes: tokenizing each of the plurality of unlabeled documents into a corresponding sequence of word tokens and spatial embeddings; computing individual masking ratios for word tokens that identify information relevance of corresponding word tokens in a plurality of unlabeled documents; and Representations of a first model are learned as a document base model, wherein the first model includes weights as parameters of the model defining connection strengths across different layers in the model.

19. The system of claim 16, wherein: A single token includes at least one of a single word or a sub-word.

20. The system of claim 16, wherein: Pre-training includes: defining a sequence of word tokens for each of the plurality of unlabeled documents, wherein the sequence has a different length corresponding to a count of word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and Get the individual masking ratio for each word token in the word token sequence.

Images