Fine-grained analysis based on distribution changes of document images

Abstract

The application provides a fine-grained analysis based on document image distribution change, relates to the field of document understanding in cross-modal understanding and transfer learning. The fine-grained analysis based on the document image distribution change develops a distribution out (OOD) benchmark named Do- GOOD, which is used for fine-grained analysis of document image related tasks, and the Do- GOOD benchmark defines the underlying mechanism that causes different distribution transfer, which is specific image distribution change and specific text distribution change. In the application, various distribution changes of the document image are analyzed from the aspects of the image, the text and the layout, the analysis and processing can be more detailed, 5 advanced pre-training VDU models in the generated OOD test and 2 common OOD algorithms are evaluated and compared in different document image tasks, and the proposed Do- GOOD benchmark, empirical research and in-depth analysis will be beneficial to future research, so as to improve the robustness of the pre-training VDU model.

Description

Technical Field

[0001] This invention relates to the field of document understanding in cross-modal understanding and transfer learning, specifically to a fine-grained analysis based on document image distribution changes, used to study the robustness of existing document pre-training models when test samples undergo different distribution changes. There is a significant performance gap between the in-distribution (ID) and OOD settings of document images, and the fine-grained analysis of distribution changes reveals the vulnerability of existing pre-trained VDU models and OOD generalization algorithms. Background Technology

[0002] Document images (such as invoices and lease agreements) often contain rich contextual text and structural information, and are common in modern work and life environments. Automatic processing and understanding of document images has a wide range of use cases in real-world scenarios, such as document image classification, document image information extraction, and document visualization question answering. In recent years, many pre-training techniques for document image understanding have been proposed and proven effective for various types of documents.

[0003] While these models have achieved encouraging results, there is no guarantee that models designed with the same training and testing data distributions will continue to perform well when the testing and testing data distributions differ. However, most document datasets are designed based on the IID assumption, meaning that the training and testing data come from the same distribution.

[0004] To enable document classification models to handle out-of-distribution (OOD) document images, Larson et al. proposed a new OOD testing platform based on the widely used document classification benchmark dataset, RVL-CDIP. This RVL-CDIP OOD benchmark is only used to develop and evaluate the robustness of document image classification methods; it only requires the model to have the ability to model coarse-grained information on document images. Although the RVL-CDIP OOD benchmark reveals that image information is crucial for document classification, it plays a relatively minor role in other document imaging tasks, such as information extraction. For example, on the NER task of the state-of-the-art pre-trained visual document understanding (VDU) model LayoutLMv3, the model's predicted F1 score did not change when image x was masked to blank at different scales. Furthermore, document images naturally possess three distinct features: image, text, and layout information. Tasks such as extracting information from document images and answering document visualization questions require a fine-grained understanding of the complex interactions between image, text, and layout information.

[0005] Therefore, those skilled in the art provide a fine-grained analysis based on changes in document image distribution to address the problems mentioned in the background section. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a fine-grained analysis based on document image distribution variations. It enables fine-grained analysis of various distribution variations in document images from the perspectives of image, text, and layout, allowing for more detailed analysis and processing. It evaluates and compares five state-of-the-art pre-trained VDU models and two common OOD algorithms in different document image tasks. Furthermore, its proposed Do-GOOD benchmark, empirical research, and in-depth analysis will benefit future research to improve the robustness of pre-trained VDU models.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A fine-grained analysis based on document image distribution variations is presented. A Do-GOOD out-of-distribution (OOD) benchmark is developed for fine-grained analysis of document image-related tasks. The Do-GOOD benchmark defines the underlying mechanisms leading to different distribution shifts, namely specific image distribution variations, specific text distribution variations, and specific layout distribution variations. It includes nine OOD datasets covering three document image-related tasks: document visual information extraction, document visual classification, and document visual question answering. The robustness of visual document image understanding (VDU) models is evaluated, and fine-grained analysis is performed on these OOD datasets against five existing VDU pre-trained models and two common OOD generalization algorithms.

[0011] The Do-GOOD out-of-distribution (OOD) benchmark, using existing datasets such as FUNSD, prepares training and test samples under the in-distribution (IID) assumption. Given a training input x, the data distribution p... train The goal of the document image model f is to minimize the risk, as shown below:

[0012]

[0013] in, It is a loss function specific to the task. Due to confounding factors such as selection bias and random data splitting during data collection, in practice, training and test data rarely follow the same data distribution (i.e., p). trajn ≠p test Since the distributions of training and test data are different, a model trained on the training data is expected to generalize well to the test data. This necessitates carefully designing OOD benchmarks to accurately evaluate the model's generalization ability.

[0014] The specific image distribution variation, the image distribution shift, has two background variants: natural image background and distorted image background. Formally, y image Using finite sets Define the image. For training, y image The attribute is raw. When testing on undistributed data with a natural image background, the attribute y is set... image =y natural To obtain the attribute p natural (y 1：K The marginal distribution of latent factors and attributes is used to induce the joint distribution of latent factors and attributes.

[0015] p natural (z, y) 1：K )=p(z|y 1：K )p natural (y 1：K ),

[0016] Subsequently, the input data used for the test can be obtained, and its joint distribution is:

[0017] p natural (x, y) 1：K )=∫ p(x|z) p natural (z, y) 1：K ),

[0018] On the other hand, a method similar to generating test data for natural images can be used to deduce that the background is a distorted image p. distorted (z, y) 1：K The non-distributed test set.

[0019] To simulate a real-world scenario where the input document image may contain problematic text due to OCR errors, the specific text distribution variation employs two text attack strategies: (1) Bert-Attack; (2) Word-Swap. Formally, y text Using finite sets Define the text. For training, attribute y text It is the original. Referring to the analysis of specific image distribution variations in OOD benchmark tests, a similar method can be used to obtain BERT-Attack p. generation (x, y) 1：K ) and Word-Swap p swap (x, y) 1：K The out-of-distribution test data.

[0020] The specific layout distribution change involves two layout operations: merging and moving. The merge operation aims to study the impact of changing layout information from a fine-grained level to a coarse-grained level while preserving image and text information. The move operation is used to study the effect of moving content to different positions, making adjacent information on the content of a specific bounding box effective. Formally, y layout Using finite sets Define the layout. For training, y layout The attributes are raw. Similarly, by analyzing image-specific OOD and text-specific OOD benchmarks, the merge operation p can also be obtained. merge (x, y) 1：K OOD test data and Move operation p swap (x, y) 1：K OOD test data.

[0021] The document visual information extraction task, specifically for visual document information extraction, primarily utilizes the FUNSD dataset to generate the OOD dataset. FUNSD is a dataset sampled from the RVL-CDIP dataset for understanding noisy scanned forms, consisting of 199 documents (149 for training and 50 for testing) and 9743 semantic entities. The task of FUNSD is sequence labeling, which aims to assign labels to words.

[0022] FUNSD-L is a variant of FUNSD that includes OOD samples generated through two strategies based on layout-specific distribution transitions (merging and moving). As mentioned above, the Move operation is based on semantic strength determined by the model itself. Specifically, bounding boxes in the document image are randomly shuffled, and this process is repeated 30 times. For text content, fewer model prediction errors indicate stronger semantics.

[0023] FUNSD-T is a variant of FUNSD that contains OOD samples generated by the two text attack methods described in the specific text distribution variation above;

[0024] FUNSD-R is a real-world OOD dataset variant of FUNSD. It first samples data examples from the large-scale document classification dataset RCL-CDIP, then observes and selects data examples with a distribution different from FUNSD. These selected examples are then manually labeled. FUNSD-R contains 50 document images.

[0025] FUNSD-H is a modified version of the FUNSD dataset, creating an OOD dataset variant of FUNSD with artificial intervention. In practice, variations are constructed by moving some weak text entities to create layouts and image shifts, or by adding semantically weak text around semantically strong content. Finally, 50 OOD samples were obtained.

[0026] The document visual classification task, RVL-CDIP, is a document classification dataset used to predict the category of a given document. It includes 400,000 data examples across 16 categories, divided into 320,000 training samples, 40,000 validation samples, and 40,000 test samples.

[0027] RVL-CDIP-T is one of the OOD dataset variants of RVL-CDIP, containing OOD samples generated by two text attack methods.

[0028] As a variant of RVL-CDIP, RVL-CDIP produces OOD samples through two layout-specific distribution shifts. RVL-CDIP-I is a distorted OOD variant generated by a specific OOD variant of a natural image and an image distribution shift of RVL-CDIP-I2.

[0029] The document visual question answering task, DocVQA, is a dataset used to predict the answer to a given document image and question. The model needs to understand the content of the document and learn to reason about it. The original DocVQA data consists of 10,194 / 1,286 / 1,287 images, containing 39,463 / 5,349 / 5,188 questions for training / validation / testing.

[0030] DocVQA-T is an OOD dataset variant of DocVQA. To construct DocVQA-T, text, questions, and answers are first collected from OCR results and the Microsoft-READ API. Then, OOD samples generated by two text attack methods are obtained.

[0031] The robustness of pre-trained VDU models (large models) fine-tuned for downstream tasks using the Do-GOOD benchmark was evaluated using five existing VDU pre-trained models and two common OOD generalization algorithms. The large models include pre-trained models for text and layout patterns: BROS and LiLT; and pre-trained models for text, layout, and image patterns: LayoutLMv1, LayoutLMv2, and LayoutLMv3. The two OOD generalization algorithms are Deep Coral and Mixup. Deep Coral achieves domain adaptation by adjusting the second-order statistics between the source and target domains. Mixup achieves data augmentation by interpolating input features and labels.

[0032] A fine-grained analysis method based on document image distribution variations includes the following steps:

[0033] Step 1: Data Preprocessing

[0034] Three public datasets were selected for the experiment, including FUNSD, RVL-CDIP, and

[0035] DocVQA generates nine OOD datasets. Optimal checkpoints are selected for ID and OOD tests based on the results of the ID and OOD validation sets. The evaluation metrics used are the same as those used in the original dataset paper, such as "F1" for FUNSD and its OOD variants, "Accuracy" for RVL-CDIP and its OOD variants, and "ANLS" for DocVQA and its variants. All pre-trained models are based on huggingface.

[0036] Step 2: Use different documents and training models

[0037] Different documents and training models were used, including pre-trained models for text and layout patterns: BROS and LiLT; and pre-trained models for text, layout, and image patterns: LayoutLMv1, LayoutLMv2, and LayoutLMv3. For the document visual information extraction task, the learning rate was set to 3e-5 and the training epoch was set to 70. For the document visual classification task, since the original RVL-CDIP corpus does not provide text information, the Tesseract3 OCR engine was used to extract words and their positions, with a learning rate of 1e-6 and a training cycle of 30 epochs. For the document visual question answering task, the learning rate was set to 2e-5 and the epoch was set to 40. All input images had a resolution of 224x224 pixels. The batch size during training was set to 4, and the batch size during testing was set to 1.

[0038] Step 3: Predict the classification results of the test dataset.

[0039] The selected model was trained on the three source domain datasets mentioned in step 1 as described in steps 2-4, and tested on the nine datasets based on the changes in document image distribution.

[0040] Step 4: Evaluate the original training dataset used

[0041] The robustness of popular pre-trained VDU models fine-tuned using the original training dataset was evaluated using the Do-GOOD benchmark. Several representative OOD algorithms were then run on the Do-GOOD benchmark to demonstrate that out-of-distribution performance degradation still exists. Two OOD baseline methods were compared across all OOD datasets on three downstream tasks. Based on the experimental results, the effectiveness of different OOD methods was briefly analyzed. All experimental results are based on LayoutLMv3. BASE The baseline method uses Empirical Risk Minimization (ERM) and two OOD algorithms as baselines. Deep Coral achieves domain adaptation by adjusting the second-order statistics between the source and target domains. In the experiment, this method was only added to the last layer of the model and λ was set to 1. Mixup achieves data augmentation by interpolating input features and labels. The interpolated input features include text embedding, layout embedding and image embedding, and α and β are set to 0.4 of the Beta distribution.

[0042] Step 5: Investigate the impact of OOD samples generated by different models

[0043] Since the generation of OOD samples for a certain model depends on the model itself, experiments are conducted to study whether the performance of the model will also drop significantly when other models generate OOD sample layout distribution shifts.

[0044] Step 6: The impact of different image replacement strategies

[0045] For image distribution shifting, the original background image of the document image is replaced with a natural image. In practice, there are many replacement options available. To investigate whether other background replacement strategies would degrade the OOD performance of the existing VDU model, experiments were conducted using the LayoutLMv3 model to compare background replacement strategies, including the original image from MSCOCO, the grayscale image from MSCOCO, and the blank image without replacement.

[0046] (III) Beneficial Effects

[0047] This invention provides a fine-grained analysis based on changes in document image distribution. It offers the following advantages:

[0048] 1. This invention provides a fine-grained analysis based on the distribution changes of document images. It performs fine-grained analysis on various distribution changes of document images from the perspectives of images, text, and layout, enabling more detailed analysis and processing.

[0049] 2. This invention provides a fine-grained analysis based on the distribution changes of document images. In order to generate OOD benchmarks that meet the three proposed criteria, an automatic strategy is introduced to generate OOD data.

[0050] 3. This invention provides a fine-grained analysis based on document image distribution changes, evaluating and comparing five state-of-the-art pre-trained VDU models and two common OOD algorithms in different document image tasks. Furthermore, its proposed Do-GOOD benchmark, empirical research, and in-depth analysis will benefit future research to improve the robustness of pre-trained VDU models. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the implementation of the distribution offset method of the present invention;

[0052] Figure 2 This is a schematic diagram of the distribution offset type of the present invention;

[0053] Figure 3 This is a schematic diagram of a portion of the data generated in this invention;

[0054] Figure 4 The figure shows a comparison of the ID and OOD performance of the existing model of this invention on the FUNSD, RVL-CDIP and DocVQA datasets;

[0055] Figure 5 The graph shows the analysis results of the original training dataset used for the evaluation of this invention.

[0056] Figure 6 This is a graph showing the performance of LayoutLM in all models on the OOD dataset generated by the present invention.

[0057] Figure 7 This diagram illustrates the replacement strategy of LayoutLMv3 in all categories of OOD settings according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1:

[0060] like Figure 1-3As shown, this invention provides a fine-grained analysis based on document image distribution changes to study the robustness of existing document pre-trained models during the fine-tuning stage when training and test data undergo distribution shifts. It develops an out-of-distribution (OOD) benchmark called Do-GOOD for fine-grained analysis of document image-related tasks. The Do-GOOD benchmark defines the underlying mechanisms leading to different distribution shifts, namely specific image distribution changes, specific text distribution changes, and specific layout distribution changes. It includes nine OOD datasets covering three document image-related tasks: document visual information extraction, document visual classification, and document visual question answering. The robustness of visual document image understanding (VDU) models is evaluated, and fine-grained analysis is performed on these OOD datasets against five existing VDU pre-trained models and two common OOD generalization algorithms.

[0061] The Do-GOOD out-of-distribution (OOD) benchmark, using existing datasets such as FUNSD, prepares training and test samples under the in-distribution (IID) assumption. Given a training input x, the data distribution p... train The goal of the document image model f is to minimize the risk, as shown below:

[0062]

[0063] in, It is a loss function specific to the task. Due to confounding factors such as selection bias and random data splitting during data collection, in practice, training and test data rarely follow the same data distribution (i.e., p). trajn ≠p test Since the distributions of training and test data are different, a model trained on the training data is expected to generalize well to the test data. This necessitates carefully designing OOD benchmarks to accurately evaluate the model's generalization ability.

[0064] The specific image distribution variation, the image distribution shift, has two background variants: natural image background and distorted image background. Formally, y image Using finite sets Define the image. For training, y image The attribute is raw. When testing on undistributed data with a natural image background, the attribute y is set... image =y natural To obtain the attribute p natural (y 1：K The marginal distribution of latent factors and attributes is used to induce the joint distribution of latent factors and attributes.

[0065] p natural (z, y) 1：K )=p(z|y1：K )p natural (y 1：K ),

[0066] Subsequently, the input data used for the test can be obtained, and its joint distribution is:

[0067] p natural (x, y) 1：K )=∫p(x|z)p natural (z, y) 1：K ),

[0068] On the other hand, a method similar to generating test data for natural images can be used to deduce that the background is a distorted image p. distorted (z, y) 1：K The non-distributed test set.

[0069] To simulate a real-world scenario where the input document image may contain problematic text due to OCR errors, the specific text distribution variation employs two text attack strategies: (1) Bert-Attack; (2) Word-Swap. Formally, y text Using finite sets Define the text. For training, attribute y text It is the original. Referring to the analysis of specific image distribution variations in OOD benchmark tests, a similar method can be used to obtain BERT-Attack p. generation (x, y) 1：K ) and Word-Swap p swap (x, y) 1：K The out-of-distribution test data.

[0070] The specific layout distribution change involves two layout operations: merging and moving. The merge operation aims to study the impact of changing layout information from a fine-grained level to a coarse-grained level while preserving image and text information. The move operation is used to study the effect of moving content to different positions, making adjacent information on the content of a specific bounding box effective. Formally, y layout Using finite sets Define the layout. For training, y layout The attributes are raw. Similarly, by analyzing image-specific OOD and text-specific OOD benchmarks, the merge operation p can also be obtained. merge (x, y) 1：K OOD test data and Move operation p swap (x, y) 1：K OOD test data.

[0071] The document visual information extraction task, specifically for visual document information extraction, primarily utilizes the FUNSD dataset to generate the OOD dataset. FUNSD is a dataset sampled from the RVL-CDIP dataset for understanding noisy scanned forms, consisting of 199 documents (149 for training and 50 for testing) and 9743 semantic entities. The task of FUNSD is sequence labeling, which aims to assign labels to words.

[0072] FUNSD-L is a variant of FUNSD that includes OOD samples generated through two strategies based on layout-specific distribution transitions (merging and moving). As mentioned above, the Move operation is based on semantic strength determined by the model itself. Specifically, bounding boxes in the document image are randomly shuffled, and this process is repeated 30 times. For text content, fewer model prediction errors indicate stronger semantics.

[0073] FUNSD-T is a variant of FUNSD that contains OOD samples generated by the two text attack methods described in the specific text distribution variation above;

[0074] FUNSD-R is a real-world OOD dataset variant of FUNSD. It first samples data examples from the large-scale document classification dataset RCL-CDIP, then observes and selects data examples with a distribution different from FUNSD. These selected examples are then manually labeled. FUNSD-R contains 50 document images.

[0075] FUNSD-H is a modified version of the FUNSD dataset, creating an OOD dataset variant of FUNSD with artificial intervention. In practice, variations are constructed by moving some weak text entities to create layouts and image shifts, or by adding semantically weak text around semantically strong content. Finally, 50 OOD samples were obtained.

[0076] The document visual classification task, RVL-CDIP, is a document classification dataset used to predict the category of a given document. It includes 400,000 data examples across 16 categories, divided into 320,000 training samples, 40,000 validation samples, and 40,000 test samples.

[0077] RVL-CDIP-T is one of the OOD dataset variants of RVL-CDIP, containing OOD samples generated by two text attack methods.

[0078] As a variant of RVL-CDIP, RVL-CDIP produces OOD samples through two layout-specific distribution shifts. RVL-CDIP-I is a distorted OOD variant generated by a specific OOD variant of a natural image and an image distribution shift of RVL-CDIP-I2.

[0079] The document visual question answering task, DocVQA, is a dataset used to predict the answer to a given document image and question. The model needs to understand the content of the document and learn to reason about it. The original DocVQA data consists of 10,194 / 1,286 / 1,287 images, containing 39,463 / 5,349 / 5,188 questions for training / validation / testing.

[0080] DocVQA-T is an OOD dataset variant of DocVQA. To construct DocVQA-T, text, questions, and answers are first collected from OCR results and the Microsoft-READ API. Then, OOD samples generated by two text attack methods are obtained.

[0081] The robustness of pre-trained VDU models (large models) fine-tuned for downstream tasks using the Do-GOOD benchmark was evaluated using five existing VDU pre-trained models and two common OOD generalization algorithms. The large models include pre-trained models for text and layout patterns: BROS and LiLT; and pre-trained models for text, layout, and image patterns: LayoutLMv1, LayoutLMv2, and LayoutLMv3. The two OOD generalization algorithms are Deep Coral and Mixup. Deep Coral achieves domain adaptation by adjusting the second-order statistics between the source and target domains. Mixup achieves data augmentation by interpolating input features and labels.

[0082] This fine-grained analysis method based on document image distribution variations is characterized by the following steps:

[0083] Step 1: Data Preprocessing

[0084] Experiments were conducted on three publicly available datasets: FUNSD, RVL-CDIP, and Doc-VQA, generating nine OOD datasets for each. Optimal checkpoints were selected for ID and OOD testing based on the results of the ID and OOD validation sets. The evaluation metrics used were the same as those used in the original dataset paper, such as "F1" for FUNSD and its OOD variants, "Accuracy" for RVL-CDIP and its OOD variants, and "ANLS" for DocVQA and its variants. All pre-trained models were based on huggingface.

[0085] Step 2: Use different documents and training models

[0086] Different documents and training models were used, including pre-trained models for text and layout patterns: BROS and LiLT; and pre-trained models for text, layout, and image patterns: LayoutLMv1, LayoutLMv2, and LayoutLMv3. For the document visual information extraction task, the learning rate was set to 3e-5 and the training epoch was set to 70. For the document visual classification task, since the original RVL-CDIP corpus does not provide text information, the Tesseract3 OCR engine was used to extract words and their positions, with a learning rate of 1e-6 and a training cycle of 30 epochs. For the document visual question answering task, the learning rate was set to 2e-5 and the epoch was set to 40. All input images had a resolution of 224x224 pixels. The batch size during training was set to 4, and the batch size during testing was set to 1.

[0087] Step 3: Predict the classification results of the test dataset.

[0088] The selected model was trained on the three source domain datasets mentioned in step 1 as described in steps 2-4, and tested on the nine datasets based on the changes in document image distribution.

[0089] As attached Figure 4 As shown, it reports the comparison results of the existing model's ID and OOD performance on the FUNSD, RVL-CDIP and DocVQA datasets. Based on the differences between ID and OOD across all VDU tasks for each distribution, there is a consistent performance gap between ID and OOD settings.

[0090] In most cases, layoutlv3 achieves the best performance, including ID settings across all datasets, OODT and OODI settings for FUNSD and RVL-CDIP, and OODT settings for DocVQA, indicating that layoutlv3 is one of the most robust models on VDU tasks. These incentives use LayoutLMv3 as the base model for OOD algorithms that are common on the Do-GOOD benchmark. BROS performs well across the four OOD settings of FUNSD, possibly because BROS's pre-training targets the relative position of the encoded text and region masking strategies. Based on the success of LayoutLMv3 and BROS, it is hypothesized that fine-grained modeling, such as patch-level or region-level modeling, is very useful for improving the robustness of models in OOD environments.

[0091] Step 4: Evaluate the original training dataset used

[0092] The robustness of popular pre-trained VDU models fine-tuned using the original training dataset was evaluated using the Do-GOOD benchmark. Several representative OOD algorithms were then run on the Do-GOOD benchmark to demonstrate that out-of-distribution performance degradation still exists. Two OOD baseline methods were compared across all OOD datasets on three downstream tasks. Based on the experimental results, the effectiveness of different OOD methods was briefly analyzed. All experimental results are based on LayoutLMv3. BASE The baseline method uses Empirical Risk Minimization (ERM) and two OOD algorithms as baselines. Deep Coral achieves domain adaptation by adjusting the second-order statistics between the source and target domains. In the experiment, this method was only added to the last layer of the model and λ was set to 1. Mixup achieves data augmentation by interpolating input features and labels. The interpolated input features include text embedding, layout embedding and image embedding, and α and β are set to 0.4 of the Beta distribution.

[0093] As attached Figure 5 As shown, no OOD generalization algorithm consistently outperforms ERM; in fact, ERM outperforms Deep Coral in most cases. Mixup outperforms ERM in the OODR and OODH settings on FUNSD, but underperforms ERM in the OODT and OODL settings. This suggests that developing fine-grained comprehensive evaluation is crucial for OOD generalization.

[0094] In the information extraction task, Deep Coral and Mixup outperformed ERM in OODR and OODH. These results are reasonable and demonstrate that commonly used OOD algorithms are also applicable to VDU ​​models. In terms of layout distribution changes on FUNSD, ERM performs slightly better than Deep Coral and Mixup.

[0095] For document image classification, Deep Coral and Mixup outperform ERM in OODI1 and OODI2 settings, but still perform worse than ERM in OODL setting.

[0096] The results show that common OOD algorithms perform well in document classification tasks that require high modeling of image information. Deep Coral and Mixup score slightly lower than ERM in document visualization question answering tasks. Ordinary OOD algorithms cannot be used for distribution variations in complex tasks (such as document visualization question answering).

[0097] Step 5: Investigate the impact of OOD samples generated by different models

[0098] Since the generation of OOD samples for a certain model depends on the model itself, experiments are conducted to study whether the performance of the model will also drop significantly when other models generate OOD sample layout distribution shifts.

[0099] As attached Figure 6 As shown, LayoutLM consistently performs poorly on all model-generated OOD datasets. This indicates that LayoutLM trained with fixed layout information is highly dependent on layout information and struggles to cope with changes in layout distribution. LayoutLMv3 and BROS, on the other hand, perform well on all model-generated OOD datasets. This demonstrates that fine-grained information modeling, such as patch-level and region-level information modeling, can improve the robustness of the model.

[0100] Step 6: The impact of different image replacement strategies

[0101] For image distribution shifting, the original background image of the document image is replaced with a natural image. In practice, there are many replacement options available. In order to investigate whether other background replacement strategies will reduce the OOD performance of the existing VDU model, experiments were conducted using the LayoutLMv3 model to compare background replacement strategies, including the original image from MSCOCO, the gray image from MSCOCO, and the blank image without replacement.

[0102] As attached Figure 7 As shown, LayoutLMv3 performs poorly with any replacement strategy across all OOD settings. Furthermore, blank images have the least impact on model performance. Finally, based on the results, both the original MSCOCO image and the gray MSCOCO image can be used to test the model's robustness and generalization.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A fine-grained analysis based on document image distribution variations, characterized in that, They developed an out-of-distribution (OOD) benchmark called Do-GOOD for fine-grained analysis of document image-related tasks. The benchmark defines the underlying mechanisms that lead to different distribution shifts, namely specific image distribution changes, specific text distribution changes, and specific layout distribution changes. It includes nine OOD datasets covering three document image-related tasks: document visual information extraction, document visual classification, and document visual question answering. The robustness of visual document image understanding models was evaluated, and fine-grained analysis was performed on five existing VDU pre-trained models and two common OOD generalization algorithms on these OOD datasets. The Do-GOOD out-of-distribution (OOD) benchmark, existing datasets, train and test samples prepared under in-distribution assumption; given the data distribution p of the training input x train The goal of the document image understanding model f is to minimize the risk as follows: in, It is a loss function for a specific task; due to selection bias and random data segmentation confounding factors during the data collection process, in practice, training data and test data are unlikely to follow the same data distribution; since the distributions of training data and test data are different, a model trained on the training data is expected to generalize well to the test data; The specific image distribution variation, the image distribution shift has two background variants: natural image background and distorted image background; formally, y image Using finite sets Define the image; for training, y image The attribute is raw; when testing on undistributed data with a natural image background, the attribute y is set. image =y natural , obtain attribute p natural (y 1：K The marginal distribution of latent factors and attributes is used to induce the joint distribution of latent factors and attributes. p natural (z,y 1：K )=p(z|y 1：K )p natural (y 1：K ), Subsequently, the input data used for the test is obtained, and its joint distribution is: p natural (x,y 1：K )=∫p(x|z)p natural (z,y 1：K ), On the other hand, methods for generating natural images are used to deduce that the background is a distorted image p. distorted (z,y 1 ：K The non-distributed test set; The specific text distribution variation, i.e., the input document image may contain problematic text caused by OCR errors, employs two text attack strategies for the text distribution shift: (1) Bert-Attack; (2) Word-Swap; formally, y text Using finite sets Define the text; for training, attribute y text It is the original; based on the analysis of OOD benchmark tests with specific image distribution variations, BERT-Attackp is obtained. generation (x,y 1：K ) and Word-Swapp swap (x,y 1：K Out-of-distribution test data; The specific layout distribution change involves two layout operations: merging and moving. The merge operation aims to study the impact of changing layout information from a fine-grained level to a coarse-grained level while preserving image and text information. The move operation is used to study the effect of moving content to different positions, making adjacent information on the content of a specific bounding box effective. Formally, y layout Using finite sets Define the layout; for training, y layout The attributes are raw; based on the analysis of image-specific OOD and text-specific OOD benchmarks, the merge operation p is also obtained. merge (x,y 1：K OOD test data and Move operation p swap (x,y 1：K OOD test data; The document visual information extraction task is mainly based on the FUNSD OOD dataset; FUNSD is a dataset on understanding noisy scanned forms sampled from the RVL-CDIP dataset, consisting of 199 documents and 9743 semantic entities, of which 149 documents are used for training and 50 are used for testing. The document visual classification task uses an OOD dataset generated based on RVL-CDIP. RVL-CDIP is a document classification dataset used to predict the category of a given document. It includes 400,000 data examples in 16 categories, divided into 320,000 training samples, 40,000 validation samples, and 40,000 test samples. The document visual question answering task uses an OOD dataset generated based on DocVQA, which is a dataset used to predict the answer to a given document image and question.

Images