A method for building a structured document understanding model

Abstract

The present invention discloses a method for establishing a structured document understanding model, and relates to the field of document intelligence technology. The present invention at least includes S1: after extracting text information and layout information from images in a data set through OCR technology, two self-attention encoders are used to process the information of each modality separately; S2: a gating module combined with the residual idea is used to filter the information of each modality; S3: a cross-attention mechanism is introduced, and text features and layout features are fused through an SCA module; S4: a GCAF network is pre-trained through a data set. The method provided by the present invention is used to establish a structured document understanding model based on a gating mechanism and cross-attention, and the SCA module that introduces the cross-attention mechanism better fuses the layout features and text features of the document image; at the same time, a new residual gating module is proposed to dynamically extract important information of each modality.

Description

Technical Field

[0001] The present invention relates to the field of document intelligence technology, and in particular to a method for establishing a structured document understanding model. Background Art

[0002] Structured document understanding (SDU) is an important part of intelligent document processing, which aims to read and analyze the text and structured information contained in scanned / digitized documents. Specifically, it involves the task of semantic entity recognition (SER) from visually rich documents such as forms and receipts. With the acceleration of digitalization, it has been widely used in various fields such as finance, healthcare, and insurance.

[0003] The study of structured document understanding is an intersection of natural language processing and computer vision. It can be classified into single-stream and dual-stream structures according to the modal interaction mode. Previous methods mostly use single-stream structures, such as LayoutLM and subsequent models such as StructuralLM, BROS and FormNet, which regard layout information as two-dimensional spatial coordinates, convert it into vectors, embed it with the text of the multimodal pre-trained model in the embedding layer, and send it to a single self-attention encoder. However, the single-stream model uses the self-attention mechanism to learn the arrangement between modalities from the bottom layer. The processing of different modal information is the same, which easily ignores the importance of different modal information to the model's understanding of documents. The dual-stream model solves this problem well. The Selfdoc model processes text information and image information differently, and uses different models to process image regions for feature extraction and encoding into feature vectors. LiLT decouples text information and layout information in the pre-training stage, and proposes a new bidirectional attention complementary mechanism BiACM to strengthen the interaction between the two modalities, and then performs simple splicing for fine-tuning.

[0004] However, the current two-stream model uses a self-attention layer to process single modal information. Since the modal features are blended through multiple nonlinear transformations, it is easy for unimportant information to be redundant and important information to be lost, causing mutual interference between modal information and loss of effect. In addition, the two-stream model uses a simple splicing strategy to integrate multiple modal information, which does not promote a better integration of text information and layout information at a deeper level.

[0005] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a method for establishing a structured document understanding model to solve the technical problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for establishing a structured document understanding model, comprising at least the following steps:

[0008] S1: After extracting text and layout information from the images in the dataset using OCR technology, two self-attention encoders are used to process the information of each modality separately;

[0009] S2: A gating module combined with residual idea is used to filter the information of each modality;

[0010] S3: Introducing the cross attention mechanism to fuse text features and layout features through the SCA module;

[0011] S4: Pre-train the GCAF network using the dataset.

[0012] Furthermore, the S1 at least comprises the following steps:

[0013] The images in the dataset are passed through the OCR system, so that all text strings in the image are first segmented to extract word units, and then the text words are connected into a sequence S by sorting the corresponding text bounding boxes from top left to bottom right. t And other processing to obtain text features;

[0014] Construct a sequence S with the text t Two-dimensional position sequence S of the same length l Get layout features;

[0015] Then the text features and layout features are sent to the autoencoder for learning respectively;

[0016] The encoder trains the input text features and layout features separately to generate high-level features.

[0017] Furthermore, the encoder simultaneously learns text information and layout information through the BiACM module to enhance the cross-modal interaction of the two encoders.

[0018] Furthermore, the S2 at least includes the following steps:

[0019] The residual gating proposed by combining the idea of ​​the residual structure in ResNet to retain the initial information and the gating mechanism to filter information is used to learn the cross-features between the text features and layout features, so as to dynamically filter the important information between the layers of the interactive text and layout features and ensure lossless input to the fusion module. The residual gating function is defined as follows:

[0020]

[0021] Furthermore, the SCA module in S3 adds a cross attention layer to S of the N attention blocks based on a skip connection strategy. The cross attention layer consists of a self-attention layer (SA), a cross attention layer (CA) and a feed-forward network (FFN). The steps of fusing text features and layout features in the SCA module are as follows:

[0022] First, the input layout feature l n-1 Send it to the self-attention layer to get

[0023] Then the text feature Tn-1 is injected into the layout feature through the cross attention layer In, get

[0024] Attention and cross attention The outputs of are added and input into the FFN layer to obtain the representation l that combines layout features and text features. n ;

[0025] The SCA module expression is as follows:

[0026]

[0027]

[0028]

[0029] Among them, LN is the abbreviation of layer normalization.

[0030] Furthermore, the step of pre-training the GCAF network using the data set in S4 is:

[0031] The GCAF network is pre-trained using the document understanding dataset FUNSD;

[0032] The 199 documents in the rich text document dataset are divided into training set and test set.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The method provided by the present invention is used to establish a structured document understanding model based on a gating mechanism and cross-attention. The SCA module with the cross-attention mechanism is introduced to better integrate the layout features and text features of the document image. At the same time, a new residual gating module is proposed to dynamically extract important information of each modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0036] Figure 1 It is the structural diagram of the GCAF model of the present invention;

[0037] Figure 2 : is a structural diagram of the gated residual module of the present invention;

[0038] Figure 3 This is a structural diagram of the SCA module of the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0040] Embodiment 1:

[0041] A method for establishing a structured document understanding model includes at least the following steps:

[0042] S1: After extracting text and layout information from the images in the dataset using OCR technology, two self-attention encoders are used to process the information of each modality separately;

[0043] S2: A gating module combined with residual idea is used to filter the information of each modality;

[0044] S3: Introducing the cross attention mechanism to fuse text features and layout features through the SCA module;

[0045] S4: Pre-train the GCAF network using the dataset.

[0046] Embodiment 2:

[0047] See also Figure 1 - Figure 3 This embodiment is used to further disclose a training method for a structured document understanding model based on the above embodiment, thereby forming a structured document understanding model based on a gating mechanism and cross-attention.

[0048] GCAF network structure

[0049] GCAF is proposed, and the GCAF model architecture diagram is as follows Figure 1As shown. GCAF is based on the LiLT network, and the GCAF network adopts a dual-stream structure. After the text information and layout information of the image are extracted through OCR technology, two self-attention encoders are used to process the information of each modality separately. After that, the residual gating module filters the important information of each modality and passes it losslessly, and inputs it into the SCA module. It is worth mentioning that the calculation of SCA adopts a skip connection strategy, and SCA is used to fuse with the text information on the side of the more important layout information. Finally, the encoded text and layout features are connected, and an additional head is added on top of it for self-supervised pre-training or downstream fine-tuning.

[0050] GCAF mainly involves the structure of residual gating module and SCA module;

[0051] Residual Gating Module

[0052] In the LiLT model, text information and layout information interact through the BiACM module. In order to better integrate text features and layout features in the GCAF model, this paper combines the idea of ​​the residual structure in ResNet to retain initial information and the gating mechanism in the expert system to filter information, and proposes a residual gating module to dynamically control the important information between the text and layout modal layers after filtering interaction and ensure lossless input to the fusion module. The residual gating module is divided into three layers, and each layer is calculated by the residual gating function. The residual gating function is defined as follows:

[0053]

[0054] Among them, c0 represents the input of the residual gating module, c l ∈R D represents the output of layer l and the input of layer l+1, c l+1 ∈R D represents the output of the residual gating module, b1∈R D are two learnable matrices and a bias vector respectively. ⊙ represents the Hadmard product, × represents matrix multiplication, and σ represents the sigmoid activation function.

[0055] In each gating layer, there are two core components: the layer feature matrix and soft gate matrix The matrix is ​​used to calculate the first layer feature c0 and the i-th layer c i The hierarchical relevance and intrinsic importance between them. and the bias vector b l It is used to adaptively learn hierarchical features, amplify important information in each layer, and reduce the impact of unimportant information. Finally, borrowing the idea of ​​residual, the residual gating matrix input c0 is added to the calculation results of the layer feature matrix and the soft gating matrix to generate the final output vector c l+1 .

[0056] SCA Module

[0057] The GCAF network consists of N attention blocks. Based on the skip connection strategy, the SCA module adds a cross attention layer to S of the N attention blocks. The cross attention layer consists of a self-attention layer (SA), a cross attention layer (CA), and a feed-forward network (FFN). First, the input layout feature l n-1 Send it to the self-attention layer to get Then the text feature Tn-1 is injected into the layout feature through the cross attention layer In, get Attention and cross attention The outputs of are added and input into the FFN layer to obtain the representation l that combines layout features and text features. n :

[0058]

[0059]

[0060]

[0061] Among them, LN is the abbreviation of layer normalization.

[0062] Based on the above two embodiments, the following experimental methods are disclosed:

[0063] The dataset used for pre-training and evaluation of the GCAF model, the detailed experimental environment and parameter settings during the experiment, the experimental results of the GCAF model, and the subsequent ablation experiment results and analysis.

[0064] Dataset

[0065] In order to pre-train and evaluate the GCAF network, the present invention selects the rich text document dataset FUNSD for experiments. The FUNSD dataset mainly solves the form understanding task. The dataset consists of original images of various types of real documents and annotation results in JSON format. There are 199 documents in total, of which 149 are used for training and 50 are used for testing. The most important task in the FUNSD dataset is the SER task. SER classifies entities in form documents into questions, answers, and titles. RE analyzes the association relationship between the extracted entities. The evaluation indicators of the FUNSD dataset are precision, recall, and F1 score. The relationship between the three evaluation indicators is:

[0066]

[0067] Experimental setup

[0068] The GCAF experiment environment was conducted on an NVIDIA 4090 GPU server with CUDA version 11.1. The parameter initialization of the GCAF model was completed by calling the weights of the pre-trained model LiLT. The main parameters of the pre-trained model follow the common settings of the Transformer encoder: the hidden layer size is 768, and the number of 12 attention layers is 12. The model was trained using the Adam optimizer.

[0069] Experimental Results

[0070] Table 1 shows the results of different models for the semantic entity recognition (SER) task on the FUNSD dataset. This paper compares the network with StrucText, Doc2Graph, LayoutLMv2-base and LiLT(EN-R)-BASE, and the results are shown in Table 1. Experiments show that for the SER task, the network of this paper obtains the highest F1 score of 89.12% among the listed methods. More specifically, GCAF has a 1.01% higher precision, 0.47% higher recall, and 0.71% higher F1 score than the original baseline model LilT network. This is because the GCAF model further promotes the fusion of multiple modal information through the residual gating mechanism and the SCA module.

[0071] Table 1 SER task results on the FUNSD dataset

[0072]

[0073] Ablation experiment

[0074] The ablation experiment in this paper uses the SER task of the FUNSD dataset, and gradually integrates the residual gating module Rgate and SCA module of GCAF based on the LiLT network to prove the effectiveness of each module, and analyzes the settings of each module separately. The results of the ablation experiment are shown in Table 2.

[0075] Table 2 Ablation experiments on FUNSD dataset

[0076]

[0077]

[0078] As shown in Table 2, this paper first inserts the residual gating module into the pre-trained model of LiLT, and the F1 score increases by 0.28% compared with the baseline model, verifying that the residual gating module can effectively extract important features of a single modality and ensure lossless output of important features. In the next step, this paper introduces the SCA module into LiLT to fuse text features and layout features, thereby improving the task and improving the F1 score by 0.42% compared to the original LiLT. The last row shows the results of the combined residual gating module and SCA module.

[0079] In summary, the overall improvement over the baseline LiLT demonstrates the effectiveness of our two contributions, i.e., the residual gating module and the SCA module.

[0080] Based on the above summary

[0081] A structured document understanding model GCAF based on gating mechanism and cross attention was proposed through a structured document understanding model establishment method.

[0082] Different from the previous pre-training model, in order to solve the problem of information redundancy and loss of important information caused by the self-attention layer before the single modal information is sent to the SCA module, a new residual gating module is proposed to ensure that the important information in each modality is input to the fusion module without loss. In view of the problem that the previous method only used the early simple fusion strategy for multiple modalities in the form document, which may cause modal information redundancy and thus reduce the model performance, it is proposed to use the SCA module on the layout information side to better integrate the model's text modality and layout information.

[0083] The present invention has conducted experiments on the FUNSD dataset, and the GCAF model outperforms the strong baseline model and achieves satisfactory results. In addition, the present invention has designed an ablation experiment, which proves that the residual gating module and SCA module designed by the present invention are both effective.

[0084] In the follow-up research on the GCAF network, this paper hopes to further explore the network structure of GCAF and the improvement of pre-training tasks for fine-tuning GCAF. And this paper hopes to apply GCAF to different datasets, especially data from non-English regions and large datasets that are better than pre-training.

[0085] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for establishing a structured document understanding model, characterized in that: At least the following steps are included: S1: After extracting text and layout information from the images in the dataset using OCR technology, two self-attention encoders are used to process the information of each modality separately; S2: A gating module combined with residual idea is used to filter the information of each modality; The S2 at least comprises the following steps: The residual gating proposed by combining the idea of ​​the residual structure in ResNet to retain the initial information and the gating mechanism to filter information is used to learn the cross-features between the text features and layout features, so as to dynamically filter the important information between the layers of the interactive text and layout features and ensure lossless input to the fusion module. The residual gating function is defined as follows: ; S3: Introducing the cross attention mechanism to fuse text features and layout features through the SCA module; The SCA module in S3 adds a cross attention layer to S of the N attention blocks based on the skip connection strategy. The steps of fusing text features and layout features in the SCA module are as follows: First, the input layout features Send it to the self-attention layer to get ; Then the text features Injecting layout features via cross-attention layers In, get ; Attention and cross attention The outputs of are added and input into the FFN layer to obtain a representation that combines layout features and text features. ; The SCA module expression is as follows: ， Among them, LN is the abbreviation of layer normalization; S4: Pre-train the GCAF network using the dataset; The GCAF network is based on the LiLT network. The GCAF network adopts a dual-stream structure. The GCAF mainly involves the structures of the residual gating module and the SCA module.

2. A method for establishing a structured document understanding model according to claim 1, characterized in that: The S1 at least comprises the following steps: The images in the dataset are passed through the OCR system, so that all text strings in the image are first segmented to extract word units, and then the text words are connected into a sequence S by sorting the corresponding text bounding boxes from top left to bottom right. t And other processing to obtain text features; Construct a sequence S with the text t Two-dimensional position sequence S of the same length l Get layout features; Then the text features and layout features are sent to the autoencoder for learning respectively; The encoder trains the input text features and layout features to generate high-level features; The text information and layout information are learned simultaneously through the BiACM module in the encoder.

3. A method for establishing a structured document understanding model according to claim 2, characterized in that: The encoder simultaneously learns text information and layout information through the BiACM module to enhance the cross-modal interaction of the two encoders.

4. The method for establishing a structured document understanding model according to claim 1, characterized in that: The steps of pre-training the GCAF network using the data set in S4 are: The GCAF network is pre-trained using the document understanding dataset FUNSD; The 199 documents in the rich text document dataset are divided into a training set and a test set.

Images