Rich vision document information extraction method based on graph neural network enhanced language model

Abstract

In the aspect of document information extraction, a document pre-training model becomes a mainstream, but is limited by large model pre-training time and computing resource overhead, document fine-grained information cannot be injected into a large model, and long-distance structure information is easy to ignore, but it is difficult to improve the pre-training model fundamentally. Therefore, the rich vision document information extraction method based on the graph neural network enhanced language model is provided, the graph neural network is used for enhancing the reconstruction capability of the document of a pre-training model, and the vision, the text and the structure of the document are input into the graph neural network; and then fusion fine tuning is performed on the graph neural network and the output of the pre-training model, so that the knowledge lacked by the pre-training model can be further supplemented, and the performance of downstream tasks can be effectively improved. The framework is a lightweight enhancement scheme, does not increase too much calculation burden, can be widely applied to various pre-training models, and brings a new breakthrough to a document information extraction technology.

Description

Technical Field

[0001] The present invention can be applied to the fields of document information extraction, natural language processing, multimodal learning, etc., and specifically proposes a rich visual document information extraction method based on a graph neural network enhanced language model, which can extract key information from image documents and PDF documents. Background Art

[0002] In various production scenarios of modern enterprises, a large number of document images are often generated, such as scanned document images, system-exported documents, and photographed documents. These documents contain a large amount of information, including not only the processing details and knowledge accumulation of various industrial affairs, but also a large amount of physical and digital information related to the industry. Making full use of the information in the documents will bring the following advantages to the enterprise:

[0003] (1) Information is critical to industrial enterprises. A large number of documents, reports, data sheets and other information sources are scattered among various departments of the enterprise, containing valuable knowledge, guidance and historical records.

[0004] (2) The modern industrial field is increasingly paying attention to automation and digital transformation. By integrating document information extraction with automated processes, companies can obtain the required information more quickly and accurately, thereby improving production efficiency and the intelligence level of production lines.

[0005] However, these documents are generally unstructured, diverse, low-quality and complex. Diversity is reflected in the wide variety of document layouts and formats; low-quality is reflected in the low quality of documents, such as unclear scanned documents and images generated by wrinkled paper; complexity is reflected in the complex structure of document templates, which contain a lot of content and information. Manual extraction of this information is not only time-consuming, laborious, and low in accuracy, but also has low reusability. Therefore, the previous method of extracting text information from images by manual analysis alone can no longer meet the requirements.

[0006] The two main tasks of document information extraction are named entity recognition and entity relationship extraction. A common method for document information extraction is based on the graph neural network model. This method constructs a global document graph structure, in which each text block is regarded as a node. In addition to text information, node information also contains layout information and visual information, and the spatial arrangement between text blocks is regarded as edge features. Therefore, through the calculation of the graph neural network, the information transmission and update of the nodes, the category prediction of the entity and the relationship prediction between entities are completed. The graph-based method can make full use of the text and spatial information in the visual document to model the text information, and communicate local and global information through graph convolution, which is more convenient for the prediction of entity information.

[0007] The field of visual documents has a large amount of unlabeled visual document data. Drawing on the success of the pre-training model for text, the pre-training model has also been proposed for visual documents. Document-based pre-training technology can use the visual information, layout information and other knowledge of the document for pre-training, solving the problem that traditional language pre-training models cannot learn the relationship between multimodal information well.

[0008] In summary, the existing document visual information extraction methods have the following defects:

[0009] (1) Methods based on graph neural networks do not utilize the large amount of data sets in the field of visual documents. Moreover, graph neural networks are mostly only targeted at specific types of information extraction tasks, do not have better versatility, and the extraction effect is not as good as that of pre-trained large models.

[0010] (2) The document information extraction model based on pre-training focuses more on the feature representation of entities within the document, but ignores the structural information between entity pairs in unstructured documents. For entity pairs that are far apart within the document, the extraction effect is not ideal. When combined with structural information, the accuracy of information extraction will be further improved.

[0011] Therefore, it is necessary to invent a technical solution that can solve the above defects in document information extraction. Summary of the invention

[0012] The purpose of the present invention is to propose a joint framework based on graph neural network and pre-trained model, which can overcome the problems existing in the above-mentioned existing information extraction methods and extract the key information of the document more effectively.

[0013] To achieve the above object, the present invention provides a method for extracting rich visual document information based on a graph neural network enhanced language model, comprising:

[0014] Acquire a document image. If the document is a parseable PDF document, parse the document to obtain text information. If the document is an image document, detect and recognize text on the document image to obtain text information.

[0015] Performing feature extraction on the text information and the document image to obtain multimodal features;

[0016] The multimodal features are respectively input into the pre-trained model module and the graph neural network module for fine-tuning, and the outputs of the two are fused through the attention mechanism to obtain the multimodal features;

[0017] The obtained multimodal features are used to predict entity categories and entity links to extract key information from rich visual documents.

[0018] Preferably, the process of acquiring text information from the document image includes:

[0019] For parseable PDF documents, directly parse and obtain text information results;

[0020] For image documents and unparseable PDF documents, the document images are detected and recognized based on a text detection model and a recognition model to obtain text detection and recognition results.

[0021] Preferably, the process of obtaining the text features of the input model includes:

[0022] The text information result obtained according to the above process is segmented, and then input into the text mapping model to obtain text features input into the pre-trained model and the graph neural network model;

[0023] Preferably, the process of obtaining the layout features of the input model includes:

[0024] Input the layout features of the pre-trained model module, map the text position information of the document obtained according to the above process, that is, the coordinates of the text in the document, to the position coordinates, and obtain the layout features of the input pre-trained model;

[0025] Input the layout features of the graph neural network module, construct the text position information of the document obtained according to the above process into polar coordinates, and then map the polar coordinates to obtain the layout features input into the graph neural network;

[0026] Preferably, the process of obtaining the visual features of the input model includes:

[0027] The document image is segmented according to the text information obtained from the above process to obtain image blocks corresponding to the text areas, and then the visual features of the image blocks are obtained based on the image feature extraction module.

[0028] Preferably, the obtained multimodal information is input into the pre-training module and the graph neural network module respectively, and the features output by the two modules are fused using the attention module to obtain the final features.

[0029] Preferably, for the finally obtained features, the entity classification prediction head and the entity link prediction head are used to obtain the output results of the two tasks.

[0030] Compared with the prior art, the present invention has at least the following advantages:

[0031] By combining graph neural networks with pre-trained models, the present invention makes up for the deficiency of simply using graph neural networks and pre-trained models to extract key information of documents while adding a small amount of computing resources. It can make full use of the fine-grained information in custom documents, thus improving the effect of document key information extraction tasks.

[0032] The pre-trained model module of the present invention can utilize a variety of pre-trained models on the market and can select a model according to the task to achieve better results;

[0033] The output feature information fusion module of the present invention can effectively fuse the output features of the graph neural network module and the pre-training module, and exhibits excellent performance in document entity classification tasks and entity linking tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to make the contents of the present invention more clearly understood, and to constitute a part of the specification, the present invention is further described in detail below according to specific embodiments and in conjunction with the accompanying drawings, wherein the accompanying drawings include:

[0035] Figure 1 is a schematic diagram of a process for extracting document information according to an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of the overall model architecture of the present invention;

[0037] Figure 3 is a schematic diagram of a document to be processed according to an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of the document information extraction result in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the information extraction process of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that this embodiment is only one of the embodiments of the present invention, not all of the embodiments. The present invention can be used for multilingual documents and documents of different types. The embodiments are described in detail below in conjunction with the accompanying drawings.

[0040] Embodiment 1:

[0041] Figure 1 : is a flowchart of a method for extracting key information from a document provided in Embodiment 1 of the present invention, comprising the following steps:

[0042] In the input stage (step S1), the document to be extracted is input into the information extraction system.

[0043] In the preprocessing stage (step S2), the documents to be extracted are preprocessed. The input documents can be divided into two categories, image documents and parseable PDF documents. For image documents, step S21 is performed to use OCR technology to perform text detection and text recognition on the document to obtain the text information of the image; for parseable PDF documents, step S22 is performed to use PDF parsing technology to directly obtain the text information of the PDF document. The text information includes the text and the location of the text. At this point, we already have complete input information (step S23).

[0044] Multimodal feature extraction (step S3), the multimodal information is passed through the corresponding feature mapping module to obtain the corresponding multimodal features, and the feature extraction process of each modality is as follows:

[0045] Step S31. Segment the text information result of the document obtained according to the above process, and then input it into the text mapping model to obtain text features input into the pre-training model and the graph neural network model;

[0046] Step S32. Map the text position information of the document obtained according to the above process, that is, the coordinates of the text in the document, to the position coordinates to obtain the layout features of the input pre-trained model and the graph neural network;

[0047] Step S33. The layout information of the graph neural network module is used as the input of the edge, the text position information of the document obtained according to the above process is constructed into polar coordinates, and then the polar coordinate information is mapped to obtain the layout features of the edge input into the graph neural network;

[0048] Step S34. The process of obtaining the visual features of the input model includes: segmenting the document image according to the text information obtained in the above process to obtain image blocks corresponding to the text area, and then obtaining the visual features of the image blocks based on the image feature extraction module;

[0049] Step S4. We have obtained three multimodal features: text, layout, and vision. Then, we input the multimodal features into the pre-trained model module and the graph neural network module respectively to obtain their respective output features. First, we obtain multimodal information from the document, then use the multimodal feature extraction module to extract multimodal features, and then input the multimodal features into the graph neural network to obtain graph features after calculation.

[0050] Feature fusion (step S5), the features output by different modules are fused. The present invention adopts a cross-attention module to fuse the outputs of the pre-training module and the graph neural network module to obtain the final output features. Figure 2 It is a schematic diagram of the overall structure of the present invention.

[0051] Prediction and output (step S6), for the final output features, the entity classification prediction module and the entity link prediction module are used for prediction.

[0052] The task prediction result is obtained (step S7). So far, we have completed the information processing and circulation process of the present invention and realized the extraction of key information of the document.

[0053] In this example, Figure 3 is the input image document, Figure 4 The rich visual document information extraction method based on graph neural network enhanced language model proposed in the present invention is used for the image document, and the predicted entity classification and entity linking results are shown, which shows that the method used in this example can effectively extract the key information of the image document.

[0054] The method proposed in the present invention has demonstrated excellent performance in the task of extracting key information from documents, providing strong support for promoting the advancement and application of artificial intelligence technology in the field of document processing.

[0055] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting rich visual document information based on a graph neural network enhanced language model, characterized in that: The following steps are involved: a) Obtaining a document. If the document is a parseable PDF document, performing PDF parsing to obtain text information; if it is an image document, performing text detection and recognition to obtain text information; b) extracting features from the text information and the document image to obtain multimodal features, including text features, layout features, and visual features; c) Inputting the multimodal features into the pre-trained model module and the graph neural network module; d) Use the attention mechanism to fuse the output features of the pre-trained model module and the graph neural network module; e) Use fused features to predict entity categories and entity links, thereby extracting key information from documents.

2. The method for extracting multimodal information according to claim 1, characterized in that: In step a), for image documents, OCR technology is used to perform text detection and text recognition on the document to obtain the text information of the image; for parseable PDF documents, PDF parsing technology can be used to directly obtain the text information of the PDF document.

3. The method of obtaining multimodal features from multimodal information according to claim 1, characterized in that: In step b), the text information is processed by word segmentation and then mapped to obtain text features suitable for the pre-trained model and the graph neural network model; the text position information is mapped to layout features, which are respectively suitable for the pre-trained model and the graph neural network model; and the document image is segmented to extract the visual features of the corresponding text area.

4. The multimodal features according to claim 1 are respectively input into a pre-trained model module and a graph neural network module, characterized in that: In step c), the pre-trained model module and the graph neural network module process the input multimodal features respectively. The former uses pre-trained knowledge to capture semantic information, and the latter analyzes contextual dependencies through graph structure.

5. The pre-trained model module according to claim 1, characterized in that: Pre-trained Models Different pre-trained models can be used.

6. The feature fusion module according to claim 1, characterized in that: In step d), the attention mechanism is used to fuse the outputs of the pre-trained model and the graph neural network to enhance the model's ability to capture key information.

7. The prediction module according to claim 1, characterized in that: In step e), specific prediction heads, such as entity classification prediction head and entity link prediction head, are applied to output the final entity recognition and relationship extraction results based on the fused features.

8. A system for implementing the method according to any one of claims 1 to 7, comprising: Document processors, which are used to process different types of documents and extract text information; Feature extractor, responsible for extracting multimodal features from text information and document images; The pre-trained model module and the graph neural network module process multimodal features and output features respectively; Attention fusion module, used to fuse the output features of the above two modules; And the prediction module performs entity category prediction and entity link prediction based on the fused features.

Images