A multimodal document information extraction method based on graph neural network
Through the multimodal document information extraction method based on graph neural network, the spatial layout information and structured hierarchical information of complex layout documents are captured, and the problem of insufficient accuracy of entity recognition and relationship extraction in the existing technology is solved, and a higher level of development is achieved semantic accuracy and universality.
Patent Information
- Application Number
- CN202210056911.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-01-18
AI Technical Summary
The existing multimodal document information extraction method has poor effect when processing complex layout documents, and cannot effectively capture the spatial layout information and structured hierarchical information of the document, resulting in low accuracy of entity recognition and relationship extraction.
Using a graph neural network-based method, the document content is abstracted and the graph node features and edge features are used to classify and link prediction to capture the document space through optical character recognition, precoding, graph construction, graph division and graph depth encoding steps. Layout information and local structures, blocking methods that conform to human reading habits.
It effectively improves semantic accuracy and universality, can accurately identify entities and relationships in complex layout documents, and improves the performance of information extraction.
Smart Images

Figure CN114419304B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer software, and in particular to a multimodal document information extraction method based on graph neural network. Background Art
[0002] With the digital transformation of the industry, business data is mainly stored in unstructured formats of multimodal documents (such as PDF, scans, document images, etc.). Document automation processing is crucial to improving enterprise production efficiency. For example, in the field of supply chain finance, it is necessary to extract key effective information from electronic documents such as invoices, contracts, and tax bills to quickly understand the risk status of upstream and downstream companies and assist core companies in supply chain management. The multimodal document information extraction task includes extracting specific category entities from the document, outputting specific relationships between entities, and outputting structured information in the form of key-value.
[0003] Traditional document information extraction is mainly based on manually customized rules, template matching, and small sample statistical learning. This method is effective for a small number of fixed-style documents, but has poor versatility and high migration costs, and cannot adapt to the current characteristics of multimodal document types and styles. The deep learning-based method uses a large number of unlabeled electronic documents for self-supervised pre-training, and uses multimodal technology to fuse text, vision, and layout information to obtain document features. Finally, it is serialized in a specific order (generally from top to bottom and from left to right) according to the position of the text box, and entity recognition is performed through NER technology, and relationship extraction is performed by triple classification. This method can learn effective features of different types of documents and has a significant improvement in the performance of entity recognition tasks. However, since entity recognition is finally performed through serialized text, the effect is poor for documents with complex layouts (such as entities may be cut in column documents). At the same time, the spatial relationship of text blocks is not fully utilized when performing relationship extraction, and the general relationship matching effect is poor.
[0004] The current mainstream multimodal document information extraction methods mainly include the following processes:
[0005] 1) Use OCR technology to identify candidate text boxes in the document;
[0006] 2) Extract the features of the candidate text boxes through the pre-trained model, including text features, visual features and location coordinates;
[0007] 3) The features of different dimensions extracted in 2) are fused and encoded on the candidate text blocks through multimodal fusion technology, including maximum (average) pooling, bilinear model, attention mechanism, etc.;
[0008] 4) According to the position coordinates of the candidate text boxes, they are serialized in order from top to bottom and from left to right, and then entity recognition is performed using NER technology. Relationship extraction is performed by classifying the triples formed by combining candidate text boxes in pairs (with relative position information added).
[0009] Existing technologies rely on the output of OCR models for serialization, and generally follow the reading principle of "from top to bottom, from left to right". However, for complex layouts in documents with columns, texts, pictures and tables, the reading order obtained at this time is often wrong, which can easily lead to entity segmentation and semantic confusion. At the same time, serialization cannot capture the structured hierarchical information in the document (such as the relationship between the header and cells in a table), and often performs poorly in the task of relationship extraction.
[0010] Therefore, the prior art has defects and needs to be improved. Summary of the invention
[0011] The technical problem to be solved by the present invention is to provide a multimodal document information extraction method based on graph neural network which can effectively capture the spatial layout information of documents, effectively improve the semantic accuracy and has strong versatility.
[0012] The technical solution of the present invention is as follows: A multimodal document information extraction method based on graph neural network, comprising the following steps: Step S1: Optical character recognition (OCR): recognizing the text in the picture, converting it into text format and outputting the text block, and outputting the position coordinates of the text block; Step S2: Precoding: extracting features of the text block output by the optical character recognition (OCR), including three modal features of text, vision and layout, and performing feature fusion between modalities, and outputting graph node features; Step S3: Graph construction: abstracting the content of the text block and outputting the initial graph G= (V, E, A): where V represents graph nodes, E represents edges between nodes, and A represents the adjacency matrix of the graph; Step S4: Graph partitioning: The initial graph in Step S3 is partitioned by assigning graph nodes to different clusters, and the assignment matrix of graph nodes is iteratively updated, and the graph is partitioned by iterative multiple assignments; Step S5: Graph deep encoding: The final representation of the graph nodes is updated according to the assignment matrix and cluster representation of each stage in Step S4; Step S6: Information extraction: Node classification and link prediction are performed on the graph nodes in Step S5, and the final result is output.
[0013] Applied to the above technical solution, in the multimodal document information extraction method based on graph neural network, in the step S1, the text in the picture is recognized including text detection and text recognition, and the text block and the position coordinates of the text block are output through text detection and text recognition, wherein text detection uses lightweight DBNet as the backbone network; text recognition uses CRNN for line text recognition, and CenterLoss is used to improve the recognition effect.
[0014] Applied to the above-mentioned technical solutions, in the multimodal document information extraction method based on graph neural network, in the step S2, RoBERTa is used to extract text features, ResNet is used as the backbone network, and the visual features within the corresponding text block range are obtained through RoIAlign according to the position coordinates of the text block; and the initial layout features are generated according to the position coordinates and size of the text block; finally, the three modal features of the extracted text features, visual features, and layout features are fused using the Block mechanism to output graph node features.
[0015] Applied to the above-mentioned technical solutions, in the multimodal document information extraction method based on graph neural network, in the step S3, E represents the edge between nodes, which is defined by the spatial relationship and semantic similarity between text blocks, and the edge features are obtained by calculating the relative position relationship between text blocks and the visual range features of both; A represents the adjacency matrix of the graph, and the edge features are obtained and calculated using the multi-head self-attention mechanism.
[0016] Applied to the above-mentioned technical solutions, in the multimodal document information extraction method based on graph neural network, in the step S4, the graph partitioning is achieved by assigning graph nodes to different clusters, specifically including: assuming that the current number of graph nodes is N, first predefine the attenuation factor c to determine the number of clusters N*c in the next layer, use the first graph convolutional network to perform information interaction and graph node allocation matrix learning N x N*c, and after the allocation is completed, input it into another second graph convolutional network for information interaction between clusters, and the dimension of the graph adjacency matrix changes from N x N to N*cx N*c; and implement graph partitioning by iterating multiple allocations.
[0017] Applied to the above-mentioned technical solutions, in the multimodal document information extraction method based on graph neural network, in the step S5, the final representation of the graph nodes is updated, including: the graph nodes in the same cluster are sorted and encoded in the horizontal position according to the principle of "from left to right, from top to bottom"; the sequential allocation results of the graph nodes are encoded in the vertical position, and the hierarchical block information and local reading order are integrated into the graph node representation.
[0018] Applied to the above-mentioned technical solutions, in the multimodal document information extraction method based on graph neural network, in the step S6, the link prediction is performed by concatenating the representations of two candidate graph nodes as input, and finally a classifier is used to calculate the probability of the edge between the two.
[0019] The beneficial effects of the present invention are:
[0020] The present invention abstracts the document content in the form of a graph, which can effectively capture the spatial layout information of the document and perform better in the relationship extraction task; and, the document content is locally segmented by graph partitioning, which conforms to the human habit of hierarchical segmented reading and effectively improves the semantic accuracy; the present invention has strong versatility: the model can effectively extract the local structural information of the document and is also applicable to documents with complex layouts.
[0021] In addition, the present invention is based on a graph neural network, which abstracts the document content into a graph, can effectively capture the hierarchical information of the document, and at the same time effectively identify the local structure of the document through graph division, which is in line with the human habit of hierarchical and block reading, and can also accurately understand documents with complex layouts. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0023] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0024] This embodiment provides a multimodal document information extraction method based on graph neural network. Figure 1 As shown, the multimodal document information extraction method based on graph neural network includes the following steps: Step S1: Optical Character Recognition OCR: recognize the text in the picture, convert it into text format and output the text block, and output the position coordinates of the text block at the same time; wherein, recognizing the text in the picture includes text detection and text recognition, and the text block and the position coordinates of the text block are output through text detection and text recognition, wherein text detection uses lightweight DBNet as the backbone network; text recognition uses CRNN for line text recognition, and CenterLoss is used to improve the recognition effect.
[0025] Step S2: Precoding: Extract features from the text blocks output by the optical character recognition (OCR), including three modal features: text, vision, and layout. Perform feature fusion between the modalities and output graph node features. Use RoBERTa to extract text features, and use ResNet as the backbone network to obtain visual features within the corresponding text block range based on the position coordinates of the text block through RoIAlign. Generate initial layout features based on the position coordinates and size of the text block. Finally, use the Block mechanism to fuse the three modal features of the extracted text features, visual features, and layout features, and output graph node features.
[0026] Step S3: Graph construction: abstract the content of the text block and output the initial graph G = (V, E, A): wherein V represents the graph node, E represents the edge between nodes, and A represents the adjacency matrix of the graph; wherein V represents the graph node, which is the text block output by OCR, and the graph node feature is represented by the result in step S2, E represents the edge between nodes, which is defined by the spatial relationship and semantic similarity between each text block, and the edge feature is obtained by calculating the relative position relationship between the text blocks and the visual range feature containing both; A represents the adjacency matrix of the graph, which is calculated using the obtained edge features using the multi-head self-attention mechanism.
[0027] Step S4: Graph partitioning: The initial graph in step S3 is partitioned by assigning graph nodes to different clusters, and the allocation matrix of graph nodes is iteratively updated, and the graph is divided into blocks by iterative multiple allocations; Graph partitioning is achieved by assigning graph nodes to different clusters, specifically including: assuming that the current number of graph nodes is N, first predefine the attenuation factor c to determine the number of clusters N*c in the next layer, use the first graph convolutional network to perform information interaction and learn the graph node allocation matrix N xN*c, and after the allocation is completed, it is input into another second graph convolutional network for information interaction between clusters, and the dimension of the graph adjacency matrix changes from N x N to N*cx N*c; Graph partitioning is achieved by iterative multiple allocations.
[0028] Step S5: Graph depth encoding: The final representation of the graph nodes is updated according to the allocation matrix and cluster representation of each stage in step S4; wherein, the final representation of the graph nodes is updated including: the graph nodes in the same cluster are encoded in the horizontal position sorting according to the principle of "from left to right, from top to bottom"; the sequential allocation results of the graph nodes are encoded in the vertical position, and the hierarchical block information and local reading order are integrated into the graph node representation.
[0029] Step S6: Information extraction: perform node classification and link prediction on the graph nodes in step S5, and output the final result, wherein the link prediction is performed by concatenating the representations of two candidate graph nodes as input, and finally calculating the probability of the edge between the two through a classifier.
[0030] Furthermore, embodiments of the present invention include:
[0031] 1. Optical Character Recognition (OCR): This module is divided into two sub-modules: text detection and text recognition. It mainly recognizes the text in the image and converts it into text format for output, and outputs the location coordinates of the text. In the text detection module, lightweight DBNet is used as the backbone network; in the text recognition module, CRNN is used for line text recognition, and CenterLoss is used to improve the recognition effect.
[0032] 2. Pre-coding: This module mainly extracts features from the text blocks output by OCR, including three modal features: text, vision, and layout, and performs feature fusion output between modalities. In the text encoding module, RoBERTa is used to extract text features; in the visual encoding module, ResNet is used as the backbone network to obtain the visual features within the corresponding text box range based on the text box coordinates and through RoIAlign; in the layout encoding module, the position coordinates and size of the text box are used to generate initial features. Finally, the three extracted modal features are fused using the Block mechanism to output graph node features.
[0033] 3. Graph construction: This module mainly abstracts the document content and outputs G = (V, E, A):
[0034] aV represents a graph node, which is a text block output by OCR. The node features are represented by the results in 2.
[0035] bE represents the edge between nodes, which is defined by the spatial relationship and semantic similarity between text blocks. The edge features are obtained by calculating the relative position relationship between text blocks and the visual range features that include both.
[0036] cA represents the adjacency matrix of the graph, which reflects the weight of the edges between nodes, that is, the probability of the existence of an edge. It is calculated using the edge features obtained in b and the multi-head self-attention mechanism.
[0037] 4. Graph partitioning: This module mainly partitions the initial graph in 3 and iteratively updates the graph. Graph partitioning is mainly achieved by assigning nodes to different clusters: Assuming that the current number of nodes is N, first predefine the decay factor c to determine the number of clusters in the next layer N*c, use the graph convolutional network (GAT_1) for information interaction and node allocation matrix learning (N x N*c), and after the allocation is completed, it is input into another graph convolutional network (GAT_2) for information interaction between clusters. The dimension of the adjacency matrix of the graph changes from N x N to N*cx N*c. The block division of the graph is achieved by iterative multiple allocations.
[0038] 5. Graph depth encoding: This module updates the final representation of the node according to the node allocation matrix and cluster representation of each stage in 4. The nodes in the same cluster are encoded in horizontal position sorting according to the principle of "from left to right, from top to bottom"; the node allocation results are encoded in vertical position, and the hierarchical block information and local reading order are integrated into the node representation.
[0039] 6. Information extraction: This module performs node classification and link prediction based on the hidden layer output obtained in 5 and outputs the final result. Among them, link prediction takes the representation of two candidate nodes as input, and finally calculates the probability of the edge between them through a classifier.
[0040] Based on the pre-trained model and multimodal technology, the present invention captures the local structural information of the document and the spatial dependencies between text blocks through a graph neural network, enhances the spatial semantic association of text blocks in the multimodal fusion stage, and transforms the information extraction task into a graph node classification and link prediction problem, effectively improving the integrity of entity recognition and the performance of relationship extraction tasks.
[0041] In addition, the lightweight DBNet, CRNN, CenterLoss, RoBERTa, RoIAlign, Block mechanism, self-attention mechanism, iterative update mechanism, etc. mentioned in this embodiment are all existing technical means. The present invention is implemented by adopting the above existing technical means, and the specific details of the present invention will not be repeated.
[0042] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multimodal document information extraction method based on graph neural network, characterized in that: The steps include: Step S1: Optical Character Recognition (OCR): recognize the text in the image and convert it into text format to output the text block, and output the position coordinates of the text block at the same time; Step S2: Pre-coding: Extract features of the text blocks output by optical character recognition (OCR), including features of three modalities: text, vision, and layout. Perform feature fusion between modalities and output graph node features. Step S3: Graph construction: abstract the text block content and output an initial graph G=(V, E, A): where V represents graph nodes, E represents edges between nodes, and A represents the adjacency matrix of the graph; Step S4: Graph partitioning: The initial graph in step S3 is partitioned by assigning graph nodes to different clusters, and the assignment matrix of graph nodes is iteratively updated. The graph is partitioned by iterative multiple assignments. Step S5: Graph deep encoding: Update the final representation of the graph nodes according to the allocation matrix and cluster representation of each stage in step S4; Step S6: Information extraction: perform node classification and link prediction on the graph nodes in step S5, and output the final result; In step S3, E represents the edge between nodes, which is defined by the spatial relationship and semantic similarity between the text blocks. The edge features are obtained by calculating the relative position relationship between the text blocks and the visual range features of both. A represents the adjacency matrix of the graph, which is calculated by the obtained edge features using the multi-head self-attention mechanism. In the step S4, the graph partitioning is achieved by assigning graph nodes to different clusters, specifically including: assuming that the number of current graph nodes is N, first pre-define the attenuation factor c to determine the number of clusters N*c in the next layer, use the first graph convolutional network to perform information interaction and learn the graph node allocation matrix N x N*c, and after the allocation is completed, input it into another second graph convolutional network for information interaction between clusters, and the dimension of the adjacency matrix of the graph changes from N x N to N*cx N*c; and implement the graph partitioning by iterating multiple allocations.
2. The multimodal document information extraction method based on graph neural network according to claim 1 is characterized in that: In the step S1, the text in the image is recognized including text detection and text recognition, and the text block and the position coordinates of the text block are output through text detection and text recognition, wherein text detection uses a lightweight DBNet as a backbone network; text recognition uses CRNN for line text recognition, and CenterLoss is used to improve the recognition effect.
3. The multimodal document information extraction method based on graph neural network according to claim 1 is characterized in that: In step S2, RoBERTa is used to extract text features, and ResNet is used as the backbone network. The visual features within the corresponding text block are obtained according to the position coordinates of the text block through RoIAlign; and the initial layout features are generated according to the position coordinates and size of the text block; finally, the three modal features of the extracted text features, visual features, and layout features are fused using the Block mechanism to output the graph node features.
4. The multimodal document information extraction method based on graph neural network according to claim 1 is characterized in that: In step S5, the final representation of the graph nodes is updated, including: the graph nodes in the same cluster are encoded in horizontal position sorting according to the principle of "from left to right, from top to bottom"; the sequential allocation results of the graph nodes are encoded in vertical position, and the hierarchical block information and local reading order are integrated into the graph node representation.
5. The multimodal document information extraction method based on graph neural network according to claim 1 is characterized in that: In step S6, the link prediction is performed by concatenating the representations of two candidate graph nodes as input, and finally calculating the probability of the edge between the two through a classifier.
Citation Information
Cited By
A document structuring method based on multi-modal data fusion and dynamic rule driving and a storage medium
CN122903526A
