Visual document information extraction method based on attention-corrected graph neural network

Abstract

The present invention relates to a visual document information extraction method based on an attention-corrected graph neural network, which includes first extracting entity node images, text, and layout features based on the entity and its text content and text box information, obtaining the n entity node indexes closest to each entity node in the document to generate an edge adjacency matrix, and then inputting the entity node features and the edge adjacency matrix into the GraphSAGE graph network model to learn node features. This process integrates graph convolution with an attention correction mechanism based on structural features, calculates and corrects edge weights based on node feature distance and degree information, outputs feature representations after multi-layer graph convolution and correction operations, inputs them into a fully connected layer to complete node classification, samples documents to obtain entity node pairs and edge features, and then inputs them into a fully connected layer for edge classification after fusion. With the help of the GraphSAGE model, this method effectively integrates multimodal information, accurately captures node and graph structural characteristics, dynamically adjusts node relationships and weights, and significantly improves the accuracy of entity node classification.

Description

Technical Field

[0001] The present invention relates to a visual document information extraction method based on an attention-corrected graph neural network, which can effectively extract key information from visual documents and belongs to the technical field of visual document understanding. Background Art

[0002] A visual document is a type of document that contains text, text formatting information, and text images. Within the document processing field, visual document understanding is a key research area, aiming to automatically extract key information from documents to support various business applications. This process typically involves technologies such as automatic document classification, entity recognition, and semantic relationship extraction, significantly simplifying the complexity of document processing.

[0003] Document understanding tasks fall into two main categories: entity node classification and entity node linking. Entity node classification involves identifying and classifying entities within a document, such as names of people, places, and organizations. These entities often serve as the fundamental units of information extraction and are crucial for understanding document content and building knowledge graphs. Closely related to entity node classification is the entity node linking task, which involves extracting the relationships between entity nodes in a two-dimensional space within a document to construct an entity relationship graph.

[0004] Within the framework of GCN graph neural networks, document understanding generally involves five steps: node definition, edge definition, graph sampling, GCN processing, and description of a specific task. Currently, mainstream approaches can be divided into two types: single-stage and two-stage. Single-stage approaches first define entity node and edge features, then leverage the GCN graph neural network architecture to learn from each other. The resulting node and edge features are then used for different tasks. In contrast, two-stage approaches first learn entity node features through the GCN graph neural network to complete the entity node classification task. Then, graph sampling is performed on the original document to obtain node pairs, and edge features are defined. The entity node linking task is then processed through neural network layers. Although single-stage approaches can simultaneously learn node and edge features, the random learning between features often results in the loss of some node or edge feature information. This increases uncertainty in practical applications and reduces the expected effectiveness of both tasks.

[0005] The two-stage approach is based on the GCN graph neural network architecture, which is used for node learning when handling entity node classification tasks. Existing GCN graph neural network architectures use fixed-weight mean aggregation, treating all neighboring nodes as equally important. This approach fails to consider the varying influence of different neighboring nodes on the target node, and thus lacks differentiation of node relationships. This simple aggregation method limits the model's flexibility and makes it difficult to learn complex relationships between nodes. Consequently, it struggles to fully express node feature associations when processing complex graph data. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a visual document information extraction method based on an attention-corrected graph neural network. First, the multimodal information of the visual document is extracted and fused to obtain the entity node features and graph structure, and the GraphSAGE graph model is selected as the GCN architecture learning node. The attention correction mechanism is introduced into the learning process, and the edge weights are calculated and corrected according to the node feature distance and degree information. The neighbor features are aggregated and the node features are updated according to the weights through the GraphSAGE convolution layer. Multi-layer operation is possible, and the corrected features are then fused into the fully connected layer, and finally classified by the linear layer. This method can fuse multimodal information, accurately capture node and graph characteristics, dynamically adjust node relationships, improve the accuracy of entity node classification, and provide an efficient and accurate visual document information extraction solution.

[0007] The visual document information extraction method based on attention-corrected graph neural network includes the following steps:

[0008] Step 1: Use optical character recognition to obtain several entities in the document, as well as the text content, text box coordinates, and text box dimensions corresponding to the entities. Entity node layout features are represented based on the text box coordinates and length and width. A residual network is used to extract global image features of the document. A RoI alignment layer is then used to obtain regional entity node image features. Finally, the spacy model is used to extract text features of the document's entity nodes.

[0009] Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and its contribution to the overall task are calculated to generate the corresponding attention weight vector. These weight vectors are then used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature.

[0010] Step 3: For each entity node in the document, find the n nearest entity nodes, calculate the comprehensive distance between the upper left corner, lower right corner and center point of the entity node text box, and obtain the index of the nearest n entity nodes to generate the edge adjacency matrix;

[0011] Step 4: First, use the method of calculating attention weights based on node feature distance to aggregate neighbor node features obtained in step 2 and the edge adjacency matrix obtained in step 3 in the first layer of the GraphSAGE graph network model using a weighted sum operation. The aggregated neighbor features are then calculated using the attention weights. The node's own features are then concatenated with the aggregated neighbor features. The feature update is completed using the weight matrix and activation function to obtain the node features of the first layer. During the learning process, in the attention correction layer, the node degree information and the intermediate attention coefficient based on the node feature distance are combined to calculate the corrected attention weights and normalize them. This optimizes the edge weights and enhances the feature representation. The second layer of GraphSAGE convolution operation is then performed to aggregate neighbor features and update the node features using a weighted sum method.

[0012] Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement entity node classification through the linear layer;

[0013] Step 6: Graph sampling is performed on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined. The four features are combined to obtain the final edge features.

[0014] Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

[0015] The beneficial effects of the present invention are as follows:

[0016] The present invention proposes a visual document information extraction method based on an attention-corrected graph neural network, which focuses on optimizing the part of the graph neural network that learns entity node features. In this way, the model can comprehensively consider the similarity of node features and the characteristics of the graph structure, accurately learn the importance between different nodes and their neighbors, and then generate more refined and targeted node feature representations. In the task of visual document information extraction, this optimization effectively enhances the model's ability to capture the complex relationships between entity nodes, thereby significantly improving the accuracy of entity node classification, making the information extraction results more accurate and reliable, and is particularly suitable for processing visual document data with rich semantics and diverse structural relationships, providing a more efficient and accurate technical means for information processing and analysis in related fields. After completing the node classification, the more accurate entity node features obtained are used together with the original document entity node image features and the original document entity node text features as collaborative fusion features to define edge features, thereby solving the entity link problem. Experimental results show that this method has better robustness and adaptability. On the Funsd dataset, the F1 score for the entity classification task reached 81.97%, a 1.02% improvement over the model without attention correction. The F1 score for the entity node linking task was 84.32%, a 0.72% improvement over the model without attention correction. This method is at the forefront of the field of GCN-based graph model architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is the overall architecture diagram of key information extraction of the visual document information extraction method based on attention-corrected graph neural network of the present invention.

[0019] Figure 2 This is the architecture diagram of the graph neural network learning part of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] This embodiment provides a method for extracting visual document information based on a graph neural network with attention correction, such as Figure 1 As shown, the following steps are included:

[0022] Step 1: Use optical character recognition (OCR) to obtain several entities in the document, as well as the text content, text box coordinates, and text box dimensions corresponding to the entities. Based on this, the entity node layout features are represented according to the text box coordinates and length and width. A residual network is used to extract the global image features of the document. The RoI alignment layer is then used to obtain the regional entity node image features. Finally, the spacy model is used to extract the document entity node text features.

[0023] The step 1 specifically includes:

[0024] For the input visual document, use OCR technology to extract the entity text Text and its text box coordinates position = [x1, y1, x2, y2] in the document, where the above coordinates represent the coordinates of the upper left corner and lower right corner of the text box respectively, and obtain the width W and height H of the document node, which serve as the data foundation and basis for subsequent operations;

[0025] Use the resnet50 backbone network and RoI alignment layer to extract the image feature V of document entity node i image , which follows the following formula:

[0026] V image =Linear(Pool align (Region i ,ResNet50(I))),0≤i≤n

[0027] Linear represents the linear projection layer, Pool represents the RoIAlign pooling operation, and Region i represents the area of ​​the entity node, and I represents the original document image input to the backbone network;

[0028] Get the entity node text feature V of the document through the spacy model text As shown in the following formula:

[0029] V text =Linear(spaCy(t i )),0≤i≤n

[0030] Among them, Linear is a linear projection layer, spaCy represents the text feature extraction model, and t i Represents the text content of the entity; defines the document entity node layout characteristics V according to the text box information layout , as shown in the following formula:

[0031] Vlayout =Linear([x1,y1,x2,y2,C x ,C y ,B w ,B h ]),0≤i≤n

[0032] Where Linear is a linear projection layer, x1,y1,x2,y2,C x ,C y ,B w ,B h They represent the upper left corner coordinates, lower right corner coordinates, center coordinates, width and height of the entity node text box respectively.

[0033] Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and its contribution to the overall task are calculated to generate the corresponding attention weight vector. These weight vectors are then used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature.

[0034] The multimodal feature fusion method in step 2 specifically includes:

[0035] The acquired entity node image features, entity node text features, and entity node layout features are used as input in sequence. The feature definitions are as follows:

[0036]

[0037] Where n represents the number of entity nodes, d image d text d layout Represent the dimensions of the three modal features respectively,

[0038] In order to calculate the attention weight, vectors of different dimensions are mapped to a common semantic space through a linear projection layer, which is expressed as follows:

[0039]

[0040] The mapped vectors are concatenated and passed through a fully connected layer with a weight matrix W, and then normalized using the softmax function to obtain the attention weight w attn , the expression is as follows:

[0041]

[0042] w attn =softmax(V concat W)

[0043] [w image ,wtext ,w layout ]=W attn

[0044] The calculated attention weight vector is used to perform weighted fusion on the original layout features, image features, and text features to obtain the fused entity node features. The expression is as follows:

[0045] Entity fusion-feat =w image V image ′+w text V text ′+w layout V layout ′.

[0046] Step 3: For each entity node in the document, find the n nearest entity nodes. Calculate the distances between the upper left corner, lower right corner, and center of the entity node text box to obtain the indexes of the n nearest entity nodes and use them to generate the edge adjacency matrix.

[0047] Step three specifically includes:

[0048] Based on the coordinates of the upper left corner, lower right corner, and center point of the text box of the entity node in the document obtained in step 1, the distance between entities is defined based on the squared Euclidean distance. The expression is as follows:

[0049] Dis ij =AVG(DL ij +DR ij +DC ij )0≤i≤n,0≤j≤n,i!=j

[0050] Among them DL ij , DR ij 、DC ij Respectively represent the distances between entities at the upper left corner, lower right corner, and center point;

[0051] According to the above formula, one third of the total number of entity nodes in the document is customized to generate the edge adjacency matrix of the entity node The adjacency matrix is ​​a two-dimensional matrix that stores the associated entity node indices.

[0052] Step 4: First, use the method of calculating attention weights based on node feature distance to aggregate neighbor node features obtained in step 2 and the edge adjacency matrix obtained in step 3 in the first layer of the GraphSAGE graph network model using a weighted sum operation. The aggregated neighbor features are then calculated using the attention weights. The node's own features are then concatenated with the aggregated neighbor features. The feature update is completed using the weight matrix and activation function to obtain the node features of the first layer. During the learning process, in the attention correction layer, the node degree information and the intermediate attention coefficient based on the node feature distance are combined to calculate the corrected attention weights and normalize them. This optimizes the edge weights and enhances the feature representation. The second layer of GraphSAGE convolution operation is then performed to aggregate neighbor features and update the node features using a weighted sum method.

[0053] The attention-corrected GraphSAGE model in step 4 specifically includes:

[0054] Let G = (V, E) be a graph, where V is the set of entity nodes and E is the set of edges. For a node v∈V, its eigenvector is x v , the set of neighbor nodes is N(v);

[0055] For an edge (u, v)∈E, where u∈N(v), the attention weight w is calculated based on the node feature distance vu , use Euclidean distance to measure node feature distance, the calculation expression is:

[0056]

[0057] Get the entity node feature Entity in step 2 fusion-feat , the adjacency matrix Edge_Index in step 3, input the entity node features and the adjacency matrix into the first layer of GraphSAGE convolution layer, and consider the attention weight when aggregating neighbor node features The calculation formula is:

[0058]

[0059] The purpose of SAGE is to update the entity node through the adjacent nodes according to the adjacency matrix. The following is the feature transfer method of SAGE model node v:

[0060]

[0061] The CONCAT operation combines the node's own features with the aggregated neighbor features. Splice it together;

[0062] In the graph-based attention correction layer, the attention weight w calculated for the edge (u, v) in the first layer is vuMake corrections. First calculate the node degree related information, the expression is as follows:

[0063]

[0064] where d v is the degree of node v in graph G, where d u is the degree of node u in graph G;

[0065] Calculate degree-dependent normalization factors The expression is as follows:

[0066]

[0067] Calculate the corrected attention weights And use the softmax function for normalization, the expression is as follows:

[0068]

[0069] The output of the first layer Input to the second layer of GraphSAGE convolution layer, similar to the first layer, first according to the corrected edge weights Aggregate neighbor features and then update node features The expression is as follows:

[0070]

[0071] The final feature representation of the entity node can be obtained through the above modified GraphSAGE model layer:

[0072] Entity Final-feat =GraphSAGE_Revised(TB,Entity fusion-feat )

[0073] Where TB is the text box coordinate.

[0074] Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement entity node classification through the linear layer;

[0075] Step 5 specifically includes:

[0076] For the node classification task, the entity node feature Entity Final-feat Input to the fully connected layer for feature fusion and use the linear layer W h*c Classify the entity nodes, where H represents the hidden layer and c represents the number of categories of the entity nodes.

[0077] Step 6: Graph sampling is performed on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined. The four features are combined to obtain the final edge features.

[0078] The specific sampling method in step 6 includes:

[0079] Entity node pairs are constructed based on the original document. Potential edge connections between nodes are constructed using a position-based node search algorithm. This approach abandons the fully connected graph construction model, achieving the dual effects of cost savings and improved training efficiency.

[0080] Given the document's structural characteristics, each entity node has potential edge connections in both vertical and horizontal directions. The distance between entities is measured using a squared Euclidean distance matrix. For each entity text box, operations are performed in the four directions: up, down, left, and right. First, the nearest neighboring text box is identified and a connection is established. Then, the next nearest neighboring text box is identified and a connection is established. Finally, an index list (edge_index_list) representing these connection relationships is generated, accurately identifying eligible connections between entity nodes.

[0081] The method of defining edge features is:

[0082] The edge features are defined based on the following basic features and the index list edge_index_list of the connection relationship:

[0083] (1) Relationship feature F between entity nodes relation : According to the relationship suggestion, assuming that two nodes have a potential relationship, define a primary node S and another secondary node O, and the relationship between the two is represented by R, so:

[0084]

[0085] where x S 、y S 、w S 、h S Indicates the center coordinates, width and height of the main node. O 、y O 、w O 、h O Indicates the center point coordinates, width and height of the secondary node;

[0086] Get the minimum bounding box of S and O as R, which is defined as follows:

[0087]

[0088] The relationship feature definition between entity nodes can be:

[0089]

[0090] (2) Entity node natural reading order encoding F rope It plays a key role in accurately depicting the sequential relationship between nodes.

[0091] First, for each entity node i, let its adjacent entity node be j, and use the index difference between them as the sequence length of the entity node pair (i, j). A new and unique reading order code is assigned to the adjacent entity nodes, and a sequence length system for entity node pairs is constructed. Based on the constructed reading order index system, the reading order index of each entity node pair is substituted into the sine encoding matrix and cosine encoding matrix to obtain encoding information. The reading order index is comprehensively and carefully encoded. The expression is as follows:

[0092] F rope =RoPE_Emb(pos)

[0093] (3) Polar coordinate feature F polar , represents the distance and angle encoding of entity node pairs: for the entity node pairs generated by graph sampling, to obtain the distance and angle of each entity node pair. The distance and angle pairs are encoded using the one-hot encoding method. The specific formula is as follows:

[0094] F polar =one_hot(distances,angle,num_polar_bins)

[0095] Where distances and angles are the distance and angle sets of entity node pairs, respectively. num_polar_bins represents the encoded dimension.

[0096] (4) For entity node pair features: obtain the index list edge_index_list of the connection relationship as the entity node pair, and obtain the entity node feature Entity of the three modalities of image, text, and layout in step 4 Final-feat , get the entity node feature F pair , the specific formula is as follows:

[0097]

[0098] Entity i , Entity j Represents the node features of entity node i and entity node j in an entity node pair;

[0099] By fusing the above four features, we can get the final edge feature F edge , the expression is as follows:

[0100]

[0101] Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

[0102] First obtain the edge feature F of step six edge , represents the key data representation of node association, inputs the fully connected layer, completes feature fusion, and then uses a specific weight matrix W h*2 The linear layer infers the existence of node-to-edge edges based on weight parameters and activation functions. Finally, the nodes that meet the connection rules and threshold judgment and are interconnected are determined as the final layout instances.

[0103] This method uses the GraphSAGE model to effectively integrate multimodal information, accurately capture the characteristics of nodes and graph structures, dynamically adjust node relationships and weights, and significantly improve the accuracy of entity node classification.

[0104] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A visual document information extraction method based on attention-corrected graph neural networks, characterized by: The following steps are involved: Step 1: Use optical character recognition to obtain several entities in the document, as well as the text content, text box coordinates, and text box dimensions corresponding to the entities. Entity node layout features are represented based on the text box coordinates and length and width. A residual network is used to extract global image features of the document. A RoI alignment layer is then used to obtain regional entity node image features. Finally, the spacy model is used to extract text features of the document's entity nodes. Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and its contribution to the overall task are calculated to generate the corresponding attention weight vector. These weight vectors are then used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature. Step 3: For each entity node in the document, find the n nearest entity nodes, calculate the comprehensive distance between the upper left corner, lower right corner and center point of the entity node text box, and obtain the index of the nearest n entity nodes to generate the edge adjacency matrix; Step 4: First, use the method of calculating attention weights based on node feature distance to aggregate neighbor node features obtained in step 2 and the edge adjacency matrix obtained in step 3 in the first layer of the GraphSAGE graph network model using a weighted sum operation. The aggregated neighbor features are then calculated using the attention weights. The node's own features are then concatenated with the aggregated neighbor features. The feature update is completed using the weight matrix and activation function to obtain the node features of the first layer. During the learning process, in the attention correction layer, the node degree information and the intermediate attention coefficient based on the node feature distance are combined to calculate the corrected attention weights and normalize them. This optimizes the edge weights and enhances the feature representation. The second layer of GraphSAGE convolution operation is then performed to aggregate neighbor features and update the node features using a weighted sum method. Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement entity node classification through the linear layer; Step 6: Graph sampling is performed on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined. The four features are combined to obtain the final edge features. Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

2. The method for extracting visual document information using a graph neural network with attention correction according to claim 1, characterized in that: The step 1 specifically includes: For the input visual document, use OCR technology to extract the entity text Text and its text box coordinates position = [x1, y1, x2, y2] in the document, where the above coordinates represent the coordinates of the upper left corner and lower right corner of the text box respectively, and obtain the width W and height H of the document node as the data foundation and basis for subsequent operations; Use the resnet50 backbone network and RoI alignment layer to extract the image feature V of document entity node i image , which follows the following formula: V image =Linear(Pool align (Region i ,ResNet50(I))),0≤i≤n Linear represents the linear projection layer, Pool represents the RoIAlign pooling operation, and Region i represents the area of ​​the entity node, and I represents the original document image input to the backbone network; Get the entity node text feature V of the document through the spacy model text As shown in the following formula: V text =Linear(spaCy(t i )),0≤i≤n Among them, Linear is a linear projection layer, spaCy represents the text feature extraction model, and t i Represents the text content of the entity; defines the document entity node layout characteristics V according to the text box information layout , as shown in the following formula: V layout =Linear([x1,y1,x2,y2,C x ,C y ,B w ,B h ]),0≤i≤n Where Linear is a linear projection layer, x1,y1,x2,y2,C x ,C y ,B w ,B h They represent the upper left corner coordinates, lower right corner coordinates, center coordinates, width and height of the entity node text box respectively.

3. The method for extracting visual document information based on attention-corrected graph neural networks according to claim 2, characterized in that: The multimodal feature fusion method in step 2 specifically includes: The acquired entity node image features, entity node text features, and entity node layout features are used as input in sequence. The feature definitions are as follows: Where n represents the number of entity nodes, d image d text d layout Represent the dimensions of the three modal features respectively, Calculate the attention weight and map vectors of different dimensions into a common semantic space through a linear projection layer. The expression is as follows: The mapped vectors are concatenated and passed through a fully connected layer with a weight matrix W, and then normalized using the softmax function to obtain the attention weight w attn , the expression is as follows: w attn =softmax(V concat W) [In image ,In text ,In layout ]=W attn The calculated attention weight vector is used to perform weighted fusion on the original layout features, image features, and text features to obtain the fused entity node features. The expression is as follows: Entity fusion-feat =w image V image ′+w text V text ′+w layout V layout 。 4. The method for extracting visual document information using a graph neural network with attention correction according to claim 3, characterized in that: The step three specifically includes: Based on the coordinates of the upper left corner, lower right corner, and center point of the text box of the entity node in the document obtained in step 1, the distance between entities is defined based on the squared Euclidean distance. The expression is as follows: Dis ij =AVG(DL ij +DR ij +DC ij )0≤i≤n,0≤j≤n,i!=j Among them DL ij , DR ij 、DC ij Respectively represent the distances between entities at the upper left corner, lower right corner, and center point; According to the above formula, one third of the total number of entity nodes in the document is customized to generate the edge adjacency matrix of the entity node The adjacency matrix is ​​a two-dimensional matrix that stores the associated entity node indices.

5. The method for extracting visual document information based on attention-corrected graph neural networks according to claim 4, characterized in that: The attention-corrected GraphSAGE model in step 4 specifically includes: Let G = (V, E) be a graph, where V is the set of entity nodes and E is the set of edges. For a node v∈V, its eigenvector is x v , the set of neighbor nodes is N(v); For an edge (u, v)∈E, where u∈N(v), the attention weight w is calculated based on the node feature distance vu , use Euclidean distance to measure node feature distance, the calculation expression is: Get the entity node feature Entity in step 2 fusion-feat , the adjacency matrix Edge_Index in step 3, input the entity node features and the adjacency matrix into the first layer of GraphSAGE convolution layer, and consider the attention weight when aggregating neighbor node features The calculation formula is: The purpose of SAGE is to update the entity node through the adjacent nodes according to the adjacency matrix. The following is the feature transfer method of SAGE model node v: The CONCAT operation combines the node's own features with the aggregated neighbor features. Splice it together; In the graph-based attention correction layer, the attention weight w calculated for the edge (u, v) in the first layer is vu To make corrections, first calculate the node degree related information, the expression is as follows: where d v is the degree of node v in graph G, where d u is the degree of node u in graph G; Calculate degree-dependent normalization factors The expression is as follows: Calculate the corrected attention weights And use the softmax function for normalization, the expression is as follows: The output of the first layer Input to the second layer of GraphSAGE convolution layer, similar to the first layer, first according to the corrected edge weights Aggregate neighbor features and then update node features The expression is as follows: The final feature representation of the entity node can be obtained through the above modified GraphSAGE model layer: Entity Final-feat =GraphSAGE_Revised(TB,Entity fusion-feat ) Where TB is the text box coordinate.

6. The method for extracting visual document information based on attention-corrected graph neural networks according to claim 1, characterized in that: The step five specifically includes: For the node classification task, the entity node feature Entity Final-feat Input to the fully connected layer to fuse features and make With the linear layer W h*c Classify the entity nodes, where H represents the hidden layer and c represents the number of categories of the entity nodes.

7. The method for extracting visual document information based on attention-corrected graph neural networks according to claim 1, characterized in that: The image sampling method in step 6 specifically includes: For each entity text box, operations are performed in the four directions of up, down, left, and right: first, the nearest neighbor text box is determined and connected, then the next nearest neighbor text box is determined and connected. Finally, an index list edge_index_list representing these connection relationships is formed to accurately determine the qualified associations between entity nodes. The method of defining edge features is: The edge features are defined based on the following basic features and the index list edge_index_list of the connection relationship: For the relationship feature F between entity nodes relation : According to the relationship suggestion, assuming that two nodes have a potential relationship, define a primary node S and another secondary node O, and the relationship between the two is represented by R, so: where x S 、y S 、w S 、h S Indicates the center point coordinates, width and height of the main node, x O 、y O 、w O 、h O Indicates the center point coordinates, width and height of the secondary node; Get the minimum bounding box of S and O as R, which is defined as follows: The relationship feature definition between entity nodes can be: For entity nodes, natural reading order encoding F rope :First, for each entity node i, let its adjacent entity node be j, and use the index difference between them as the sequence length of the entity node pair (i, j). New and unique reading order codes are assigned to adjacent entity nodes, and a sequence length system for entity node pairs is constructed. Based on the constructed reading order index system, the reading order index of each entity node pair is substituted into the sine encoding matrix and cosine encoding matrix to obtain encoding information. The encoding operation is performed on the reading order index in a comprehensive and detailed manner. The expression is as follows: F rope =RoPE_Emb(pos) For the polar coordinate feature F polar : To obtain the distance and angle of each entity node pair generated by graph sampling, encode the distance and angle pairs using one-hot encoding. The specific formula is as follows: F polar =one_hot(distances,angles,num_polar_bins) Where distances and angles are the distance set and angle set of entity node pairs respectively, and num_polar_bins represents the encoded dimension. For entity node pair features: Get the index list edge_index_list of the connection relationship as the entity node pair, and get the entity node feature Entity of the three modalities of image, text, and layout in step 4 Final-feat , get the entity node feature F pair , the specific formula is as follows: Entity i , Entity j Represents the node features of entity node i and entity node j in an entity node pair; By combining the above four features, we can get the final edge feature F edge , the expression is as follows:

8. The method for extracting visual document information based on attention-corrected graph neural networks according to claim 1, characterized in that: The step seven specifically includes: First obtain the edge feature F of step six edge , represents the key data representation of node association, inputs the fully connected layer, completes feature fusion, and uses a specific weight matrix W h*2 The linear layer infers the existence of node-to-node edges based on weight parameters and activation functions, and finally determines the interconnected nodes that meet the connection rules and threshold judgment as the final layout instances.

Images