Visual document information extraction method of graph neural network based on attention correction

Abstract

The invention relates to a visual document information extraction method based on an attention correction graph neural network, and the method comprises the steps: firstly extracting entity node images, texts and layout features according to entities, text contents and textbox information, and obtaining n entity node indexes closest to each entity node in a document to generate an edge adjacency matrix; the entity node features and the edge adjacency matrix are input into a GraphSAGE graph network model to learn node features, graph convolution and an attention correction mechanism based on structural features are fused in the process, the edge weight is calculated and corrected according to node feature distance and degree information, feature representation is output after multi-layer graph convolution and correction operation, and the edge adjacency matrix is obtained. And inputting the full connection layer to complete node classification, sampling the document to obtain entity node pairs and edge features, and inputting the full connection layer to perform edge classification after fusion. According to the method, by means of the GraphSAGE model, multi-modal information is effectively fused, the node and graph structure characteristics are accurately captured, the node relation and weight are dynamically adjusted, and the entity node classification accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention relates to a visual document information extraction method based on an attention-corrected graph neural network, which can effectively extract key information from a visual document and belongs to the technical field of visual document understanding. Background Art

[0002] A visual document is a type of document that contains text, text formatting information, and text image information. In the field of document processing, visual document understanding is a key research direction that aims to automatically extract key information from documents to support various business purposes. This process usually involves technologies such as automatic document classification, entity recognition, and semantic relationship extraction, which greatly simplifies the complexity of document processing.

[0003] Document understanding tasks are mainly divided into two categories: entity node classification and entity node linking. The entity node classification task involves identifying and classifying various entities in the document, such as names of people, places, and organization names. These entities are usually used as the basic units for information extraction and are important for understanding document content and building knowledge graphs. Closely related to entity node classification is the entity node linking task, which obtains the relationship between entity nodes in a two-dimensional space from the document to construct an entity relationship graph.

[0004] In the framework of GCN graph neural network, document understanding generally includes five steps: node definition, edge definition, graph sampling, GCN processing and description of specific tasks. At present, mainstream methods can be divided into two types: single-stage and two-stage. The single-stage method first defines the features of entity nodes and edges, and then uses the architecture of GCN graph neural network to learn node and edge features from each other. The node and edge features finally obtained are used for different tasks. The two-stage method first learns entity node features through GCN graph neural network to complete the entity node classification task; then, the original document is sampled to obtain node pairs, and edge features are defined, and the entity node link task is processed through the neural network layer. Although the single-stage method can learn node and edge features at the same time, due to the influence of random learning between features, it often leads to the loss of some node or edge feature information, which increases uncertainty in practical applications and reduces the expected effects of the two types of tasks.

[0005] The two-stage method is based on the GCN graph neural network architecture. When processing entity node classification tasks, the GCN graph neural network architecture is used for node learning. The existing GCN graph neural network architecture uses fixed-weight mean aggregation and treats all neighbor nodes as equally important. This method does not consider the different influences of different neighbor nodes on the target node, and thus lacks differentiation of node relationships. This simple aggregation method limits the flexibility of the model and makes it difficult to learn complex relationships between nodes. Therefore, it is difficult to fully express the feature associations of nodes when processing complex graph data. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a visual document information extraction method based on an attention-corrected graph neural network. First, extract and fuse the multimodal information of the visual document to obtain the entity node features and graph structure, and select the GraphSAGE graph model as the GCN architecture learning node. In the learning, an attention correction mechanism is introduced to calculate and correct the edge weights based on the node feature distance and degree information. The neighbor features are aggregated and the node features are updated according to the weights through the GraphSAGE convolution layer. Multi-layer operation is possible, and the corrected features are then fused into the fully connected layer, and finally classified in the linear layer. This method can fuse multimodal information, accurately capture node and graph characteristics, dynamically adjust node relationships, improve the accuracy of entity node classification, and provide an efficient and accurate visual document information extraction solution.

[0007] The method for extracting visual document information based on attention-corrected graph neural network includes the following steps:

[0008] Step 1: Use optical character recognition to obtain several entities in the document and the text content, text box coordinate values ​​and text box size corresponding to the entities. According to the text box coordinate values ​​and length and width, the entity node layout features are represented. The residual network is used to extract the global image features of the document. The RoI alignment layer is then used to obtain the regional-level entity node image features. The spacy model is used to extract the entity node text features of the document.

[0009] Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and the contribution to the overall task are calculated to generate the corresponding attention weight vector, and then these weight vectors are used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature;

[0010] Step 3: For each entity node in the document, find the n nearest entity nodes, calculate the comprehensive distance between the upper left corner, lower right corner and center point of the entity node text box, and obtain the indexes of the nearest n entity nodes to generate the edge adjacency matrix;

[0011] Step 4: First, use the method of calculating attention weights based on node feature distance. For the entity node features obtained in step 2 and the edge adjacency matrix in step 3, in the first layer of the GraphSAGE graph network model, use weighted sum operation to aggregate neighbor node features and calculate the aggregated neighbor features in combination with attention weights. Then, concatenate the node's own features with the aggregated neighbor features, complete feature update through weight matrix and activation function to obtain the node features of the first layer. In the learning process, in the attention correction layer, the corrected attention weights are calculated and normalized by integrating node degree information and intermediate attention coefficients based on node feature distance, so as to optimize edge weights and enhance feature representation. Then, the second layer of GraphSAGE convolution operation is performed to aggregate neighbor features in a weighted sum manner and update node features.

[0012] Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement the classification of entity nodes through the linear layer;

[0013] Step 6: Perform graph sampling on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined to obtain the final edge features.

[0014] Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

[0015] The beneficial effects of the present invention are as follows:

[0016] The present invention proposes a visual document information extraction method based on an attention-corrected graph neural network, which focuses on optimizing the part of the graph neural network that learns entity node features. In this way, the model can comprehensively consider the node feature similarity and graph structure characteristics, accurately learn the importance between different nodes and their neighbors, and then generate a more refined and targeted node feature representation. In the task of visual document information extraction, this optimization effectively enhances the model's ability to capture the complex relationships between entity nodes, thereby significantly improving the accuracy of entity node classification, making the information extraction results more accurate and reliable, especially suitable for processing visual document data with rich semantics and diverse structural relationships, and providing more efficient and accurate technical means for information processing and analysis in related fields. After completing the node classification, the more accurate entity node features obtained are used together with the original document entity node image features and the original document entity node text features as collaborative fusion features to define edge features, thereby solving the entity link problem. Experimental results show that this method has better robustness and adaptability. On the Funsd dataset, the F1 value of the entity classification task reached 81.97%, which is 1.02% higher than the model without attention correction; the F1 value of the entity node link task was 84.32%, which is 0.72% higher than the model without attention correction. This method is at the leading level in the field of GCN graph model architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is the overall architecture diagram of key information extraction of the visual document information extraction method based on the attention-corrected graph neural network of the present invention.

[0019] Figure 2 It is an architectural diagram of the graph neural network learning part of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] This embodiment provides a visual document information extraction method based on attention-corrected graph neural network, such as Figure 1 As shown, the following steps are included:

[0022] Step 1: Use optical character recognition (OCR) to obtain several entities in the document and the text content, text box coordinate values ​​and text box size corresponding to the entities. On this basis, the entity node layout features are represented according to the text box coordinate values ​​and length and width. The document global image features are extracted using the residual network, and then the regional entity node image features are obtained using the RoI alignment layer. The entity node text features of the document are extracted through the spacy model.

[0023] The step 1 specifically includes:

[0024] For the input visual document, use OCR technology to extract the entity text Text in the document and its text box coordinates position = [x1, y1, x2, y2], where the above coordinates represent the coordinates of the upper left corner and lower right corner of the text box respectively, and obtain the width W and height H of the document node, which will serve as the data basis and basis for subsequent operations;

[0025] Use the resnet50 backbone network and RoI alignment layer to extract the image feature V of document entity node i image , which follows the following formula:

[0026] V image =Linear(Pool align (Region i ,ResNet50(I))),0≤i≤n

[0027] Linear represents the linear projection layer, Pool represents the RoIAlign pooling operation, and Region i represents the area of ​​the entity node, and I represents the original document image input to the backbone network;

[0028] Get the entity node text feature V of the document through the spacy model text As shown in the following formula:

[0029] V text =Linear(spaCy(t i )),0≤i≤n

[0030] Linear is a linear projection layer, spaCy represents the text feature extraction model, and t i Represents the text content of the entity; defines the document entity node layout feature V according to the text box information layout , as shown in the following formula:

[0031] Vlayout =Linear([x1,y1,x2,y2,C x ,C y ,B w ,B h ]),0≤i≤n

[0032] Where Linear is a linear projection layer, x1, y1, x2, y2, C x ,C y ,B w ,B h They respectively represent the upper left corner coordinates, lower right corner coordinates, center coordinates, width and height of the entity node text box.

[0033] Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and the contribution to the overall task are calculated to generate the corresponding attention weight vector, and then these weight vectors are used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature;

[0034] The multimodal feature fusion method in step 2 specifically includes:

[0035] The acquired entity node image features, entity node text features, and entity node layout features are used as inputs in sequence. The feature definitions are as follows:

[0036]

[0037] Where n represents the number of entity nodes, d image ,d text ,d layout Represent the dimensions of the three modal features respectively,

[0038] In order to calculate the attention weight, vectors of different dimensions are mapped to a common semantic space through a linear projection layer, which is expressed as follows:

[0039]

[0040] The mapped vectors are concatenated and passed through a fully connected layer with a weight matrix W, and then normalized using the softmax function to obtain the attention weight w attn , the expression is as follows:

[0041]

[0042] w attn =softmax(V concat W)

[0043] [w image ,wtext ,w layout ]=W attn

[0044] The calculated attention weight vector is used to perform weighted fusion on the original layout features, image features, and text features to obtain the fused entity node features. The expression is as follows:

[0045] Entity fusion-feat =w image V image ′+w text V text ′+w layout V layout ′.

[0046] Step 3: For each entity node in the document, find the n entity nodes closest to it, calculate the comprehensive distance between the upper left corner, lower right corner and center point of the entity node text box, and obtain the indexes of the nearest n entity nodes to generate the edge adjacency matrix;

[0047] Step three specifically includes:

[0048] According to the coordinates of the upper left corner, lower right corner and center point of the text box of the entity node in the document obtained in step 1, the distance between entities is defined based on the squared Euclidean distance. The expression is as follows:

[0049] Dis ij =AVG(DL ij +DR ij +DC ij )0≤i≤n,0≤j≤n,i!=j

[0050] Among them DL ij ,DR ij 、DC ij Respectively represent the distances between entities at the upper left corner, lower right corner, and center point;

[0051] According to the above formula, one third of the total number of entity nodes in the document is customized to generate the edge adjacency matrix of the entity node The adjacency matrix is ​​a two-dimensional matrix that stores the associated entity node indices.

[0052] Step 4: First, use the method of calculating attention weights based on node feature distance. For the entity node features obtained in step 2 and the edge adjacency matrix in step 3, in the first layer of the GraphSAGE graph network model, use weighted sum operation to aggregate neighbor node features and calculate the aggregated neighbor features in combination with attention weights. Then, concatenate the node's own features with the aggregated neighbor features, complete feature update through weight matrix and activation function to obtain the node features of the first layer. In the learning process, in the attention correction layer, the corrected attention weights are calculated and normalized by integrating node degree information and intermediate attention coefficients based on node feature distance, so as to optimize edge weights and enhance feature representation. Then, the second layer of GraphSAGE convolution operation is performed to aggregate neighbor features in a weighted sum manner and update node features.

[0053] The attention-corrected GraphSAGE model in step 4 specifically includes:

[0054] Let G = (V, E) be a graph, where V is the set of entity nodes and E is the set of edges. For a node v∈V, its feature vector is x v , the set of neighbor nodes is N(v);

[0055] For an edge (u,v)∈E, where u∈N(v), the attention weight w is calculated based on the node feature distance vu , use Euclidean distance to measure node feature distance, and the calculation expression is:

[0056]

[0057] Get the entity node feature Entity in step 2 fusion-feat , the adjacency matrix Edge_Index in step 3, input the entity node features and the adjacency matrix into the first layer of GraphSAGE convolutional layer, and consider the attention weight when aggregating neighbor node features The calculation formula is:

[0058]

[0059] The purpose of SAGE is to update the entity node through the adjacent nodes according to the adjacency matrix. The following is the feature transfer method of SAGE model node v:

[0060]

[0061] The CONCAT operation combines the node's own features with the aggregated neighbor features. Splice it together;

[0062] In the graph-based attention correction layer, the attention weight w calculated for the edge (u,v) in the first layer is vuMake corrections. First, calculate the node degree related information, the expression is as follows:

[0063]

[0064] where d v is the degree of node v in graph G, where d u is the degree of node u in graph G;

[0065] Calculate the degree-dependent normalization factor The expression is as follows:

[0066]

[0067] Calculate the corrected attention weights And use the softmax function for normalization, the expression is as follows:

[0068]

[0069] The output of the first layer Input to the second GraphSAGE convolutional layer, similar to the first layer, first according to the corrected edge weights Aggregate neighbor features and then update node features The expression is as follows:

[0070]

[0071] The final feature representation of the entity node can be obtained through the above modified GraphSAGE model layer:

[0072] Entity Final-feat =GraphSAGE_Revised(TB,Entity fusion-feat )

[0073] Where TB is the text box coordinates.

[0074] Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement the classification of entity nodes through the linear layer;

[0075] Step 5 specifically includes:

[0076] For the node classification task, the entity node feature Entity Final-feat Input to the fully connected layer for feature fusion and use the linear layer W h*c Classify the entity nodes, where H represents the hidden layer and c represents the number of categories of the entity nodes.

[0077] Step 6: Perform graph sampling on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined to obtain the final edge features.

[0078] The sampling method in step 6 specifically includes:

[0079] The construction of entity node pairs is carried out based on the original document. The potential edge connections between nodes are constructed through the position-based node search algorithm. This method abandons the fully connected graph construction mode and achieves the dual effects of cost savings and improved training efficiency.

[0080] Given the characteristics of the document structure, each entity node has the possibility of potential edge connection in both vertical and horizontal directions. The distance between entities is measured using the squared Euclidean distance matrix. For each entity text box, operations are performed in the four directions of up, down, left, and right: first determine the text box with the nearest neighbor position and build a connection, then determine the text box with the second nearest neighbor position to complete the connection construction, and finally form an index list edge_index_list that represents these connection relationships, so as to accurately determine the qualified association situations between entity nodes.

[0081] The method to define edge features is:

[0082] The edge features are defined based on the following basic features and the edge_index_list index list of the connection relationship:

[0083] (1) Relationship feature F between entity nodes relation : According to the relationship suggestion, assuming that two nodes have a potential relationship, define a primary node S and another secondary node O, and the relationship between the two is represented by R, so:

[0084]

[0085] where x S ,y S 、w S 、h S Indicates the center point coordinates, width and height of the main node. O ,y O 、w O 、h O Indicates the center point coordinates, width and height of the secondary node;

[0086] Get the minimum bounding box of S and O as R, which is defined as follows:

[0087]

[0088] The relationship feature definition between entity nodes can be:

[0089]

[0090] (2) Entity node natural reading order encoding F rope It plays a key role in accurately describing the sequential relationship between nodes.

[0091] First, for each entity node i, let its adjacent entity node be j, and use the index difference between them as the sequence length of the entity node pair (i, j), give the adjacent entity nodes a new and unique reading order code, and build a sequence length system for entity node pairs. According to the constructed reading order index system, for the reading order index of each entity node pair, substitute it into the sine encoding matrix and the cosine encoding matrix to obtain the encoding information, and perform encoding operations on the reading order index comprehensively and meticulously. The expression is as follows:

[0092] F rope =RoPE_Emb(pos)

[0093] (3) Polar coordinate feature F polar , represents the distance and angle encoding of entity node pairs: to obtain the distance and angle of each entity node pair generated by graph sampling. Encode the distance and angle pair set by one-hot encoding. The specific formula is as follows:

[0094] F polar =one_hot(distances,angle,num_polar_bins)

[0095] Where distances and angles are the distance set and angle set of entity node pairs respectively. num_polar_bins represents the encoded dimension.

[0096] (4) For entity node pair features: obtain the index list edge_index_list of the connection relationship as the entity node pair, and obtain the entity node feature Entity of the three modalities of image, text, and layout in step 4 Final-feat , get the entity node pair feature F pair , the specific formula is as follows:

[0097]

[0098] Entity i , Entity j Represents the node features of entity node i and entity node j in an entity node pair;

[0099] By combining the above four features, we can get the final edge feature F edge , the expression is as follows:

[0100]

[0101] Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

[0102] First obtain the edge feature F of step six edge , represents the key data representation of node association, inputs the fully connected layer, completes feature fusion, and then uses a specific weight matrix W h*2 The linear layer infers the existence of the node edges based on the weight parameters and activation function. Finally, the nodes that meet the connection rules and threshold judgment and are interconnected are determined as the final layout instances.

[0103] With the help of the GraphSAGE model, this method effectively integrates multimodal information, accurately captures the characteristics of nodes and graph structures, dynamically adjusts node relationships and weights, and significantly improves the accuracy of entity node classification.

[0104] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A visual document information extraction method based on attention-corrected graph neural network, characterized by: The following steps are involved: Step 1: Use optical character recognition to obtain several entities in the document and the text content, text box coordinate values ​​and text box size corresponding to the entities. According to the text box coordinate values ​​and length and width, the entity node layout features are represented. The residual network is used to extract the global image features of the document. The RoI alignment layer is then used to obtain the regional-level entity node image features. The spacy model is used to extract the entity node text features of the document. Step 2: For the entity node image features, entity node text features, and entity node layout features obtained in step 1, the interaction relationship between each modal feature and the contribution to the overall task are calculated to generate the corresponding attention weight vector, and then these weight vectors are used to perform weighted fusion on each modal feature to obtain the final multimodal fusion entity node feature; Step 3: For each entity node in the document, find the n nearest entity nodes, calculate the comprehensive distance between the upper left corner, lower right corner and center point of the entity node text box, and obtain the indexes of the nearest n entity nodes to generate the edge adjacency matrix; Step 4: First, use the method of calculating attention weights based on node feature distance. For the entity node features obtained in step 2 and the edge adjacency matrix in step 3, in the first layer of the GraphSAGE graph network model, use weighted sum operation to aggregate neighbor node features and calculate the aggregated neighbor features in combination with attention weights. Then, concatenate the node's own features with the aggregated neighbor features, complete feature update through weight matrix and activation function to obtain the node features of the first layer. In the learning process, in the attention correction layer, the corrected attention weights are calculated and normalized by integrating node degree information and intermediate attention coefficients based on node feature distance, so as to optimize edge weights and enhance feature representation. Then, the second layer of GraphSAGE convolution operation is performed to aggregate neighbor features in a weighted sum manner and update node features. Step 5: Input the features after two layers of GraphSAGE convolution and attention correction into the fully connected layer for feature fusion, and implement the classification of entity nodes through the linear layer; Step 6: Perform graph sampling on the original document to obtain entity node pairs. The edge features are composed of the relationship features between entity nodes, the reading order features of entity nodes, and the polar coordinate features. The entity node features learned by the graph neural network in step 4 are combined to obtain the final edge features. Step 7: Input the final edge features in step 6 into the fully connected layer for edge classification.

2. The method for extracting visual document information based on attention-corrected graph neural network according to claim 1, characterized in that: The step 1 specifically includes: For the input visual document, use OCR technology to extract the entity text Text in the document and its text box coordinates position = [x1, y1, x2, y2], where the above coordinates represent the coordinates of the upper left corner and lower right corner of the text box respectively, and obtain the width W and height H of the document node as the data basis and basis for subsequent operations; Use the resnet50 backbone network and RoI alignment layer to extract the image feature V of document entity node i image , which follows the following formula: V image =Linear(Pool align (Region i ,ResNet50(I))),0≤i≤n Linear represents the linear projection layer, Pool represents the RoIAlign pooling operation, and Region i represents the area of ​​the entity node, and I represents the original document image input to the backbone network; Get the entity node text feature V of the document through the spacy model text As shown in the following formula: V text =Linear(spaCy(t i )),0≤i≤n Linear is a linear projection layer, spaCy represents the text feature extraction model, and t i Represents the text content of the entity; defines the document entity node layout feature V according to the text box information layout , as shown in the following formula: V layout =Linear([x1,y1,x2,y2,C x ,C y ,B w ,B h ]),0≤i≤n Where Linear is a linear projection layer, x1, y1, x2, y2, C x ,C y ,B w ,B h They respectively represent the upper left corner coordinates, lower right corner coordinates, center coordinates, width and height of the entity node text box.

3. The method for extracting visual document information based on attention-corrected graph neural network according to claim 2, characterized in that: The multimodal feature fusion method in step 2 specifically includes: The acquired entity node image features, entity node text features, and entity node layout features are used as inputs in sequence. The feature definitions are as follows: Where n represents the number of entity nodes, d image ,d text ,d layout Represent the dimensions of the three modal features respectively, Calculate the attention weights and map vectors of different dimensions to a common semantic space through a linear projection layer. The expression is as follows: The mapped vectors are concatenated and passed through a fully connected layer with a weight matrix W, and then normalized using the softmax function to obtain the attention weight w attn , the expression is as follows: w attn =softmax(V concat W) [In image ,In text ,In layout ]=W attn The calculated attention weight vector is used to perform weighted fusion on the original layout features, image features, and text features to obtain the fused entity node features. The expression is as follows: Entity fusion-feat =w image V image ′+w text V text ′+w layout V layout 。 4. The method for extracting visual document information based on attention-corrected graph neural network according to claim 3, characterized in that: The step three specifically includes: According to the coordinates of the upper left corner, lower right corner and center point of the text box of the entity node in the document obtained in step 1, the distance between entities is defined based on the squared Euclidean distance. The expression is as follows: Dis ij =AVG(DL ij +DR ij +DC ij )0≤i≤n,0≤j≤n,i!=j Among them DL ij ,DR ij 、DC ij Respectively represent the distances between entities at the upper left corner, lower right corner, and center point; According to the above formula, one third of the total number of entity nodes in the document is selected to generate the edge adjacency matrix of the entity node. The adjacency matrix is ​​a two-dimensional matrix that stores the associated entity node indices.

5. The method for extracting visual document information based on attention-corrected graph neural network according to claim 4, characterized in that: The attention-corrected GraphSAGE model in step 4 specifically includes: Let G = (V, E) be a graph, where V is the set of entity nodes and E is the set of edges. For a node v∈V, its feature vector is x v , the set of neighbor nodes is N(v); For an edge (u,v)∈E, where u∈N(v), the attention weight w is calculated based on the node feature distance vu , use Euclidean distance to measure node feature distance, and the calculation expression is: Get the entity node feature Entity in step 2 fusion-feat , the adjacency matrix Edge_Index in step 3, input the entity node features and the adjacency matrix into the first layer of GraphSAGE convolutional layer, and consider the attention weight when aggregating neighbor node features The calculation formula is: The purpose of SAGE is to update the entity node through the adjacent nodes according to the adjacency matrix. The following is the feature transfer method of SAGE model node v: The CONCAT operation combines the node's own features with the aggregated neighbor features. Splice it together; In the graph-based attention correction layer, the attention weight w calculated for the edge (u,v) in the first layer is vu To make corrections, first calculate the node degree related information, the expression is as follows: where d v is the degree of node v in graph G, where d u is the degree of node u in graph G; Calculate the degree-dependent normalization factor The expression is as follows: Calculate the corrected attention weights And use the softmax function for normalization, the expression is as follows: The output of the first layer Input to the second GraphSAGE convolutional layer, similar to the first layer, first according to the corrected edge weights Aggregate neighbor features and then update node features The expression is as follows: The final feature representation of the entity node can be obtained through the above modified GraphSAGE model layer: Entity Final-feat =GraphSAGE_Revised(TB,Entity fusion-feat ) Where TB is the text box coordinates.

6. The method for extracting visual document information based on attention-corrected graph neural network according to claim 1, characterized in that: The step five specifically includes: For the node classification task, the entity node feature Entity Final-feat Input to the fully connected layer to fuse features and make With the linear layer W h*c Classify the entity nodes, where H represents the hidden layer and c represents the number of categories of the entity nodes.

7. The method for extracting visual document information based on attention-corrected graph neural network according to claim 1, characterized in that: The image sampling method in step 6 specifically includes: For each entity text box, operations are performed in the four directions of up, down, left, and right: first determine the text box with the nearest neighbor position and build a connection, then determine the text box with the second nearest neighbor position to complete the connection construction, and finally form an index list edge_index_list that represents these connection relationships, accurately determining the qualified association situations between entity nodes; The method to define edge features is: The edge features are defined based on the following basic features and the edge_index_list index list of the connection relationship: For the relationship feature F between entity nodes relation : According to the relationship suggestion, assuming that two nodes have a potential relationship, define a primary node S and another secondary node O, and the relationship between the two is represented by R, so: where x S ,y S 、w S 、h S Indicates the center point coordinates, width and height of the main node, x O ,y O 、w O 、h O Indicates the center point coordinates, width and height of the secondary node; Get the minimum bounding box of S and O as R, which is defined as follows: The relationship feature definition between entity nodes can be: For entity nodes, the natural reading order encoding F rope :First, for each entity node i, set its adjacent entity node to j, and use the index difference between them as the sequence length of the entity node pair (i, j), give the adjacent entity nodes a new and unique reading order code, and build a sequence length system for entity node pairs. According to the constructed reading order index system, for the reading order index of each entity node pair, substitute it into the sine encoding matrix and the cosine encoding matrix to obtain the encoding information, and perform encoding operations on the reading order index comprehensively and meticulously. The expression is as follows: F rope =RoPE_Emb(pos) For the polar coordinate feature F polar : To obtain the distance and angle of each entity node pair generated by graph sampling, encode the distance and angle pair sets through one-hot encoding. The specific formula is as follows: F polar =one_hot(distances,angles,num_polar_bins) Where distances and angles are the distance set and angle set of entity node pairs respectively, and num_polar_bins represents the encoded dimension. For entity node pair features: Get the index list edge_index_list of the connection relationship as the entity node pair, and get the entity node feature Entity that integrates the three modalities of image, text, and layout in step 4 Final-feat , get the entity node pair feature F pair , the specific formula is as follows: Entity i , Entity j Represents the node features of entity node i and entity node j in an entity node pair; By combining the above four features, we can get the final edge feature F edge , the expression is as follows:

8. The method for extracting visual document information based on attention-corrected graph neural network according to claim 1, characterized in that: The step seven specifically includes: First obtain the edge feature F of step six edge , represents the key data representation of node association, inputs the fully connected layer, completes feature fusion, and uses a specific weight matrix W h*2 The linear layer infers the existence of the node edges based on the weight parameters and activation function, and finally determines the interconnected nodes that meet the connection rules and threshold judgment as the final layout instances.

Images