A document reading order correction and optimization system based on a graph neural network

Abstract

The application discloses a document reading order correction and optimization system based on a graph neural network, relates to the technical field of artificial intelligence, and comprises a document preprocessing unit, which is used for preprocessing and aligning each page of input documents to generate initial multi-modal feature vectors; a graph structure construction unit, which is used for calculating the distance between each extracted node by using a Euclidean distance formula and constructing a graph structure; a reading order prediction unit, which sequentially performs feature decomposition and splicing fusion on the initial multi-modal feature vectors of each node by using a preconfigured graph neural network model; and a reading order correction unit, which is used for performing global relationship reasoning by using a preconfigured deep learning model, predicting the direct successor relationship between final node embeddings, performing global sequence optimization, and generating an optimized correct text order. The application models document elements and their relationships into a graph, so that the model can learn long-distance dependencies and non-local relationships between elements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a document reading order correction and optimization system based on a graph neural network. BACKGROUND

[0002] In the prior art, after electronic documents, especially scanned PDFs or complex layouts such as multi-column, picture-text mixed layout, table of contents, footnotes, etc., are subjected to OCR or text analysis, a sequence of text blocks that are physically continuous but logically chaotic is often generated. This inconsistency between physical order and logical reading order seriously affects the automatic processing, information extraction, intelligent retrieval, and accessible reading of documents, such as screen readers.

[0003] Traditional reading order correction methods usually rely on heuristic rules, such as coordinates, font size, line spacing, etc., or simple machine learning models. These methods may be effective when dealing with regular layout documents, but for complex and variable document layouts, such as documents containing cross-column titles, floating pictures, complex tables, and irregular text flows, their robustness and accuracy are greatly compromised. In addition, they are difficult to capture the complex semantic and spatial relationships between document elements, resulting in unsatisfactory correction results. For example, traditional text analysis tools may simply concatenate the left and right column texts in a two-column document from top to bottom, rather than following the logical order of reading the left column first and then the right column. Therefore, how to efficiently and accurately restore the logical reading order of a document is a key challenge in the current field of intelligent document processing.

[0004] To address the problems in the related art, no effective solutions have been proposed so far. SUMMARY

[0005] To address the problems in the related art, the present application proposes a document reading order correction and optimization system based on a graph neural network to overcome the above technical problems existing in the prior art.

[0006] To this end, the specific technical solutions adopted by the present application are as follows:

[0007] The present application provides a document reading order correction and optimization system based on a graph neural network, comprising:

[0008] A document preprocessing unit for preprocessing and aligning each page of input document to generate an initial multi-modal feature vector;

[0009] A graph structure construction unit for calculating the distance between each extracted node using the Euclidean distance formula and establishing edges through a nearest neighbor algorithm, and constructing a graph structure based on the spatial relationship and geometric features between the encoded nodes;

[0010] The reading order prediction unit is used to perform feature decomposition and concatenation fusion of the initial multimodal feature vectors of each node in sequence based on the graph structure and using a pre-configured graph neural network model to obtain the concatenated fusion vector; the fusion vector is transformed by a multilayer perceptron to generate the final node embedding for reading order prediction.

[0011] The reading order correction unit is used to perform global relation reasoning using a pre-configured deep learning model to predict the direct successor relationship between the final node embeddings; it performs global sequence optimization on the direct successor relationship through a list-level optimization loss function to generate the optimized correct text order.

[0012] Furthermore, the document preprocessing unit includes:

[0013] The document extraction module is used to extract all text words and bounding box coordinates on the page using a text recognition algorithm, and to segment the page image itself into fixed-size, non-overlapping image blocks.

[0014] The feature concatenation module is used to concatenate text word sequences and image patch sequences using a multimodal feature extraction model to generate an initial multimodal feature vector.

[0015] Furthermore, the step of concatenating the text word sequence and the image patch sequence using a multimodal feature extraction model to generate an initial multimodal feature vector includes:

[0016] Text lexical units are initialized using the word embedding matrix of a pre-trained language model, and text embedding vectors are generated by combining one-dimensional sequence embedding with normalized two-dimensional spatial layout embedding.

[0017] The image patch pixel values ​​are mapped to a high-dimensional space through a linear projection mechanism, and a one-dimensional position embedding is added to generate an image embedding vector.

[0018] The text embedding vector and the image embedding vector are concatenated into a unified vector and then input into a single encoder. Cross-modal fusion processing using masked language model, masked image model and word block alignment mechanism is used to finally generate an initial feature vector that deeply fuses multimodal information.

[0019] Furthermore, the graph structure building unit includes:

[0020] The feature transformation module is used to identify each logical block on the page using a feature extractor and transform it into the initial features of the node. The initial features of the node are then transformed into the high-dimensional feature vector of the node through a multimodal model.

[0021] The edge building module is used to calculate the distance between the center points of the high-dimensional feature vectors of nodes using the Euclidean distance formula, and to build edges using the nearest neighbor algorithm to form directed or undirected edges.

[0022] The graph construction module is used to assign feature vectors to directed or undirected edges based on the relative spatial relationships and geometric features between encoded nodes, and finally construct the graph structure.

[0023] Furthermore, the reading order prediction unit includes:

[0024] The feature decomposition module is used to decompose the initial multimodal feature vector of each node using an inductive graph neural network model to obtain semantic branches and layout branches.

[0025] The semantic layout update module is used to construct two independent inductive graph neural network models that share the same graph structure. The semantic branch and the layout branch are respectively processed by message passing and aggregation to obtain the updated semantic representation and layout representation.

[0026] The node splicing module is used to splice and merge the updated semantic representation and layout representation to obtain the spliced ​​node representation;

[0027] The node transformation module is used to perform nonlinear transformations on the concatenated node representations through a fully connected layer to generate the final node embeddings for reading order prediction.

[0028] Furthermore, the reading order correction unit includes:

[0029] The heterogeneous document graph structure generation module is used to divide document elements into different types of nodes based on layout analysis results, and to establish connection edges between nodes by defining spatial, logical and semantic relationship types to generate a heterogeneous document graph structure.

[0030] The Directed Acyclic Graph (DAG) module is used to redefine task objectives based on direct successor relationships, input heterogeneous document graphs into deep learning models, and generate DAGs.

[0031] The node representation generation module is used to leverage a pre-configured deep learning model to deeply understand the complex relationships in a document and generate information-rich node representations.

[0032] The text optimization module is used to optimize the deep learning model using a list-level maximum likelihood estimation loss function. It generates the correct text order by maximizing the log-likelihood of the predicted sequence to produce the true sorted list.

[0033] Furthermore, the step of redefining the task objective based on direct successor relationships, inputting heterogeneous document graphs into the deep learning model, and generating a directed acyclic graph includes:

[0034] The task objective is redefined based on direct successor relationships, and the reading order modeling is transformed from generating a single linear sequence to predicting the ordering relationships between document elements.

[0035] Heterogeneous document graphs are input into a deep learning model, and a relation representation vector is generated through self-attention mechanism.

[0036] The relation classification network is used to classify the constructed relation representation vectors, and finally the input is fed into the neural network activation function to calculate the probability and generate probability values;

[0037] Based on a preset threshold, determine whether the constructed relation representation vector has a direct successor relationship, and construct a directed acyclic graph.

[0038] Furthermore, the step of determining whether a direct successor relationship exists in the constructed relation representation vector based on a preset threshold, and constructing a directed acyclic graph, includes:

[0039] When the probability value is within a preset threshold, the confidence level between the two predicted nodes is calculated.

[0040] When the probability value is greater than the preset threshold, it is determined that there is a directed edge that is a direct successor of the node.

[0041] Furthermore, the step of using a pre-configured deep learning model to deeply understand the complex relationships in a document and generate information-rich node representations includes:

[0042] By inputting heterogeneous document graph node features, a self-attention mechanism is used to enable each node to interact with global information.

[0043] By injecting learnable biases consisting of inter-node structural paths and node types, attention weights that integrate content, structure, and type information are generated and transformed into node representations.

[0044] Furthermore, the optimization of the deep learning model using a list-level maximum likelihood estimation loss function, and the generation of the optimized correct text order by maximizing the log-likelihood of the predicted sequence to generate the true ordered list, includes:

[0045] Based on the probabilistic prediction model, the document ranking process is regarded as a sequential selection process, in which ranking items are selected sequentially from the candidate items, and the predicted score of each document element is calculated.

[0046] By mapping these scores to the probability domain using the natural logarithm and exponential function, the log probability of each position in the true ranking list is calculated, and the loss value that maximizes the predicted sequence is obtained.

[0047] By summing the results at each position in the sequence using a loss function, the deep learning model is optimized to generate the log-likelihood of the true sorted list, thus generating the optimized correct text order.

[0048] The beneficial effects of this invention are as follows:

[0049] 1) This invention effectively integrates multimodal information such as visual, text, and layout through graph neural networks, which can more accurately capture the complex relationships between document elements and improve the accuracy of restoring the logical reading order of complex layout documents, such as multi-column, mixed text and image layouts, and scanned documents.

[0050] 2) This invention can effectively address issues such as different fonts, font sizes, layout styles, and image interference, and has stronger adaptability and robustness to document quality and complexity.

[0051] 3) This invention models document elements and their relationships as a graph, enabling the model to learn long-distance dependencies and non-local relationships between elements, rather than relying solely on the simple proximity principle.

[0052] 4) This invention achieves fully automated processing from document input to logical reading sequence output, greatly reducing manual intervention and improving document processing efficiency.

[0053] 5) This invention provides an accurate logical reading order, providing high-quality input for downstream applications such as information extraction, intelligent retrieval, knowledge graph construction, document summarization, speech synthesis, and accessible reading, significantly improving the performance and user experience of these applications.

[0054] 6) This invention utilizes the inherent scalability of the graph neural network framework, which can easily incorporate features of more modalities or more complex node relationships. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a structural block diagram of a document reading order correction and optimization system based on a graph neural network according to an embodiment of the present invention.

[0057] In the picture:

[0058] 1. Document preprocessing unit; 2. Graph structure construction unit; 3. Reading order prediction unit; 4. Reading order correction unit. Detailed Implementation

[0059] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention.

[0060] According to an embodiment of the present invention, a document reading order correction and optimization system based on graph neural networks is provided.

[0061] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, a document reading order correction and optimization system based on a graph neural network according to an embodiment of the present invention includes:

[0062] Document preprocessing unit 1 is used to preprocess and align each page of the input document to generate an initial multimodal feature vector;

[0063] Graph structure building unit 2 is used to calculate the distance between each extracted node using the Euclidean distance formula, and to establish edges through the nearest neighbor algorithm. Based on the spatial relationship and geometric features (i.e. size and overlap) between encoded nodes, a graph structure is constructed.

[0064] Reading order prediction unit 3 is used to perform feature decomposition and concatenation fusion of the initial multimodal feature vectors of each node in sequence based on the graph structure and using a pre-configured graph neural network model to obtain the concatenated fusion vector; the fusion vector is transformed by a multilayer perceptron to generate the final node embedding for reading order prediction.

[0065] Reading order correction unit 4 is used to perform global relation reasoning using a pre-configured deep learning (i.e., graph Transformer) model to predict the direct successor relationship (i.e., ISDR) between the final node embeddings; and to perform global sequence optimization of the direct successor relationship through a list-level optimized loss function (i.e., ListMLE) to generate the optimized correct text order.

[0066] In this optional embodiment, the document preprocessing unit includes:

[0067] The document extraction module is used to extract all text words and bounding box coordinates on the page using a text recognition algorithm (i.e., an OCR tool), and to segment the page image itself into fixed-size, non-overlapping image blocks;

[0068] The feature concatenation module is used to concatenate text word sequences and image patch sequences using a multimodal feature extraction model to generate an initial multimodal feature vector.

[0069] Specifically, for each page of the input document, two processes are performed in parallel: 1. Text and layout extraction: using a high-precision OCR tool (paddleocr) to extract all text tokens on the page and their precise bounding box coordinates. 2. Image patching: dividing the page image itself (e.g., resized to 224×224 pixels) into a series of fixed-size, non-overlapping image patches, 16×16 pixels each. Unified embedding layer: The core of the multimodal feature extraction model lies in its input layer, which concatenates the text token sequence and the image patch sequence into a single input sequence, fed into a unified Transformer encoder. This multimodal feature extraction model aims to generate a context-rich feature vector, deeply fused with multimodal information, for each logical unit (such as a text block) in the document, serving as the initial features for nodes in the subsequent graph neural network. In this way, it replaces the traditional method of relying on a large CNN backbone network for image feature extraction, reducing the number of parameters in the multimodal feature extraction model.

[0070] In this optional embodiment, the step of concatenating the text word sequence and the image patch sequence using a multimodal feature extraction model to generate an initial multimodal feature vector includes:

[0071] Text lexical units are initialized using the word embedding matrix of a pre-trained language model, and text embedding vectors are generated by combining one-dimensional sequence embedding with normalized two-dimensional spatial layout embedding.

[0072] The image patch pixel values ​​are mapped to a high-dimensional space through a linear projection mechanism, and a one-dimensional position embedding is added to generate an image embedding vector.

[0073] The text embedding vector and the image embedding vector are concatenated into a unified vector and then input into a single encoder (i.e., the Transformer encoder). Cross-modal fusion processing using masked language model, masked image model and word block alignment mechanism is used to finally generate an initial feature vector that deeply fuses multimodal information.

[0074] Specifically, Text Embedding: The final embedding vector for each text term consists of three parts: Word Embedding: Initialized using a word embedding matrix from a powerful pre-trained language model (RoBERT). One-Dimensional Position Embedding (1DPositionEmbedding): A standard absolute position embedding representing the index position of the term in the text sequence. Two-Dimensional Layout Embedding (2DLayoutEmbedding): The bounding box coordinates (x0, y0, x1, y1, width, height) of each term are normalized (e.g., scaled to the range of 0-1000) and then transformed into a high-dimensional vector through a learnable embedding layer. This allows the pre-trained language model to understand the spatial location information of the text. Image Embedding: The embedding vector for each image patch consists of two parts: Linear Projection: The flattened image patch pixel values ​​are mapped to the same dimension as the text embedding through a simple fully connected layer (linear projection). One-Dimensional Position Embedding: A learnable one-dimensional position embedding is added to each image patch, representing its position in the image patch sequence.

[0075] Pre-training Objectives: The powerful capabilities of this feature extractor stem from its pre-training on massive amounts of unlabeled document data. The pre-training task forces the model to learn deep alignment relationships between text, layout, and visual information. Key pre-training objectives include: 1. Masked Language Modeling (MLM): Randomly masking a portion of text words, the pre-training task forces the model to predict the masked words based on the context (including unmasked text and all image patches). 2. Masked Image Modeling (MIM): Randomly masking a portion of image patches, the pre-training task forces the model to reconstruct the discrete visual tokens of the masked image patches based on the context (including all text and unmasked image patches). 3. Word-Patch Alignment (WPA): For an unmasked text word, the pre-training task forces the model to predict whether its corresponding image region (i.e., the image patch covered by its bounding box) is masked. This task explicitly forces the pre-training task to compel the model to learn cross-modal correspondences between text and vision. Through this module, the system can generate a context-rich feature vector that deeply integrates multimodal information for each logical unit in the document, such as a text block. This vector will serve as the initial features for nodes in the subsequent graph neural network.

[0076] In this optional embodiment, the graph structure building unit includes:

[0077] The feature transformation module is used to identify each logical block on the page using the feature extractor and transform it into the initial features of the node. The initial features of the node are transformed into the high-dimensional feature vector of the node through the multimodal model (i.e. the LayoutLMv3 paradigm module).

[0078] The edge building module is used to calculate the distance between the center points of the high-dimensional feature vectors of nodes using the Euclidean distance formula, and to build edges using the nearest neighbor algorithm to form directed or undirected edges.

[0079] The graph construction module is used to assign feature vectors to directed or undirected edges based on the relative spatial relationships and geometric features between encoded nodes, and finally construct the graph structure.

[0080] Specifically, node definition: Each logical block identified by the aforementioned feature extractor, such as a paragraph, an image, or a table, becomes a node in the graph. The initial feature of each node is its corresponding high-dimensional feature vector output by the LayoutLMv3 paradigm module. Edge construction strategy: To avoid constructing a computationally intensive and noisy fully connected graph, a k-nearest-neighbor (k-NN) strategy is used to construct the graph's topology. Algorithm: For each node in the graph, calculate its Euclidean distance to the center points of the bounding boxes of all other nodes. Then, connect the node to its k nearest neighbor nodes, forming directed or undirected edges.

[0081] Calculating the Euclidean distance is a fundamental step in constructing edges in a k-NN network. The specific algorithm is as follows:

[0082] Center point coordinate extraction: For any node in the document (such as a text block, image, etc.), obtain its bounding box. coordinate ,in The coordinates of the top left corner The coordinates are the bottom right corner. The coordinates are the center point of this node. The calculation formula is as follows:

[0083]

[0084] Euclidean distance calculation: For any two nodes A and B in the graph, the coordinates of their center points are respectively... and The Euclidean distance between them Calculated using the following formula: .

[0085] k-Nearest Neighbor Selection: For each node A in the graph, calculate its distance to all other nodes B. Sort all calculated distances and select the k nodes with the smallest distances as the neighbors of node A, and establish edges with A. This process is the application of the k-Nearest Neighbor (k-NN) algorithm.

[0086] The initial value of k is set to 4. This choice is based on a core assumption: the logical function and reading order of a layout element are largely determined by its most immediate local environment. k=4 allows for attention to local dependencies while maintaining the connectivity of the entire graph through multi-hop connections, effectively connecting distant nodes. Edge feature definition: Edges in the graph can also carry information. Each edge can be assigned a feature vector to describe the specific relationship between the two nodes it connects. This vector can encode the following information: Relative spatial relationship: the normalized distance and relative angle between the center points of the two connected nodes. Size and overlap: the ratio of the width and height of the bounding boxes of the two nodes, and their intersection-over-union (IoU).

[0087] In this optional embodiment, the reading order prediction unit includes:

[0088] The feature decomposition module is used to decompose the initial multimodal feature vector of each node using the inductive graph neural network model (i.e., GraphSAGE) to obtain semantic branches and layout branches.

[0089] The semantic layout update module is used to construct two independent inductive graph neural network models that share the same graph structure. The semantic branch and the layout branch are respectively processed by message passing and aggregation to obtain the updated semantic representation and layout representation.

[0090] The node splicing module is used to splice and merge the updated semantic representation and layout representation to obtain the spliced ​​node representation;

[0091] The node transformation module is used to perform nonlinear transformations on the concatenated node representations through a fully connected layer to generate the final node embeddings for reading order prediction.

[0092] Specifically, for the initial multimodal feature vector of each node, it is decomposed into two parts: a text / semantic branch (mainly derived from word embeddings) and a visual / layout branch (mainly derived from image patch embeddings and 2D layout embeddings). Two independent GraphSAGE models are constructed, which share the same graph topology (i.e., the k-NN graph constructed above).

[0093] GNN Branch 1 (Semantic): This GNN performs message passing and aggregation only on textual / semantic features. It learns the content-based logical relationships between nodes.

[0094] GNN Branch 2 (Layout): This GNN only performs message passing and aggregation on visual / layout features. It learns the structural relationships between nodes based on spatial arrangement.

[0095] After each of the two GNN branches performs information propagation through multiple layers (2-3 layers), each node receives two output vectors: an updated semantic representation and an updated layout representation. These two vectors are concatenated to form a final node representation that integrates depth information from both modalities. The concatenated fused vector is then input into one or more fully connected layers (Multilayer Perceptron, MLP) for a final nonlinear transformation, yielding the final node embedding used for reading order prediction.

[0096] The design logic of this dual-branch architecture lies in its ability to allow the GraphSAGE model to learn and refine information independently in two different modal spaces before performing high-level fusion. This effectively prevents one modality, such as a strong visual signal, from prematurely overwhelming or contaminating another modality, such as subtle semantic signals, in the early stages of information dissemination, thus achieving more efficient and robust multimodal information fusion.

[0097] In this optional embodiment, the reading order correction unit includes:

[0098] The heterogeneous document graph structure generation module is used to divide document elements into different types of nodes based on layout analysis results, and to establish connection edges between nodes by defining spatial, logical and semantic relationship types to generate a heterogeneous document graph structure.

[0099] The Directed Acyclic Graph (DAG) module is used to redefine task objectives based on the Direct Successor Relation (ISDR), inputting heterogeneous document graphs into a deep learning model to generate a DAG.

[0100] The node representation generation module is used to generate information-rich node representations by leveraging a pre-configured deep learning (i.e., graph Transformer) model to deeply understand the complex relationships in a document.

[0101] The text optimization module is used to optimize the deep learning model using the list-level maximum likelihood estimation (ListMLE) loss function. It generates the optimized correct text order by maximizing the log-likelihood of the predicted sequence to produce the true sorted list.

[0102] Specifically, the document graph defines a formal meta-schema containing various node and edge types. NodeTypes: Based on the preliminary results of layout analysis, each document element is assigned a specific type. These types include, but are not limited to: heading, paragraph, list item, figure, caption, table, footnote, header, and footer.

[0103] Edge Types: Relationships between nodes are also assigned explicit semantic types. This allows subsequent GNN models to learn type-specific message passing functions. Edge types can include:

[0104] Spatial Relations: ('paragraph', 'spatial_adjacency', 'paragraph'): Describes the adjacency relationship between two paragraphs based purely on geometric location. ('any', 'vertical_overlap', 'any'): Describes the vertical overlap between two elements.

[0105] Logical Relations: ('figure', 'caption_of', 'caption'): Describes a figure title as a description of a particular image; this is a strong logical binding. ('heading', 'leads_section', 'paragraph'): Describes a heading that leads to a subsequent paragraph, reflecting the document's hierarchical structure. ('list_item', 'continues_list', 'list_item'): Describes the continuity between two list items within the same list. Semantic Relations: ('paragraph', 'semantic_continuation', 'paragraph'): Describes a high degree of coherence between two paragraphs in terms of textual content (e.g., determined by calculating the cosine similarity of their feature vectors). ('paragraph', 'refers_to', 'footnote'): Describes a reference in the text that points to a footnote.

[0106] This allows GNN models to learn highly contextualized rules. For example, a GNN model can learn that the rules determining the connection between a figure and a caption (primarily logical binding and close proximity) are quite different from the rules determining the connection between two paragraphs (which may rely more on semantic coherence and multi-column layout rules).

[0107] Table 1. Specifications of Heterogeneous Document Schema

[0108]

[0109] The attention score (attention(i,j)) between nodes i and j is determined by several factors: the dot product of the query vector of node i and the key vector of node j, as is the standard Transformer approach; a learnable scalar or vector that depends on the shortest path distance between nodes i and j in the heterogeneous graph and the encoding of the edge type on that path. For example, if two nodes are directly connected by a caption_of edge, they will receive a very strong attention bias, encouraging the GNN model to focus on the relationship between them; and a learnable bias term that depends on the types of nodes i and j. For example, the GNN model can learn that heading nodes naturally have a higher attention to the paragraph nodes that follow them. This structure-aware self-attention mechanism allows the GNN model to learn complex, non-local, context-dependent relationships. For example, the GNN model can easily learn that there is a strong association between a footnote node at the bottom of the page and a reference marker in a paragraph node at the top of the page, even if they are spatially far apart, because they are connected in the heterogeneous graph by a referers_to type path. This fundamentally solves the problem that standard GNNs struggle to capture long-range dependencies.

[0110] In this optional embodiment, redefining the task objective based on direct successor relationships, inputting the heterogeneous document graph into the deep learning model, and generating a directed acyclic graph includes:

[0111] The task objective is redefined based on direct successor relationships (ISDR), transforming reading order modeling from generating a single linear sequence to predicting the ordering relationships between document elements;

[0112] Heterogeneous document graphs are input into a deep learning model, and a relation representation vector is generated through self-attention mechanism.

[0113] The relation representation vector is classified using a relation classification network (i.e., ClassificationHead), and finally input into the neural network activation function (i.e., Sigmoid activation function) for probability calculation to generate probability values;

[0114] Based on a preset threshold, determine whether there is a direct successor relationship between the constructed relation representation vectors, and construct a directed acyclic graph (DAG).

[0115] Specifically, the final output of the deep learning model is no longer a one-dimensional list of element indices, but a Directed Acyclic Graph (DAG). In this graph, a directed edge from node A to node B is precisely defined as follows: in at least one valid reading path, element B is read immediately after element A. Through this definition, the reading order detection task is successfully transformed from a ranking problem into a relation extraction or link prediction problem on a graph. The task of the deep learning model becomes predicting whether an ISDR relationship exists between any pair of elements (A, B) in the document. The specific steps for predicting the existence of an ISDR relationship between any two nodes (A, B) are: obtaining the final node representation: inputting the constructed heterogeneous document graph into the graph Transformer model. After multiple layers of self-attention computation, the deep learning model generates a final embedding vector, denoted as h, for each node in the graph (such as node A and node B), which deeply integrates structural and semantic information. A and h B .

[0116] To predict the directed relationship (A to B) between A and B, their embedding vectors are combined to form a vector h representing that specific relationship. A至B One efficient way to combine components is through concatenation, i.e. Other methods can also be used, such as element difference h. A -h B or element product h A *h B The generated relational representation vector h A至B The input is fed into a simple classification head. This classification head is typically one or more fully connected layers (i.e., a multilayer perceptron, MLP) and ultimately passes through a sigmoid activation function.

[0117] ;

[0118] In the formula, the output of the Sigmoid function is a probability value between 0 and 1; MLP represents a multilayer sensing mechanism. This represents the confidence level of a deep learning model in predicting node B immediately after node A. By setting a threshold (e.g., 0.5), if... If the value exceeds this threshold, an ISDR directed edge from A to B is determined to exist. By repeating the above process for all possible node pairs in the document, a directed acyclic graph (DAG) representing the complete reading flow of the document can be constructed. This relation-based representation has unparalleled expressive power. It can naturally represent that after a paragraph ends, the reading flow can branch into two side-by-side sections. It can represent that a main text column and a sidebar are read in parallel. It can clearly represent the non-local jump relationships between elements such as footnotes and references and the main text.

[0119] In this optional embodiment, the step of determining whether a direct successor relationship exists in the constructed relation representation vector based on a preset threshold and constructing a directed acyclic graph includes:

[0120] When the probability value is within a preset threshold, the confidence level between the two predicted nodes is calculated.

[0121] When the probability value is greater than the preset threshold, it is determined that there is a directed edge that is a direct successor of the node.

[0122] In this optional embodiment, the step of using a pre-configured deep learning model to deeply understand the complex relationships in a document and generate information-rich node representations includes:

[0123] By inputting heterogeneous document graph node features, a self-attention mechanism is used to enable each node to interact with global information.

[0124] By injecting learnable biases consisting of inter-node structural paths and node types, attention weights that integrate content, structure, and type information are generated and transformed into node representations.

[0125] In this optional embodiment, optimizing the deep learning model using a list-level maximum likelihood estimation loss function to generate the optimized correct text order by maximizing the log-likelihood of the predicted sequence to produce the true sorted list includes:

[0126] Based on the probabilistic prediction model (i.e., Plackett-Luce), the document ranking process is regarded as a sequential selection process, which selects ranking items from the candidates in turn and calculates the prediction score for each document element.

[0127] By mapping these scores to the probability domain using the natural logarithm and exponential function, the log probability of each position in the true ranking list is calculated, and the loss value that maximizes the predicted sequence is obtained.

[0128] By summing the results at each position in the sequence using a loss function, the deep learning model is optimized to generate the log-likelihood of the true sorted list, thus generating the optimized correct text order.

[0129] Specifically, the ListMLE loss function is based on the Plackett-Luce model in probability theory, which is a Plackett-Luce model used to describe the probability of ranking preferences. Its core idea is to view the generation of a ranked list as a sequential selection process: from all candidates, the first-ranked item is selected with a certain probability, and then the second-ranked item is selected from the remaining candidates. The goal of ListMLE is to maximize the log-likelihood of the scores predicted by the Plackett-Luce model in generating the ground-truth permutation.

[0130] For a given document, suppose its actual reading order is y=(y1,y2,...,y...). n The Plackett-Luce model is used for each document element x. i The predicted score is s i The ListMLE loss function (i.e., negative log-likelihood) can be expressed as:

[0131] ;

[0132] In the formula, This represents the final calculated ListMLE loss value, y=(y1,y2,...,y n ) represents a sequence of length n, which is the ground-truth reading order of n elements in the document; y i This indicates that the document element is ranked i-th in the actual ranking; s represents the score vector predicted by the Plackett-Luce model for all elements in the document. This represents the score predicted by the Plackett-Luce model for the i-th element in the true ranking. The higher the score, the higher the Plackett-Luce model believes that the element should be ranked. Let represent the predicted score of the i-th element in the actual sorting. The fractional part can be seen as correctly selecting y from all unsorted elements (i.e., from the i-th to the n-th element) at the i-th step. i The probability of the correct choice is usually achieved through a Softmax function, and the whole formula is to sum the log probabilities of the correct choice at each step. This represents summing over each position i (from 1 to n) in the sequence; log represents the natural logarithm function, which is the logarithm with the natural constant e as the base. This represents the exponential function, e(⋅). It is used to map a fraction s from the real number field to the positive number field, facilitating probability calculations. This represents the denominator of the Softmax function, which calculates the sum of the exponents of the predicted scores of all unsorted elements (including the correct element at the current i-th position and all elements at subsequent positions) when the i-th sorting position is determined. Let y represent a softmax probability, indicating that at the i-th sorting step, the Plackett-Luce model can correctly select y from all unsorted elements. i The probability of being the i-th element.

[0133] This loss function has mature implementations in mainstream machine learning frameworks. For example, in the TensorFlow Ranking library, `tfr.keras.losses.ListMLELoss` can be directly called. In PyTorch, it can also be easily implemented based on its mathematical definition, as shown in the open-source project allRank. Providing PyTorch-based pseudocode in the patent specification will greatly enhance the full disclosure of the technical solution. Compared to pairwise loss, ListMLE directly optimizes the probability of generating the entire correct sequence. It naturally imposes a greater penalty on errors in elements at the beginning of the list, because these errors affect the denominator of the probability calculation for all subsequent steps, which is highly consistent with the sensitivity of humans to the order of opening sentences when reading. Therefore, using ListMLE can more effectively train a Plackett-Luce model that can generate the globally optimal reading path. These two techniques form a perfect closed loop with the relationship prediction paradigm proposed in Part III. The Graph Transformer is responsible for deeply understanding the internal relational structure of a document and generating information-rich node representations. The Enhanced Pointer Network acts as an intelligent navigator, planning the optimal reading path on the map formed by these node representations. The ListMLE loss function serves as a compass guiding the navigator's learning, ensuring that the planned path is as close as possible to human reading habits.

[0134] Partial code screenshots, detailed description of the heterogeneous graph construction module:

[0135] The graph construction module of this invention receives a set of document elements with initial multimodal features. The module first assigns a node type to each element based on its intrinsic attributes (such as text content, size, presence in a specific region, etc.) and / or the output of a lightweight classifier, such as "paragraph," "heading," "image," etc. The module iterates through all node pairs and establishes typed edges between nodes according to a predefined set of rules containing geometric and logical judgments. For example, for any two nodes, if the projected distance of their bounding boxes in the vertical or horizontal direction is less than a preset threshold dspatial, then an edge of the "spatial proximity" type is established between them. If an "image" node A is immediately below a "caption" node B, then an edge of the "image description" type is established from A to B. If the cosine similarity of the content feature vectors of two "paragraph" nodes is higher than the threshold ssemantic, then an edge of the "semantic continuity" type is established. Through this process, a heterogeneous graph capable of finely characterizing the document structure and semantics is constructed. Example of heterogeneous graph construction pseudocode (based on the PyTorchGeometric library representation):

[0136] "fromtorch_geometric.dataimportHeteroData

[0137] defconstruct_heterogeneous_graph(document_elements):

[0138] """

[0139] Construct a heterogeneous graph based on the list of document elements.

[0140] Args:

[0141] document_elements: A list containing all document elements (and their characteristics and types).

[0142] Returns: A PyG HeteroData object.

[0143] """

[0144] graph=HeteroData()

[0145] #1. Initialize the node feature tensor according to the element type

[0146] fornode_typein['paragraph','heading','figure','caption','table']:

[0147] #Filter out elements of this type and extract their features

[0148] elements_of_type=[elemforelemindocument_elementsifelem.type==node_type]

[0149] ifelements_of_type:

[0150] graph[node_type].x=torch.tensor([elem.featureforeleminelements_of_type])

[0151] graph[node_type].node_ids=[elem.idforeleminelements_of_type]

[0152] #2. Create edges with different types according to predefined rules.

[0153] #Example: Create a 'spatial adjacency' edge ('paragraph', 'spatial_adjacency', 'paragraph')

[0154] para_to_para_edges=

[0155] fori,elem_iinenumerate(document_elements):

[0156] ifelem_i.type!='paragraph':continue

[0157] forj,elem_jinenumerate(document_elements):

[0158] ifi==jorelem_j.type!='paragraph':continue

[0159] ifis_spatially_adjacent(elem_i,elem_j):

[0160] #Assuming the global ID has already been mapped to an index within the type.

[0161] src_idx=map_global_id_to_type_idx(elem_i.id,'paragraph')

[0162] dst_idx = map_global_id_to_type_idx(elem_j.id, 'paragraph')

[0163] para_to_para_edges.append([src_idx, dst_idx])

[0164] if para_to_para_edges:

[0165] graph['paragraph','spatial_adjacency', 'paragraph'].edge_index = torch.tensor(para_to_para_edges).t().contiguous()

[0166] # Example: Establish the 'figure, caption_of, caption' edge

[0167] fig_to_caption_edges =

[0168] for fig_elem in get_elements_by_type(document_elements, 'figure'):

[0169] for cap_elem in get_elements_by_type(document_elements, 'caption'):

[0170] if is_caption_of(cap_elem, fig_elem):

[0171] src_idx = map_global_id_to_type_idx(fig_elem.id, 'figure')

[0172] dst_idx = map_global_id_to_type_idx(cap_elem.id, 'caption')

[0173] fig_to_caption_edges.append([src_idx, dst_idx])

[0174] if fig_to_caption_edges:

[0175] graph['figure','caption_of','caption'].edge_index=torch.tensor(fig_to_caption_edges).t().contiguous()

[0176] returngraph.

[0177] In summary, by utilizing the above-mentioned technical solution of the present invention, the present invention solves the problems of low accuracy, poor robustness, and difficulty in capturing complex logical relationships in existing document reading order correction methods when processing complex format documents. This enables efficient, accurate, and intelligent reading order correction and optimization of electronic documents such as PDFs, thereby improving the automated processing and understanding capabilities of documents.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A document reading order correction and optimization system based on graph neural network, characterized in that, The method comprises the following steps: a document preprocessing unit for preprocessing and aligning each page of the input document to generate an initial multi-modal feature vector; a graph structure construction unit for calculating the distance between each extracted node using the Euclidean distance formula and establishing edges through a nearest neighbor algorithm to construct a graph structure based on the spatial relationship and geometric features between the encoded nodes; a reading order prediction unit for sequentially performing feature decomposition and concatenation fusion on the initial multi-modal feature vector of each node based on the graph structure using a pre-configured graph neural network model to obtain a concatenated fusion vector; and generating a final node embedding for reading order prediction by transforming the fusion vector through a multi-layer perceptron; a reading order correction unit for performing global relationship reasoning using a pre-configured deep learning model to predict the direct successor relationship between the final node embeddings; global sequence optimization of the direct successor relationship through a list-level optimization loss function to generate an optimized correct text order; comprising: dividing document elements into different types of nodes through layout analysis results and establishing connection edges between nodes by defining spatial, logical, and semantic relationship types to generate a heterogeneous document graph structure; redefining the task target based on the direct successor relationship and inputting the heterogeneous document graph into a deep learning model to generate a directed acyclic graph; deeply understanding the complex relationships of the document using a pre-configured deep learning model to generate an information-rich node representation; optimizing the deep learning model using a list-level maximum likelihood estimation loss function by maximizing the log-likelihood of the predicted sequence to generate the true ranking list to generate an optimized correct text order.

2. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The document preprocessing unit comprises: a document extraction module for extracting all text tokens and bounding box coordinates on the page using an optical character recognition algorithm and segmenting the page image itself into fixed-size, non-overlapping image blocks; a feature concatenation module for concatenating the text token sequence and image block sequence using a multi-modal feature extraction model to generate an initial multi-modal feature vector.

3. The document reading order correction and optimization system based on graph neural network according to claim 2, characterized in that, The method of concatenating the text token sequence and image block sequence using a multi-modal feature extraction model to generate an initial multi-modal feature vector comprises: initializing text tokens through the word embedding matrix of a pre-trained language model and combining one-dimensional sequence embedding and normalized two-dimensional spatial layout embedding to generate text embedding vectors; mapping image block pixel values to a high-dimensional space through a linear projection mechanism and adding one-dimensional position embedding to generate image embedding vectors; concatenating the text embedding vectors and image embedding vectors into a unified vector and inputting it into a single encoder to perform cross-modal fusion processing using a masked language model and a masked image model and a word block alignment mechanism to finally generate an initial feature vector that deeply fuses multi-modal information.

4. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The graph structure construction unit comprises: a feature conversion module for identifying each logical block on the page using a feature extractor and converting it into the initial features of the nodes, and converting the initial features of the nodes into high-dimensional feature vectors of the nodes through a multi-modal model; The edge establishing module is configured to calculate distances between high-dimensional feature vector center points of nodes by using a Euclidean distance formula, and establish edges by using a nearest neighbor algorithm to form directed or undirected edges; The graph constructing module is configured to assign feature vectors to the directed or undirected edges based on relative spatial relationships and geometric features between the coded nodes, and finally construct a graph structure.

5. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The reading order prediction unit comprises: The feature decomposition module is configured to decompose the initial multi-modal feature vector of each node by using an inductive graph neural network model to obtain a semantic branch and a layout branch; The semantic layout updating module is configured to construct two independent inductive graph neural network models that share the same graph structure, and perform message passing and aggregation processing on the semantic branch and the layout branch, respectively, to obtain updated semantic representations and layout representations; The node splicing module is configured to splice and fuse the updated semantic representations and layout representations to obtain spliced node representations; The node transformation module is configured to perform non-linear transformation on the spliced node representations by using a fully connected layer to generate final node embeddings for reading order prediction.

6. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The task target is redefined based on the direct successor relationship, the heterogeneous document graph is input into a deep learning model, and a directed acyclic graph is generated, which comprises: The reading order modeling is converted from generating a single linear sequence to predicting the ordering relationship between document elements by redefining the task target based on the direct successor relationship; The heterogeneous document graph is input into a deep learning model, and a relationship representation vector is generated by calculating through a self-attention mechanism; The relationship classification network is used to classify the relationship representation vector, and finally input into a neural network activation function for probability calculation to generate a probability value; The preset threshold is used to determine whether the relationship representation vector has a direct successor relationship, and a directed acyclic graph is constructed.

7. The document reading order correction and optimization system based on graph neural network according to claim 6, characterized in that, The preset threshold is used to determine whether the relationship representation vector has a direct successor relationship, and a directed acyclic graph is constructed, which comprises: When the probability value is between the preset threshold, the confidence between the two nodes is calculated; When the probability value is greater than the preset threshold, it is determined that there is a direct successor relationship of the node directed edge.

8. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The pre-configured deep learning model is used to deeply understand the complex relationship of the document, and the information-rich node representation is generated, which comprises: By inputting the node features of the heterogeneous document graph, each node is made to interact with global information by using a self-attention mechanism; By injecting a learnable bias composed of node structure paths and node types, attention weights that fuse content, structure and type information are generated and converted into node representations.

9. The document reading order correction and optimization system based on graph neural network according to claim 1, characterized in that, The deep learning model is optimized by using the list-level maximum likelihood estimation loss function, the log likelihood of the predicted sequence generating the real ranking list is maximized, and the optimized correct text order is generated, which comprises: Based on the probability prediction model, the ranking process of the document is regarded as a sequential selection process, and the ranking items are selected from the candidate items one by one, and the prediction scores of each document element are calculated; By using a natural logarithm and an exponential function, the scores are mapped to a probability domain, the log probability of each position in the real ranking list is calculated, and the loss value of the maximum predicted sequence is obtained; The deep learning model is optimized by summing up each position in the sequence using a loss function to generate a log-likelihood of the true ranking list, generating an optimized correct text order.

Images