Corpus processing method and apparatus, storage medium, and program product

Abstract

This application discloses a corpus processing method, apparatus, storage medium, and program product, relating to the field of artificial intelligence technology. The method includes: identifying a file to be identified to obtain identification results, and performing clustering processing on the identification results to determine an aggregation result; constructing a full node graph based on the aggregation result, and determining a tree structure graph corresponding to the aggregation result according to the semantic relationships between nodes in the full node graph; classifying and identifying nodes in the tree structure graph, determining tree structure data, and constructing question-answer pairs to represent the semantic relationships corresponding to the file to be identified. The solution of this application solves the problems of neglecting tree graph parsing, difficulty in identifying text structure associations in OCR, insufficient feature utilization, and low extraction accuracy in existing corpus processing solutions.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a corpus processing method, apparatus, storage medium, and program product. Background Technology

[0002] In the current large-scale model training process, high-quality corpora are a key prerequisite for improving the model's core performance, such as understanding and generation accuracy. This is especially true when processing mixed text and image content in industry technical documents. Accurately extracting the structured information from these combined text and images has become a core challenge in corpus annotation tasks. Currently, most document corpus processing solutions focus on extracting and analyzing pure text content, paying very little attention to parsing the image information contained within documents. However, in actual industry documents, tree diagrams or tree-like distribution diagrams are a common form of information presentation. These diagrams contain a large amount of structured information and clear semantic relationships. This information is crucial for improving the structured understanding of industry knowledge by large models. However, existing corpus processing workflows generally ignore the effective information in these images, leading to information gaps in the training corpora for large models. Meanwhile, although Optical Character Recognition (OCR) technology has made progress in extracting pure text, it cannot accurately identify the text-structural relationship features when parsing tree diagrams. Existing corpus extraction methods either have incomplete information or suffer from low accuracy, poor stability, and low output efficiency in structured image recognition. When extracting text content from images, OCR cannot handle the text relationships in tree diagrams, and the effect of merging text blocks based on position rules is not good. Methods for extracting tree diagram structural relationships from OCR recognition results also have problems such as insufficient feature utilization and insufficient model prediction ability. Summary of the Invention

[0003] At least one embodiment of this application provides a corpus processing method, apparatus, storage medium, and program product to address the problems of neglecting tree diagram parsing, difficulty in identifying text structure relationships in OCR, insufficient feature utilization, and low extraction accuracy in existing corpus processing solutions.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a corpus processing method, including:

[0006] The document to be identified is identified to obtain the identification result, and the identification result is clustered to determine the aggregation result;

[0007] A full node graph is constructed based on the aggregation result, and a tree structure graph corresponding to the aggregation result is determined according to the semantic relationships between the nodes in the full node graph.

[0008] The nodes in the tree structure graph are classified and identified to determine the tree structure data and construct question-answer pairs to represent the semantic relationships of the files to be identified.

[0009] Optionally, the document to be identified is identified to obtain identification results, and the identification results are clustered to determine the aggregation results, including:

[0010] The acquired file to be identified is preprocessed to obtain preprocessed data; the preprocessing includes at least one of orientation correction, grayscale processing, and size adjustment;

[0011] Optical character recognition is performed on the preprocessed data to obtain recognition results with multiple lines of text information;

[0012] The position information in the text line information is standardized, and the coordinates of the midpoints of the left and right borders and the rotation angle of the borders are calculated. The coordinates of the vertex of the border, the coordinates of the midpoints of the left and right borders, and the rotation angle of the borders are used as position features.

[0013] A pre-trained language model is used to preprocess the text content in the text line information to obtain text semantic features;

[0014] The location features and the text semantic features are fused to obtain the fused features;

[0015] Based on the fusion features, determine the neighborhood map features;

[0016] Based on the features of the neighbor graph, a graph convolutional network model is constructed;

[0017] The recognition results are input into the graph convolutional network model to determine the aggregation result.

[0018] Optionally, based on the fusion features, the neighborhood map features are determined, including:

[0019] Each fused feature is treated as a node, and the total number of text lines corresponding to the fused feature is taken as the total number of nodes. The local density of each node is calculated. The local density is obtained by Euclidean distance between nodes and preset hyperparameters.

[0020] Based on the local density of each node, and combined with the minimum number of neighbors and the local density ratio coefficient, the number of neighbors K of the node is adaptively adjusted to ensure that each node has at least one neighbor node.

[0021] Select the K nearest neighbors of each node to form a neighbor set, construct an edge set based on the neighbor set, and determine the features of the neighborhood graph.

[0022] Optionally, based on the features of the neighboring graph, a graph convolutional network model is constructed, including:

[0023] A multi-layer graph convolutional network is constructed, taking the node features of the neighboring graph features as input, and obtaining the final node embedding of each node through multi-layer feature mapping based on the adjacency matrix with added self-loops, degree matrix, learnable weight matrix and activation function.

[0024] Calculate the difference in the final node embeddings of any two nodes, and use it as the relation feature between the nodes;

[0025] The relational features are input into the fully connected layer, the loss value is calculated using the cross-entropy loss function, and the model is trained using the stochastic gradient descent method to obtain the graph convolutional network model.

[0026] Optionally, a full node graph is constructed based on the aggregation result, and a tree structure graph corresponding to the aggregation result is determined according to the semantic relationships between nodes in the full node graph, including:

[0027] Based on the aggregation results, all text blocks obtained after clustering are taken as a set of nodes, and each node is connected to all other nodes except itself to construct a full node graph;

[0028] Based on the full node graph, a first classification model combining a graph attention network and a multilayer perceptron is constructed; the first classification model is used to output the semantic relationship categories between nodes and their corresponding score weights.

[0029] Based on the output of the first classification model, edges with semantic relationship categories that have upper and lower semantic relationships are retained, edges with semantic relationship categories that do not have upper and lower semantic relationships are removed, and the full node graph is updated to obtain an initial non-redundant graph.

[0030] The initial non-redundant graph is traversed using a depth-first search algorithm to detect whether there are loop structures in the graph;

[0031] If a loop structure is detected, remove the edges in the loop in order of increasing score weight. After each edge removal, re-detect the loop structure until no loop exists in the graph.

[0032] The tree structure graph is constructed from the final set of remaining nodes and edges.

[0033] Optionally, the nodes in the tree structure graph are classified and identified to determine the tree structure data and construct question-answer pairs to represent the semantic relationships of the files to be identified, including:

[0034] Extract the tree diagram node features of each node in the tree structure diagram;

[0035] Based on the tree graph node features, a second classification model combining a graph attention network and a multilayer perceptron is constructed; the second classification model is used to output a classification label to determine whether a node is a root node;

[0036] Based on the tree structure diagram and the root node classification labels output by the second classification model, all root nodes are determined, and for each root node, the corresponding child nodes are retrieved based on the connection relationship between nodes, and the complete tree structure data is extracted.

[0037] Based on the tree-structured data, all non-root nodes in the graph are traversed. The text content of each node on the path from the root node to the non-root node is concatenated into a question sentence, and the text content of the corresponding child nodes of the non-root node is concatenated into an answer sentence, thus constructing multiple question-answer pairs.

[0038] Optionally, a classification model combining graph attention networks and multilayer perceptrons can be constructed, including:

[0039] A multi-layer graph attention network is constructed, introducing a multi-head attention mechanism, with the target node features as the initial input; the target node features are the node features of the full node graph or the tree graph; the feature update of the single-layer graph attention network is realized based on the learnable weight matrix, attention coefficients, and non-linear activation functions, and the attention coefficients are calculated through learnable attention vectors, feature concatenation, and non-linear activation functions;

[0040] After processing by a multi-layer graph attention network, high-dimensional embedding features of each node are obtained. At the same time, global features are constructed based on the position and text information corresponding to the file to be identified. The global features are input into the multi-layer graph attention network to obtain global high-dimensional embedding features.

[0041] By fusing the high-dimensional embedding features of any two nodes with the global high-dimensional embedding features, we obtain the relationship feature vector between the nodes;

[0042] A multilayer perceptron model is constructed, and the relation feature vector is input into the multilayer perceptron model. The score function is calculated through the weight matrix, bias and activation function of each layer.

[0043] Based on the scoring function, a cross-entropy loss function is constructed, and the loss value of the multilayer perceptron model is calculated based on the cross-entropy loss function.

[0044] The entire graph attention network and multilayer perceptron model are trained using the stochastic gradient descent method to obtain a classification model.

[0045] Secondly, embodiments of this application provide a corpus processing apparatus, including:

[0046] The first determining module is used to identify the file to be identified, obtain the identification result, and perform clustering processing on the identification result to determine the aggregation result;

[0047] The first processing module is used to construct a full node graph based on the aggregation result, and determine the tree structure graph corresponding to the aggregation result according to the semantic relationship between the nodes of the full node graph;

[0048] The second processing module is used to classify and identify the nodes in the tree structure diagram, determine the tree structure data, and construct question-answer pairs to represent the semantic relationships of the files to be identified.

[0049] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any one of the first aspects.

[0050] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method as described in any one of the first aspects.

[0051] Compared with existing technologies, the corpus processing method, apparatus, storage medium, and program product provided in this application solve the problems of neglecting tree diagram parsing and OCR's difficulty in identifying text structure associations by first clustering the file recognition results to obtain aggregated results, then constructing a full node graph and generating a tree structure graph based on the semantic relationships of the nodes; at the same time, the tree structure graph nodes are classified and identified to obtain tree structure data, and then semantic question-answer pairs are constructed to fully mine and utilize text features, thereby improving the accuracy of corpus extraction and solving the problems of insufficient feature utilization and low extraction accuracy. Attached Figure Description

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0053] Figure 1 A flowchart illustrating the corpus processing method provided in the embodiments of this application;

[0054] Figure 2 A flowchart illustrating the process of determining fused features based on location features and text semantic features, provided for embodiments of this application;

[0055] Figure 3 This is a schematic diagram illustrating the process of constructing a tree structure diagram provided in an embodiment of this application;

[0056] Figure 4 A flowchart illustrating the classification of tree diagram nodes provided in this application embodiment;

[0057] Figure 5 This is a schematic diagram of the structure of the corpus processing system provided in the embodiments of this application;

[0058] Figure 6 This is a schematic diagram of the corpus processing device provided in the embodiments of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0061] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0062] To enable those skilled in the art to better understand the embodiments of this application, the following description is provided first:

[0063] Existing corpus extraction methods for large model training can be mainly divided into two categories, as follows:

[0064] (1) Conventional tool extraction method: The most common method is to use professional layout analysis tools to segment the document and extract plain text information such as titles, lists, and paragraphs. Then, the extracted plain text is organized according to preset rules to build a large model training corpus. The core drawback of this method is that it emphasizes text over images, completely ignoring the semantic meaning and structured information carried by images (especially tree diagrams) in the document. This results in incomplete corpus information and an inability to achieve diversified integration of plain text corpus and structured image corpus, thereby limiting the large model's ability to fully understand and accurately generate the overall content of the document.

[0065] (2) Question-generating model-based methods: Some existing technologies propose to obtain question-answer pairs corresponding to images in documents through question-generating models, thereby supplementing image-related corpora. However, such methods rely on the image recognition capabilities of large models and have many problems when processing structured images in the domain: low recognition accuracy, poor stability of recognition results, low output efficiency, insufficient domain adaptability, and lack of fine-grained semantic alignment. These problems directly affect the quality and practicality of image-related question-answer pair corpora and cannot provide high-quality supplementary corpora for training large models.

[0066] As described in the background section, existing technologies suffer from low accuracy and efficiency in extracting structured information from existing document tree diagrams, difficulty in identifying text-structure relationships, and incomplete corpus extraction. To address at least one of these issues, this application provides a corpus processing method, apparatus, storage medium, and program product to achieve diversified, refined, and high-quality corpus processing of documents and images. Specifically, this includes: using an adaptive K-nearest neighbor-graph convolutional neural network method to cluster image OCR recognition results, achieving multi-line text clustering within the same text box; using a graph attention network-multilayer perceptron weighted classification method to extract the semantic relationships of tree diagram nodes in the OCR text clustering results and remove redundant edges; using a tree diagram-based node classification method to identify root nodes, organize hierarchical structures, and construct question-and-answer pairs; and combining layout analysis and chapter rules to process text and images, constructing text-image question-and-answer pairs.

[0067] This application provides a corpus processing method, apparatus, storage medium, and program product. The method and apparatus are based on the same concept, and since the principles by which they solve problems are similar, their implementations can be mutually referenced; repeated details will not be repeated.

[0068] Please refer to Figure 1 This application provides a corpus processing method, including:

[0069] Step 11: Identify the file to be identified to obtain the identification result, and perform clustering processing on the identification result to determine the aggregation result.

[0070] In this application, the document to be identified is recognized in all dimensions. The document to be identified includes various technical documents containing text and images. Optical Character Recognition (OCR) and layout analysis technologies are used to completely extract the text information (including the text recognized in the images) in the document, and the recognition results containing scattered text blocks, text fragments and corresponding position information are obtained. Using preset clustering rules, based on the semantic relevance and positional proximity of the text blocks, the scattered recognition results belonging to the same semantic unit, the same text box or the same content module are aggregated and integrated, invalid recognition noise is eliminated, and a normalized aggregated result is output, which provides basic data support for subsequent semantic structure analysis.

[0071] Step 12: Construct a full node graph based on the aggregation result, and determine the tree structure graph corresponding to the aggregation result according to the semantic relationship between the nodes in the full node graph.

[0072] Here, based on the aggregation results obtained in step 11, each independent aggregation unit is used as a node to construct a full node graph containing all nodes, and a preliminary association framework for each aggregation unit is established. The core semantic relationships (including hierarchical, subordinate, and parallel relationships) between nodes in the full node graph are further analyzed. The edge relationships of the full node graph are filtered and the hierarchy is sorted out. Redundant edges are removed and core semantic relationships are retained. The full node graph is transformed into a tree structure graph that can accurately reflect the logical hierarchy and semantic context of the file content, and the content structure relationship of the file to be identified is clarified.

[0073] Step 13: Classify and identify the nodes in the tree structure diagram, determine the tree structure data, and construct question-answer pairs to represent the semantic relationships of the files to be identified.

[0074] In this embodiment, each node in the tree structure diagram is classified and identified. Combining the semantic features, hierarchical position, and inter-node relationships, the type attributes and logical relationship details of each node are determined, generating standardized tree structure data. Based on this tree structure data, the core question points and corresponding answer points of the document to be identified are mined, and the hierarchy and semantic relationships of the tree structure are transformed into question-answer pairs that conform to natural language logic. These question-answer pairs accurately represent the semantic relationships of the document to be identified, completing the final processing of the corpus.

[0075] The steps described in this application are progressive and logically closed-loop, from the identification and aggregation of original documents to the construction of semantic tree structures and the generation of question-answer pairs. This realizes the entire process of document content processing from scattered identification to structured integration and semantic extraction, effectively solving the problems of existing technologies and improving the precision and quality of corpus processing.

[0076] Optionally, step 11 above includes:

[0077] The acquired file to be identified is preprocessed to obtain preprocessed data; the preprocessing includes at least one of orientation correction, grayscale processing, and size adjustment;

[0078] Optical character recognition is performed on the preprocessed data to obtain recognition results with multiple lines of text information;

[0079] The position information in the text line information is standardized, and the coordinates of the midpoints of the left and right borders and the rotation angle of the borders are calculated. The coordinates of the vertex of the border, the coordinates of the midpoints of the left and right borders, and the rotation angle of the borders are used as position features.

[0080] A pre-trained language model is used to preprocess the text content in the text line information to obtain text semantic features;

[0081] The location features and the text semantic features are fused to obtain the fused features;

[0082] Based on the fusion features, determine the neighborhood map features;

[0083] Based on the features of the neighbor graph, a graph convolutional network model is constructed;

[0084] The recognition results are input into the graph convolutional network model to determine the aggregation result.

[0085] It should be noted that, firstly, the text content of the document to be recognized needs to be extracted using OCR. The original OCR recognition result is multiple scattered text lines, which need to be clustered and merged into a single long sentence. Considering that the Graph Neural Network (GNN) method can effectively utilize the relationships between text lines and global structural information for clustering, and that the node relationships in the text line clustering task are simple and that this task is a prerequisite for semantic structure extraction, this embodiment designs a simple and efficient Graph Convolutional Network (GCN) model to extract text block relationships, and optimizes the clustering accuracy in the following two aspects: 1) When constructing the graph, a fully connected graph is not used, but an Adaptive K-Nearest Neighbors (AdKNN) algorithm is used to control the number of neighboring nodes and improve clustering accuracy; 2) Node features do not rely on a single positional feature, but integrate positional features and semantic features to enrich the dimension of node features.

[0086] In this embodiment, the acquired files to be recognized (focusing on files containing images) are preprocessed to obtain preprocessed data. The preprocessing includes at least one of orientation correction, grayscale processing, and size adjustment, used to optimize image clarity, standardize image format, and eliminate image interference, thereby improving the accuracy of subsequent OCR recognition and feature extraction and ensuring the accuracy of the recognition results. OCR is then performed on the images in the preprocessed data using existing mature OCR processing tools to automatically identify multiple text lines in the image. After recognition, complete information for each text line is returned. Specifically, the text line information includes the coordinates of the four vertices of the text line bounding box and the corresponding text content, forming an original OCR recognition result containing information from multiple text lines, providing basic data for subsequent feature extraction and clustering operations.

[0087] Specifically, refer to Figure 2 As shown, based on the text line information, the positional features of each text line are extracted, including: to avoid interference from different text box heights on the positional features, the coordinates of the left frame point of each text line's bounding box are determined based on the overall image size. Coordinates of the points in the right frame and the rotation angle of the bounding box Standardization is performed; the standardized coordinates of the points in the left and right bounding boxes, along with the bounding box rotation angle, are used as the positional features of this text line. This achieves the normalization of location information, eliminates the impact of size differences, and facilitates subsequent fusion and association analysis with semantic features. Let be the positional feature vector of the i-th text line, where the index "i" is used to distinguish the positional features corresponding to different text lines. Each line contains standardized information on the coordinates of the left and right bounding box points, as well as the bounding box rotation angle, to accurately represent the spatial position of a single text line in the image. Based on the text line information, semantic features of each text line are extracted, including vector extraction of the text content using the pre-trained language model BERT. First, the text content of each text line i is preprocessed by adding "[CLS]" before the text and "[SEP]" after the text to clearly define the start and end boundaries. Then, the preprocessed text is input into the BERT model, and the model calculates a token sequence of length s. ; for this token sequence Average pooling is performed to extract the global semantic information of the entire text line, resulting in the sentence vector of that line, which serves as the text semantic feature, denoted as... This fully explores the core semantic information at the textual level, enriching the feature dimensions. Among them, is the original basic semantic vector obtained after average pooling of the i-th text line, where the index "i" corresponds to the i-th text line; s is the length of the token sequence (i.e., the number of tokens contained in the sequence). Let represent the low-dimensional vector (j=1,2,...,s) corresponding to the j-th token in the i-th text line. Then, use the extracted positional features... Textual semantic features The fusion process integrates location and semantic information to obtain the final feature representation of each text line i, i.e., the fused feature. This enables the synergistic use of location and semantic information, overcomes the limitations of single features, and improves the accuracy of subsequent clustering.

[0088] Based on the fusion features and the preset adaptive K-nearest neighbor algorithm, the neighborhood graph features are determined. A graph convolutional network (GCN) model is constructed based on these neighborhood graph features. All text line information (i.e., recognition results) obtained from OCR is input into the trained GCN model. The GCN model then identifies whether an aggregation relationship exists between any two text line nodes. If multiple text line nodes are identified as having an aggregation relationship, these multiple text lines are merged, integrating the text content of each line. The bounding box coordinates of the merged text block are recalculated to obtain the merged text block (containing the merged text content and corresponding bounding box information). After all merging processes are completed, the resulting set of text blocks is the aggregation result of the recognition results. This aggregation result also serves as the input data for the subsequent tree graph semantic relationship extraction module.

[0089] Furthermore, based on the fusion features, the neighborhood map features are determined, including:

[0090] Each fused feature is treated as a node, and the total number of text lines corresponding to the fused feature is taken as the total number of nodes. The local density of each node is calculated. The local density is obtained by Euclidean distance between nodes and preset hyperparameters.

[0091] Based on the local density of each node, and combined with the minimum number of neighbors and the local density ratio coefficient, the number of neighbors K of the node is adaptively adjusted to ensure that each node has at least one neighbor node.

[0092] Select the K nearest neighbors of each node to form a neighbor set, construct an edge set based on the neighbor set, and determine the features of the neighborhood graph.

[0093] In this embodiment, the fusion features of each text line are combined and progressively processed using the AdKNN algorithm to determine the features of the neighborhood graph. Each fusion feature is treated as an independent node, denoted as v, and each node uniquely corresponds to a text line. The total number of text lines corresponding to each fusion feature is the total number of nodes in the neighborhood graph, denoted as c. All nodes constitute a node set. (c is the total number of text lines), and each node The initial feature vector is Then, the potential aggregation relationships between text lines are treated as edges. Considering that such aggregation relationships only occur among a few spatially adjacent nodes, the KNN method is used to find the neighboring nodes of each node when constructing the graph relationships. Furthermore, traditional methods have a fixed K value, which cannot automatically adjust the range of neighboring nodes for scenarios with varying node densities. Therefore, an AdKNN method based on local density adaptive adjustment is proposed to adaptively adjust the number of neighbors K to obtain a more accurate neighbor set. First, for each node... Calculate its local density As follows, among which Indicates passing through each node Based on the corresponding location features, the Euclidean distance between the two nodes is calculated. The Euclidean distance is used to characterize the spatial proximity of the text lines corresponding to the two nodes in the image. The smaller the value, the closer the two lines of text are in spatial position, and the higher the probability that there is an aggregation relationship. Hyperparameters for controlling density smoothness: Based on local density For each node Determine the number of neighbors : ;in, This represents the minimum number of neighbors, ensuring that each node has at least one connection. density By proportionality factor Adjust the magnification and round down. Then, for each node, calculate the Euclidean distance between nodes as previously obtained. Filter out the closest ones Each node consists of several neighboring nodes. Select its nearest The neighbors form a set Based on the neighbor set of all nodes Construct the edge set of the neighborhood graph : That is, when the node Belongs to node When creating a neighbor set, an edge is constructed between two nodes to represent a potential aggregation relationship between them. Ultimately, the neighborhood graph features are composed of the node set V, the edge set E, and the fused features (node ​​features) corresponding to each node. This provides complete graph structure support for the construction of subsequent graph convolutional network models.

[0094] Furthermore, based on the features of the neighbor graph, a graph convolutional network model is constructed, including:

[0095] A multi-layer graph convolutional network is constructed, taking the node features of the neighboring graph features as input, and obtaining the final node embedding of each node through multi-layer feature mapping based on the adjacency matrix with added self-loops, degree matrix, learnable weight matrix and activation function.

[0096] Calculate the difference in the final node embeddings of any two nodes, and use it as the relation feature between the nodes;

[0097] The relational features are input into the fully connected layer, the loss value is calculated using the cross-entropy loss function, and the model is trained using the stochastic gradient descent method to obtain the graph convolutional network model.

[0098] In this embodiment, a multi-layer graph convolutional network (GCN) is constructed. The number of network layers is set to L, which can be adjusted according to the actual corpus processing accuracy requirements. The node features in the neighbor graph features, i.e., the fused features corresponding to each node, are used as the input data for this GCN model. Features are mapped to high-dimensional embeddings in the l-th layer via GCN. and initialization The update rule for layer l of GCN is: ;in, Let l be the node feature representation matrix of the l-th layer; Adjacency matrix for adding self-loops; for The degree matrix; It is a learnable weight matrix; The activation function is sigmoid or reLU. After feature mapping through an L-layer GCN network, the final node embedding for each node is obtained. The final node embedding fully represents the deep features of the nodes and the potential relationships between them. The difference between the final node embeddings of any two nodes is calculated, and this difference is used as the relationship feature between the two nodes. That is, for nodes i and j, their clustering relationship feature is defined as the difference in their embeddings. This relational feature accurately reflects the degree of aggregation association between the text lines corresponding to two nodes, providing core feature support for subsequent aggregation relationship prediction. (The relational feature is then used in this context.) Input a fully connected (FC) layer and pass it through the activation function. The final output is obtained from the output. : ;in, The weights of the fully connected layer, This is the bias term for the fully connected layer.

[0099] The relationship characteristics between each node obtained from the above calculations Each input is fed into a fully connected layer (FC), where the relational features are further mapped and integrated; and then activated by an activation function. The final output is obtained from the output. : ;in, The weights of the fully connected layer, This is the bias term for the fully connected layer.

[0100] During model training, the cross-entropy (CE) loss function is used to calculate the loss value between the model's predicted results and the true labels, where the true labels are... Represents a node and Does an aggregation relationship exist between them (marked as 1 if it exists, 0 if it does not)? Let the actual label be... The loss is calculated using the cross-entropy (CE) loss function: Then, by using stochastic gradient descent to train the loss function, we can obtain a result based on the features of the neighborhood graph. GCN classification model for text line aggregation relationships Then, stochastic gradient descent was used to perform backpropagation training on this loss function, continuously adjusting the learnable weight matrices of each layer of the GCN model. and the weight parameters of the fully connected layer Bias terms The training continues until the model loss value converges to a preset threshold, at which point the model's prediction accuracy reaches its optimal level. After training, a GCN model for predicting text line aggregation relationships can be obtained, which can be directly used for text line clustering and merging operations.

[0101] Optionally, step 12 above includes:

[0102] Based on the aggregation results, all text blocks obtained after clustering are taken as a set of nodes, and each node is connected to all other nodes except itself to construct a full node graph;

[0103] Based on the full node graph, a first classification model combining a graph attention network and a multilayer perceptron is constructed; the first classification model is used to output the semantic relationship categories between nodes and their corresponding score weights.

[0104] Based on the output of the first classification model, edges with semantic relationship categories that have upper and lower semantic relationships are retained, edges with semantic relationship categories that do not have upper and lower semantic relationships are removed, and the full node graph is updated to obtain an initial non-redundant graph.

[0105] The initial non-redundant graph is traversed using a depth-first search algorithm to detect whether there are loop structures in the graph;

[0106] If a loop structure is detected, remove the edges in the loop in order of increasing score weight. After each edge removal, re-detect the loop structure until no loop exists in the graph.

[0107] The tree structure graph is constructed from the final set of remaining nodes and edges.

[0108] In this embodiment, based on the aggregation result, all text blocks obtained after clustering are taken as a node set. Each node is connected to all other nodes except itself, constructing a full node graph. The aggregation result is the set of text blocks obtained after clustering and merging the OCR recognition results in step 11. Each node uniquely corresponds to a text block, and the node information includes the text content of the text block and the corresponding bounding box position information. The connections between nodes are used for subsequent semantic relationship mining and filtering. Based on the full node graph, a first classification model combining a graph attention network and a multilayer perceptron is constructed. The first classification model outputs the semantic relationship categories between nodes and their corresponding score weights. The semantic relationship categories are divided into two types: "existing upper and lower layer semantic relationships" and "not existing upper and lower layer semantic relationships." The score weights are used to characterize the confidence level of the existence of corresponding semantic relationships between nodes; the higher the weight value, the stronger the confidence level. Based on the output of the first classification model, edges with semantic relationship categories containing hierarchical semantic relationships are retained, while edges with semantic relationship categories not containing hierarchical semantic relationships are removed. The full node graph is then updated to obtain an initial non-redundant graph. By filtering edges, meaningless node associations are eliminated, while core semantic associations are retained, reducing redundant computations in subsequent graph structure processing. A depth-first search algorithm is used to traverse the initial non-redundant graph to detect the presence of loop structures. The depth-first search algorithm, through recursive traversal of nodes and edges, can efficiently search for closed loop associations in the graph, ensuring the rationality of the tree structure graph construction (the tree structure is a connected graph without loops). If a loop structure is detected, edges are removed sequentially according to their score weights from low to high. After each edge removal, the loop structure is re-detected until no loops exist in the graph. This operation uses confidence priority filtering to retain edges with high score weights and closer semantic associations, ensuring that removing loop structures does not destroy the core semantic hierarchy. A tree structure graph is constructed from the final set of remaining nodes and edges. This tree structure graph accurately reflects the hierarchical semantic relationship between text blocks, clarifies the logical context of the content of the document to be identified, and provides structured support for the construction of subsequent semantic question-answering pairs.

[0109] In a specific implementation method, refer to Figure 3 The semantic relationship extraction steps of the tree diagram shown are as follows: After obtaining the aggregation results of OCR recognition, extract the fusion features for each clustered text block i, denoted as . ;in, Let be the fused feature vector of the i-th text block. This vector is a concatenation and fusion of standardized positional features and text semantic features, containing both spatial and semantic information of the text block. This provides core feature support for subsequent full-node graph construction and model training. Since the relationships and spans of the text blocks in the file to be recognized are diverse, a single proximity connection cannot cover all potential semantic associations. Therefore, edges are constructed for each node to build a full-node graph, ensuring complete capture of all types of associations between text blocks. Assume the node set at this point is . , where t is the total number of text blocks after clustering (consistent with the definition of the total number of nodes c above, only the sign is adjusted due to the change in the number of text blocks after aggregation); except for the current node All other nodes are considered as its neighbors to construct an edge set. A complete node graph is formed by the set of nodes V and the set of edges E, denoted as . .in, , represents the set of nodes in a full node graph. These are the nodes corresponding to the 1st, 2nd, ..., tth aggregated text blocks, respectively. The subscripts are used to distinguish different text block nodes. t is the total number of text blocks after clustering, and its value is determined by the aggregation result, that is, the number of text blocks obtained after merging in step 11, which varies with the content of the image to be processed. The edge set represents the set of edges in a fully nodeed graph, used to characterize the initial connections between all nodes; edge set The elements in are node pairs The value selection rule is " This means that each node establishes an edge with all other nodes except itself, ensuring that no potential semantic connection between any two text blocks is missed, and the total number of edges is [number missing]. ; This represents the completed full-node graph, where... For a set of nodes, The graph is a set of edges; it is a fully connected graph (except that there are no self-loops in the nodes themselves), which fully carries the node features of all aggregated text blocks and the initial connections between nodes, providing graph structure input for the construction of subsequent node relationship classification models.

[0110] Furthermore, a multi-layer GAT-MLP relationship-weighted classification model is constructed to predict whether semantic relationships exist between nodes. First, an M-layer GAT network is constructed, with each node... The features are mapped layer by layer through the GAT network to obtain the high-dimensional embedding features of the m-th layer. The initial high-dimensional embedding of the node The feature update rules for a single-layer GAT network are as follows: Among them, among them The vector in the middle is the feature vector of the m-th layer. It is the set of neighbors of node i. It is the learnable weight matrix of the m-th layer. It is the attention coefficient between node i and node j. This is a non-linear activation function. The attention coefficients here... The calculation formula is: ;in, It is a learnable attention vector of m layers, and || represents feature concatenation. Let i be the set of neighboring nodes of node i; LeakyReLU is a non-linear activation function, an improved version of the ReLU function, used to perform a non-linear transformation on the inner product of concatenated features and attention vectors, which can effectively alleviate the gradient vanishing problem and improve model training efficiency.

[0111] Furthermore, to enhance the model's expressive power and improve the comprehensiveness of feature extraction, a multi-head attention mechanism is introduced into the GAT model. Each attention head independently calculates its attention coefficient and updates the features. Finally, the outputs of all attention heads are averaged to obtain the final feature update result for the m-th layer. Therefore, the feature update formula for the m-th layer is modified as follows: Where K is the number of attention heads (a preset hyperparameter that can be adjusted according to the model's accuracy requirements, such as 8 heads). and These are the learnable weight matrix and attention coefficients of the k-th attention head at layer m, respectively. Dimensions and single-head attention mechanism Consistency Each attention head has an independent weight matrix, which can independently perform linear mapping and attention calculation on node features, capturing node association features of different dimensions.

[0112] Next, the hidden layer outputs of the GAT need to be transformed into relational features between node i and node j. This involves the outputs of the M-layer GAT model. and In addition, global features of the entire image also need to be added. To enhance the connection between node relationships and the overall semantics of the image, here... It's entirely constructed using the coordinates of the entire image and all the text information. The feature vectors obtained after the M-layer GAT model output are used to define the relationship feature vectors between node i and node j based on the above features, as follows: ;in, node With nodes The relationship feature vector, where the subscript "ij" corresponds to the node pair This vector integrates the deep features of the two nodes and the global features of the image, and is used for relationship classification in the subsequent MLP model. Represents a node The final high-dimensional embedding feature vector obtained after processing by an M-layer GAT network; Represents a node The final high-dimensional embedding feature vector obtained after processing by an M-layer GAT network; This represents the global high-dimensional embedding feature vector obtained after the global features of the image are processed by an M-layer GAT network. This global feature is constructed by concatenating the bounding box coordinates of the entire image (after standardization) and the text content of all text blocks (after semantic features are extracted by the BERT model). This global feature represents the overall semantics and positional relationships of the image, which can make up for the limitations of single-node features and improve the accuracy of relationship classification.

[0113] Obtain the relation feature vector Then, a p-layer MLP model is constructed to perform deep mapping and classification of relation feature vectors, and its score function is calculated. : ;in, and Let these represent the weight matrix and bias of the p-th layer, respectively. Let represent the activation function of the p-th layer. Here, for a binary classification task, the sigmoid function is used to convert the classification result into a probability. Finally, let the true label be . The loss is calculated using the cross-entropy (CE) loss function: Then, stochastic gradient descent is used to perform backpropagation training on the loss function, continuously adjusting the learnable weight matrix and attention vector of the GAT model and the weight matrices and biases of each layer of the MLP model until the model loss converges to a preset threshold. This yields the first classification model based on node relationships in a full node graph. The model output includes the semantic relationship categories of node i and node j. and corresponding score weights .

[0114] The node graph is updated based on the node relationships identified by the first classification model. It checks whether there are cycles in the graph, removes edges according to weights, and iteratively updates the calculation until all cycles are removed, thus obtaining the final acyclic tree structure graph. Tree structure diagram Its key feature is that it has only one or more tree-like relationship structures, no cycles, and can fully express the semantic information such as hierarchical and parallel relationships of text in images. It can be further used to extract high-value question-and-answer pairs. The specific post-processing process is as follows: For a given node graph... Each edge , Corresponding relationship tags A value of 1 indicates the existence of hierarchical semantic relationships that need to be preserved, while a value of 0 indicates the absence of hierarchical semantic relationships and their removal. The resulting graph is... Search Image To determine if a cycle exists in the graph, a depth-first search (DFS) algorithm is used to detect cycle structures by traversing all edges and node connections in the graph. Once a cycle is found, it is processed according to the score weights of the edges within the cycle. Edges are removed sequentially from lowest to highest weight. After each edge removal, the Depth-First Search (DFS) algorithm is used again to check for any remaining cycles in the graph. This process is repeated iteratively until no cycles remain. This operation uses confidence priority filtering to retain edges with high score weights and stronger semantic connections, ensuring that removing cycles does not disrupt the core semantic hierarchy. Finally, the remaining set of edges is combined with the nodes to construct the desired tree structure graph. .

[0115] Optionally, step 13 above includes:

[0116] Extract the tree diagram node features of each node in the tree structure diagram;

[0117] Based on the tree graph node features, a second classification model combining a graph attention network and a multilayer perceptron is constructed; the second classification model is used to output a classification label to determine whether a node is a root node;

[0118] Based on the tree structure diagram and the root node classification labels output by the second classification model, all root nodes are determined, and for each root node, the corresponding child nodes are retrieved based on the connection relationship between nodes, and the complete tree structure data is extracted.

[0119] Based on the tree-structured data, all non-root nodes in the graph are traversed. The text content of each node on the path from the root node to the non-root node is concatenated into a question sentence, and the text content of the corresponding child nodes of the non-root node is concatenated into an answer sentence, thus constructing multiple question-answer pairs.

[0120] In this embodiment, tree graph node features are extracted from each node in the tree structure graph. These tree graph node features include semantic features, positional features, and hierarchical features of the text block corresponding to the node. Semantic features are extracted using the BERT model, positional features are the standardized coordinates of the text block bounding box, and hierarchical features are determined based on the node's hierarchical position in the tree structure graph, collectively constituting a complete feature representation of the node. Based on these tree graph node features, a second classification model combining a graph attention network and a multilayer perceptron is constructed. This second classification model outputs a classification label determining whether a node is a root node. The classification label is divided into two categories: "root node" and "non-root node." A root node is a node in the tree structure graph without a parent node and at the highest level, serving as the starting node for each independent semantic branch. Based on the tree structure graph and the root node classification labels output by the second classification model, all root nodes are determined. For each root node, corresponding child nodes are retrieved based on the connection relationships between nodes, extracting complete tree structured data. The tree structured data includes the hierarchical relationships of all root nodes and child nodes, node features, and text content, achieving standardized data representation of the tree structure graph. Based on the tree-structured data, all non-root nodes in the graph are traversed. The text content of each node on the path from the root node to the non-root node is concatenated into a question sentence, and the text content of the corresponding child nodes of the non-root node is concatenated into an answer sentence, thus constructing multiple question-answer pairs. The question-answer pairs accurately correspond to the core semantic relationships in the file, realizing the semantic extraction of the file content and completing the final processing of the corpus.

[0121] After obtaining the aggregation results, this application finds it difficult to fully extract the tree-like dependencies between text blocks using only positional or semantic features. Therefore, this embodiment employs a method of constructing a full-node graph, comprehensively connecting all nodes and their relationships to fully reflect the structure and logical information of the image. A multi-layer graph attention network (GAT) model is used, leveraging its flexible attention mechanism and powerful generalization ability to extract complex node features. The features are then dimensionality-reduced using a multi-layer perceptron (MLP) model, outputting the node relationship types and their score weights. Subsequently, after removing redundant loop edges, one or more tree-structured image semantic extraction results are finally generated. This method can more comprehensively capture the complex relationships between text blocks, thereby more accurately restoring the semantic structure of the image.

[0122] In one specific implementation, a second classification model is constructed based on a tree diagram, and this model is used for node classification and image corpus extraction. The tree diagram is converted into standardized tree structure data, and question-answer pairs are extracted based on the hierarchical structure of the tree diagram. Since the extraction of tree structure data requires first clarifying the hierarchical attributes of each node (root node or non-root node), the GAT-MLP model is used again to classify whether a node is a root node based on the tree diagram. After identifying all root nodes in the diagram, the corresponding child nodes are retrieved by combining the node connection relationships in the tree diagram, thus sorting out the complete tree structure data. Finally, question-answer pairs are constructed according to the hierarchical relationships to fully extract the semantic association information in the images.

[0123] Reference Figure 4 As shown, the specific operation process is as follows. The nodes here are the tree diagram node features of each node in the tree structure diagram. The quantity t and the extracted features and graph Similarly, node features are still represented as Construct a tree diagram. Here, we directly use the completed tree structure diagram. This is sufficient; no reconstruction is needed. The tree structure graph is acyclic and contains the core semantic hierarchical relationships between nodes, providing graph structure support for subsequent node classification and tree structure data extraction. A second classification model for GAT-MLP is constructed to predict whether each node is the root node. Its construction and training process is similar to that of the first classification model of GAT-MLP described above, therefore the simplified process is as follows: The fused features of each node are input into the N-layer GAT model to obtain the processed feature vector: Then the features Inputting the data into an MLP model yields the classification score function for each node: Finally, the cross-entropy (CE) loss function is used to calculate the loss. By training the model using stochastic gradient descent with respect to the loss function, a node classification model can be obtained. The model output is the label indicating whether node i is the root node. .based on It can retrieve all node relationships in the graph, based on labels. We can identify all root nodes in the graph, and then retrieve the corresponding child nodes for each root node based on node relationships, forming a complete tree-structured data structure. Based on this tree-structured data, question-answer pairs are generated according to hierarchical rules. For the graph... A non-root node The path from the root node to itself can be defined as: In a tree, r represents the root node. Indicates from r to The middle node, Represents a node The parent node. For the parent node Define the set of all its child nodes. Then a question-and-answer pair can be constructed. : Here, the text content of each node in Q can be concatenated to form a complete question sentence, and the text of each node in A can be separated by delimiters to form a multi-phrase answer sentence. Using this method, all possible intermediate nodes in the graph can be traversed to construct multiple sets... Question and answer are correct. For example, here we can consider the nodes in question Q (root node r, intermediate nodes, etc.). Non-root nodes The text content of ) is concatenated in the path order to form a complete question sentence; the child nodes of answer A ( The text content is concatenated using preset separators (such as commas and semicolons) to form an answer sentence containing multiple phrases. Using this method, multiple question-answer pairs can be constructed by traversing all possible non-root nodes in the tree-structured data. This enables semantic extraction of image corpora, completing the final operation of corpus processing. Among these, Tree structure diagram Any non-root node in the tree structure, with the subscript "x" used to distinguish different non-root nodes; as the core node for constructing question-answer pairs, its path information determines the content of the question sentence, and the set of child nodes of its parent node determines the content of the answer sentence. r represents the root node in the tree structure, which is the highest-level node in the tree structure and has no parent node; it is the starting node of each independent semantic branch, and its text content is the starting part of the question sentence, laying the semantic foundation of the question. This represents the path from the root node r to non-root nodes. The intermediate nodes are represented by subscripts "1,2,...,k-1" to indicate their hierarchical order; their text content is concatenated in order to supplement the semantic details of the question and clarify the specific object to which the question refers. Represents a non-root node The parent node, with the subscript "k-1" indicating its hierarchical position in the path; the set of all its child nodes constitutes the answer to the question-answer pair, linking the semantic correspondence between the question and the answer. This represents a single question-and-answer pair, where Q is the question sentence and A is the corresponding answer sentence. The question-and-answer pair accurately corresponds to the hierarchical semantic association of text blocks in the image and can be directly used as high-value corpus data for subsequent tasks such as semantic retrieval and question-and-answer model training.

[0124] Furthermore, a classification model combining graph attention networks and multilayer perceptrons is constructed, including:

[0125] A multi-layer graph attention network is constructed, introducing a multi-head attention mechanism, with the target node features as the initial input; the target node features are the node features of the full node graph or the tree graph; the feature update of the single-layer graph attention network is realized based on the learnable weight matrix, attention coefficients, and non-linear activation functions, and the attention coefficients are calculated through learnable attention vectors, feature concatenation, and non-linear activation functions;

[0126] After processing by a multi-layer graph attention network, high-dimensional embedding features of each node are obtained. At the same time, global features are constructed based on the position and text information corresponding to the file to be identified. The global features are input into the multi-layer graph attention network to obtain global high-dimensional embedding features.

[0127] By fusing the high-dimensional embedding features of any two nodes with the global high-dimensional embedding features, we obtain the relationship feature vector between the nodes;

[0128] A multilayer perceptron model is constructed, and the relation feature vector is input into the multilayer perceptron model. The score function is calculated through the weight matrix, bias and activation function of each layer.

[0129] Based on the scoring function, a cross-entropy loss function is constructed, and the loss value of the multilayer perceptron model is calculated based on the cross-entropy loss function.

[0130] The entire graph attention network and multilayer perceptron model are trained using the stochastic gradient descent method to obtain a classification model.

[0131] In this embodiment, the method further includes constructing a classification model combining a graph attention network and a multilayer perceptron. Specifically, a multilayer graph attention network is constructed, and a multi-head attention mechanism is introduced, using target node features as initial input. The target node features are the node features of the full node graph or the tree graph (corresponding to the input differences of the first classification model and the second classification model). The feature update of the single-layer graph attention network is implemented based on a learnable weight matrix, attention coefficients, and a nonlinear activation function. The attention coefficients are calculated through a learnable attention vector, feature concatenation, and a nonlinear activation function. The multi-head attention mechanism can effectively capture the correlation features between nodes in different dimensions, improve the comprehensiveness of feature extraction, and the attention coefficients are used to characterize the correlation importance between nodes, realizing weighted feature updates.

[0132] After processing by a multi-layer graph attention network, high-dimensional embedding features of each node are obtained. At the same time, global features are constructed based on the position and text information of the file to be identified. The global features are input into the multi-layer graph attention network to obtain global high-dimensional embedding features. The node high-dimensional embedding features represent the deep features of a single node, while the global high-dimensional embedding features represent the overall semantics and positional association of the file. The combination of the two can make up for the limitations of single node features and improve the prediction accuracy of the classification model.

[0133] By fusing the high-dimensional embedding features of any two nodes with the global high-dimensional embedding features, a relationship feature vector between nodes is obtained. The fusion method uses vector concatenation, which concatenates the high-dimensional embedding features of two nodes with the global high-dimensional embedding features into a complete vector, comprehensively integrating individual node features and global association features, and providing sufficient feature support for the classification of relationships between nodes.

[0134] A multilayer perceptron model is constructed, and the relation feature vector is input into the multilayer perceptron model. The score function is calculated through the weight matrix, bias and activation function of each layer. The multilayer perceptron model is used to perform deep mapping and classification of relation feature vectors. The score function is used to quantify the confidence that there is a corresponding semantic relationship between nodes, and to provide a basis for the classification result output.

[0135] Based on the scoring function, a cross-entropy loss function is constructed, and the loss value of the multilayer perceptron model is calculated based on the cross-entropy loss function. The cross-entropy loss function can effectively measure the deviation between the model's prediction results and the true labels, and is suitable for binary or multi-class classification tasks, providing a loss basis for model parameter optimization.

[0136] The entire graph attention network and multilayer perceptron model are trained using stochastic gradient descent to obtain a classification model. During training, the learnable weight matrix and attention vector of the graph attention network and the weight matrices and biases of each layer of the multilayer perceptron are continuously adjusted based on backpropagation of the loss value until the model loss converges to a preset threshold, ensuring the prediction accuracy and stability of the classification model. This classification model can be used as the first classification model and the second classification model respectively, depending on the differences in the features of the input target nodes.

[0137] Reference Figure 5 As shown, to fully utilize the semantic information of document text and images to construct a high-quality large-scale model corpus, this embodiment proposes a corpus processing system. This system consists of three parts: a document input module, a corpus extraction module, and a large-scale model corpus. The corpus extraction module is the core, and all modules collaborate to complete the construction of the multimodal corpus, as detailed below:

[0138] The document input module is responsible for the access and unified format conversion of multiple document types. It supports the access of WORD, PDF and image (JPG, PNG, etc.) format documents. After receiving the original documents through the access interface, it converts WORD and image documents into PDF format and outputs them to the corpus extraction module after conversion.

[0139] The corpus extraction module is responsible for extracting text and image corpora from PDF documents and converting them into question-and-answer pairs. It mainly consists of modules for layout analysis, text segmentation, text block question-and-answer pair extraction, image preprocessing, tree-graph semantic feature extraction, and image semantic question-and-answer pair extraction. The layout analysis module uses tools such as PP-Structure and minerU to parse the PDF document layout, identify and segment content blocks such as titles, body text, and images, and annotate their location information and content type. It outputs the results to the corresponding modules according to text type (titles, body text) and image type (image blocks). The text segmentation module receives text content, sorts the titles hierarchically, and associates and segments the body text with the corresponding titles based on the title hierarchy and location information, forming a "title-body paragraph" correspondence, which is then output to the text block question-and-answer pair extraction module. The text block question-and-answer pair extraction module constructs text block question-and-answer pairs based on the "title-body paragraph" correspondence, using the title as the question sentence (Q) and the corresponding body text as the answer sentence (A).<Q,A> Multi-level headings use "superior heading and subordinate heading" as questions, with the corresponding body text as the answer, and are then summarized and output to the large model corpus. The image preprocessing module receives image content and sequentially performs orientation correction, grayscale processing, and size adjustment preprocessing to optimize image quality before outputting to the tree graph semantic feature extraction module. The tree graph semantic feature extraction module receives the preprocessed image, extracts text content, location information, and semantic relationships between text blocks, and constructs a tree structure graph. It consists of OCR text recognition, text clustering, semantic relationship extraction, and node classification modules, as detailed below:

[0140] The OCR text recognition module uses algorithms such as PP-OCR to recognize text in images and standardize bounding box coordinates. It divides the text into lines and outputs the results to the text clustering module. The text clustering module extracts text line position features and BERT semantic features, concatenates them into fused features, and uses a preset algorithm to cluster them into text blocks, which are then output to the semantic relation extraction module. The semantic relation extraction module uses these text blocks as nodes to construct a full node graph, builds a multi-layer GAT-MLP relationship-weighted first-class classification model, predicts the semantic relation categories and score weights between nodes, retains edges with hierarchical semantic relationships, detects and removes loop edges according to weight using the DFS algorithm, and constructs an acyclic tree structure graph.

[0141] The node classification module adopts the second classification model of GAT-MLP, predicts the root node label based on the tree graph node features, filters out all root nodes, retrieves child nodes by combining node connection relationships, extracts complete tree structured data, and outputs it to the image semantic question answering extraction module.

[0142] Image semantic question-answering pair extraction is used to receive tree-structured data, traverse all non-root nodes, concatenate the text of the path nodes from the root node to the current node to form the question Q, and concatenate the text of all child nodes of the parent node of the current node to form the answer A, constructing image semantic question-answering pairs, and then summarizing and outputting them to the large model corpus.

[0143] The large model corpus is used to store text blocks, image semantic question-and-answer pairs, and processed question-and-answer pair knowledge bases. It stores multimodal corpus information such as text and images extracted from layout analysis. It is indexed and managed through the corpus cataloging module to build a structured and diverse multimodal corpus.

[0144] In summary, this application extracts document text and images through layout analysis. Text is used to construct question-and-answer pairs based on the title structure. Images are recognized by OCR, and text is clustered to construct a text graph. Positional and semantic information is fused to construct node features. A GNN model is used to predict node relationships and weights, reconstructing a semantic tree structure. Root or leaf nodes are then determined through tree graph node classification to extract image question-and-answer pairs, achieving diversified document corpus extraction and improving corpus quality and completeness. A lightweight graph is constructed using the AdKNN algorithm optimized by local density estimation, fusing text block position and semantic features as vectors. Multi-layer GCN is used to achieve accurate clustering of multiple lines of text within the same text box. A full graph is constructed based on the OCR clustering results. Features are extracted through multi-layer GAT, fusing node features and global information to construct relationship features. MLP is input to predict node relationships, and redundant edges are removed using weight thresholds to obtain the tree graph extraction results. A model is constructed based on the tree graph relationships. Root nodes are classified using GAT+MLP, and the hierarchical tree structure is organized, thereby transforming the hierarchical semantic relationships of images into structured, high-value image semantic question-and-answer pairs.

[0145] Compared with the prior art, this application can achieve at least one of the following technical advantages:

[0146] 1) Breaking through the limitations of traditional methods that ignore the internal semantics of images, this method extracts both the document text and the structured semantics within the images. By combining three graph structures with the GNN method, it mines the implicit relationships in images and improves the quality of the corpus.

[0147] 2) Abandoning the shortcomings of traditional methods that rely on specific templates, this method integrates positional and semantic features, uses AdKNN to construct a lightweight graph in conjunction with GCN, and adapts to the clustering needs of various discrete texts in tree-graph images.

[0148] 3) Construct a GAT-MLP model based on a fully connected graph, integrate multi-dimensional features to improve the reasoning ability of complex relationships; eliminate redundant edges by weight threshold and loop detection to ensure the accuracy of tree structure; combine root node classification to efficiently transform graph structure into tree data and high-precision question-answer pairs.

[0149] The various methods of the embodiments of this application have been described above. Apparatus for implementing the above methods will now be provided.

[0150] Please refer to Figure 6 This application also provides a corpus processing apparatus, including:

[0151] The first determining module 61 is used to identify the file to be identified, obtain the identification result, and perform clustering processing on the identification result to determine the aggregation result;

[0152] The first processing module 62 is used to construct a full node graph based on the aggregation result, and determine the tree structure graph corresponding to the aggregation result according to the semantic relationship between the nodes of the full node graph;

[0153] The second processing module 63 is used to classify and identify the nodes in the tree structure diagram, determine the tree structure data, and construct question-answer pairs to represent the semantic relationships of the files to be identified.

[0154] Optionally, the first determining module 61 described above includes:

[0155] The first acquisition unit is used to preprocess the acquired file to be identified and obtain preprocessed data; the preprocessing includes at least one of orientation correction, grayscale processing, and size adjustment;

[0156] The first processing unit is used to perform optical character recognition on the preprocessed data to obtain a recognition result with multiple text line information.

[0157] The second processing unit is used to standardize the position information in the text line information, calculate the coordinates of the midpoints of the left and right borders of the text line bounding box and the border rotation angle, and use the vertex coordinates of the bounding box, the midpoint coordinates of the left and right borders, and the border rotation angle as position features.

[0158] The third processing unit is used to preprocess the text content in the text line information using a pre-trained language model to obtain text semantic features;

[0159] The fourth processing unit is used to fuse the location features and the text semantic features to obtain fused features;

[0160] The first determining unit is used to determine the neighborhood map features based on the fusion features;

[0161] The fifth processing unit is used to construct a graph convolutional network model based on the features of the neighbor graph;

[0162] The second determining unit is used to input the recognition result into the graph convolutional network model to determine the aggregation result.

[0163] Optionally, the first determining unit described above is specifically used for:

[0164] Each fused feature is treated as a node, and the total number of text lines corresponding to the fused feature is taken as the total number of nodes. The local density of each node is calculated. The local density is obtained by Euclidean distance between nodes and preset hyperparameters.

[0165] Based on the local density of each node, and combined with the minimum number of neighbors and the local density ratio coefficient, the number of neighbors K of the node is adaptively adjusted to ensure that each node has at least one neighbor node.

[0166] Select the K nearest neighbors of each node to form a neighbor set, construct an edge set based on the neighbor set, and determine the features of the neighborhood graph.

[0167] Optionally, the fifth processing unit described above is specifically used for:

[0168] A multi-layer graph convolutional network is constructed, taking the node features of the neighboring graph features as input, and obtaining the final node embedding of each node through multi-layer feature mapping based on the adjacency matrix with added self-loops, degree matrix, learnable weight matrix and activation function.

[0169] Calculate the difference in the final node embeddings of any two nodes, and use it as the relation feature between the nodes;

[0170] The relational features are input into the fully connected layer, the loss value is calculated using the cross-entropy loss function, and the model is trained using the stochastic gradient descent method to obtain the graph convolutional network model.

[0171] Optionally, the first processing module 62 described above is specifically used for:

[0172] Based on the aggregation results, all text blocks obtained after clustering are taken as a set of nodes, and each node is connected to all other nodes except itself to construct a full node graph;

[0173] Based on the full node graph, a first classification model combining a graph attention network and a multilayer perceptron is constructed; the first classification model is used to output the semantic relationship categories between nodes and their corresponding score weights.

[0174] Based on the output of the first classification model, edges with semantic relationship categories that have upper and lower semantic relationships are retained, edges with semantic relationship categories that do not have upper and lower semantic relationships are removed, and the full node graph is updated to obtain an initial non-redundant graph.

[0175] The initial non-redundant graph is traversed using a depth-first search algorithm to detect whether there are loop structures in the graph;

[0176] If a loop structure is detected, remove the edges in the loop in order of increasing score weight. After each edge removal, re-detect the loop structure until no loop exists in the graph.

[0177] The tree structure graph is constructed from the final set of remaining nodes and edges.

[0178] Optionally, the second processing module 63 described above is specifically used for:

[0179] Extract the tree diagram node features of each node in the tree structure diagram;

[0180] Based on the tree graph node features, a second classification model combining a graph attention network and a multilayer perceptron is constructed; the second classification model is used to output a classification label to determine whether a node is a root node;

[0181] Based on the tree structure diagram and the root node classification labels output by the second classification model, all root nodes are determined, and for each root node, the corresponding child nodes are retrieved based on the connection relationship between nodes, and the complete tree structure data is extracted.

[0182] Based on the tree-structured data, all non-root nodes in the graph are traversed. The text content of each node on the path from the root node to the non-root node is concatenated into a question sentence, and the text content of the corresponding child nodes of the non-root node is concatenated into an answer sentence, thus constructing multiple question-answer pairs.

[0183] Optionally, the apparatus of this application further includes a construction module for constructing a classification model combining a graph attention network and a multilayer perceptron, wherein the construction module is specifically used for:

[0184] A multi-layer graph attention network is constructed, introducing a multi-head attention mechanism, with the target node features as the initial input; the target node features are the node features of the full node graph or the tree graph; the feature update of the single-layer graph attention network is realized based on the learnable weight matrix, attention coefficients, and non-linear activation functions, and the attention coefficients are calculated through learnable attention vectors, feature concatenation, and non-linear activation functions;

[0185] After processing by a multi-layer graph attention network, high-dimensional embedding features of each node are obtained. At the same time, global features are constructed based on the position and text information corresponding to the file to be identified. The global features are input into the multi-layer graph attention network to obtain global high-dimensional embedding features.

[0186] By fusing the high-dimensional embedding features of any two nodes with the global high-dimensional embedding features, we obtain the relationship feature vector between the nodes;

[0187] A multilayer perceptron model is constructed, and the relation feature vector is input into the multilayer perceptron model. The score function is calculated through the weight matrix, bias and activation function of each layer.

[0188] Based on the scoring function, a cross-entropy loss function is constructed, and the loss value of the multilayer perceptron model is calculated based on the cross-entropy loss function.

[0189] The entire graph attention network and multilayer perceptron model are trained using the stochastic gradient descent method to obtain a classification model.

[0190] It should be noted that the apparatus in this embodiment corresponds to the corpus processing method described above. The implementation methods in each of the above embodiments are applicable to the embodiments of this apparatus and can achieve the same technical effect. The apparatus provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0191] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described corpus processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0192] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described corpus processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0195] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A corpus processing method, characterized in that, include: The document to be identified is identified to obtain the identification result, and the identification result is clustered to determine the aggregation result; A full node graph is constructed based on the aggregation result, and a tree structure graph corresponding to the aggregation result is determined according to the semantic relationships between the nodes in the full node graph. The nodes in the tree structure graph are classified and identified to determine the tree structure data and construct question-answer pairs to represent the semantic relationships of the files to be identified.

2. The method according to claim 1, characterized in that, The process involves identifying the file to be identified, obtaining identification results, and then performing clustering processing on the identification results to determine the aggregation results, including: The acquired file to be identified is preprocessed to obtain preprocessed data; the preprocessing includes at least one of orientation correction, grayscale processing, and size adjustment; Optical character recognition is performed on the preprocessed data to obtain recognition results with multiple lines of text information; The position information in the text line information is standardized, and the coordinates of the midpoints of the left and right borders and the rotation angle of the borders are calculated. The coordinates of the vertex of the border, the coordinates of the midpoints of the left and right borders, and the rotation angle of the borders are used as position features. A pre-trained language model is used to preprocess the text content in the text line information to obtain text semantic features; The location features and the text semantic features are fused to obtain the fused features; Based on the fusion features, determine the neighborhood map features; Based on the features of the neighbor graph, a graph convolutional network model is constructed; The recognition results are input into the graph convolutional network model to determine the aggregation result.

3. The method according to claim 2, characterized in that, Based on the fusion features, the neighborhood map features are determined, including: Each fused feature is treated as a node, and the total number of text lines corresponding to the fused feature is taken as the total number of nodes. The local density of each node is calculated. The local density is obtained by Euclidean distance between nodes and preset hyperparameters. Based on the local density of each node, and combined with the minimum number of neighbors and the local density ratio coefficient, the number of neighbors K of the node is adaptively adjusted to ensure that each node has at least one neighbor node. Select the K nearest neighbors of each node to form a neighbor set, construct an edge set based on the neighbor set, and determine the features of the neighborhood graph.

4. The method according to claim 3, characterized in that, Based on the features of the neighbor graph, a graph convolutional network model is constructed, including: A multi-layer graph convolutional network is constructed, taking the node features of the neighboring graph features as input, and obtaining the final node embedding of each node through multi-layer feature mapping based on the adjacency matrix with added self-loops, degree matrix, learnable weight matrix and activation function. Calculate the difference in the final node embeddings of any two nodes, and use it as the relation feature between the nodes; The relational features are input into the fully connected layer, the loss value is calculated using the cross-entropy loss function, and the model is trained using the stochastic gradient descent method to obtain the graph convolutional network model.

5. The method according to claim 1, characterized in that, Based on the aggregation result, a full node graph is constructed, and according to the semantic relationships between the nodes in the full node graph, the corresponding tree structure graph is determined, including: Based on the aggregation results, all text blocks obtained after clustering are taken as a set of nodes, and each node is connected to all other nodes except itself to construct a full node graph; Based on the full node graph, a first classification model combining a graph attention network and a multilayer perceptron is constructed; the first classification model is used to output the semantic relationship categories between nodes and their corresponding score weights. Based on the output of the first classification model, edges with semantic relationship categories that have upper and lower semantic relationships are retained, edges with semantic relationship categories that do not have upper and lower semantic relationships are removed, and the full node graph is updated to obtain an initial non-redundant graph. The initial non-redundant graph is traversed using a depth-first search algorithm to detect whether there are loop structures in the graph; If a loop structure is detected, remove the edges in the loop in order of increasing score weight. After each edge removal, re-detect the loop structure until no loop exists in the graph. The tree structure graph is constructed from the final set of remaining nodes and edges.

6. The method according to claim 1, characterized in that, The nodes in the tree structure graph are classified and identified to determine the tree structure data and construct question-answer pairs to represent the semantic relationships of the files to be identified, including: Extract the tree diagram node features of each node in the tree structure diagram; Based on the tree graph node features, a second classification model combining a graph attention network and a multilayer perceptron is constructed; the second classification model is used to output a classification label to determine whether a node is a root node; Based on the tree structure diagram and the root node classification labels output by the second classification model, all root nodes are determined, and for each root node, the corresponding child nodes are retrieved based on the connection relationship between nodes, and the complete tree structure data is extracted. Based on the tree-structured data, all non-root nodes in the graph are traversed. The text content of each node on the path from the root node to the non-root node is concatenated into a question sentence, and the text content of the corresponding child nodes of the non-root node is concatenated into an answer sentence, thus constructing multiple question-answer pairs.

7. The method according to claim 5 or 6, characterized in that, Construct a classification model combining graph attention networks and multilayer perceptrons, including: A multi-layer graph attention network is constructed, introducing a multi-head attention mechanism, with the target node features as the initial input; the target node features are the node features of the full node graph or the tree graph; the feature update of the single-layer graph attention network is realized based on the learnable weight matrix, attention coefficients, and non-linear activation functions, and the attention coefficients are calculated through learnable attention vectors, feature concatenation, and non-linear activation functions; After processing by a multi-layer graph attention network, high-dimensional embedding features of each node are obtained. At the same time, global features are constructed based on the position and text information corresponding to the file to be identified. The global features are input into the multi-layer graph attention network to obtain global high-dimensional embedding features. By fusing the high-dimensional embedding features of any two nodes with the global high-dimensional embedding features, we obtain the relationship feature vector between the nodes; A multilayer perceptron model is constructed, and the relation feature vector is input into the multilayer perceptron model. The score function is calculated through the weight matrix, bias and activation function of each layer. Based on the scoring function, a cross-entropy loss function is constructed, and the loss value of the multilayer perceptron model is calculated based on the cross-entropy loss function. The entire graph attention network and multilayer perceptron model are trained using the stochastic gradient descent method to obtain a classification model.

8. A corpus processing device, characterized in that, include: The first determining module is used to identify the file to be identified, obtain the identification result, and perform clustering processing on the identification result to determine the aggregation result; The first processing module is used to construct a full node graph based on the aggregation result, and determine the tree structure graph corresponding to the aggregation result according to the semantic relationship between the nodes of the full node graph; The second processing module is used to classify and identify the nodes in the tree structure diagram, determine the tree structure data, and construct question-answer pairs to represent the semantic relationships of the files to be identified.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images