Intelligent recognition and extraction method and system for multi-element of large file based on OCR

Abstract

The present application relates to the technical field of intelligent document processing, and particularly relates to a large file multi-element intelligent recognition and extraction method and system based on OCR. The method obtains an initial recognition result of a document through OCR, determines a hierarchical structure representation in combination with spatial position and semantic features, classifies elements and generates type identification and boundary information in accordance with the hierarchical structure representation, and then extracts content to form a structured result, and finally performs integrity checking based on element association relationships and dynamically adjusts the structure representation by using checking feedback to correct the relationships. The present application realizes accurate recognition, structured extraction and adaptive correction of multi-type elements in a complex document, and improves processing accuracy and reliability of the result.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, and in particular to a method and system for intelligent recognition and extraction of multiple elements in large files based on OCR. Background Technology

[0002] In the field of document information processing, optical character recognition (OCR) technology has become a key means of converting paper or image-formatted documents into editable and analyzable text data. With the deepening of enterprise digital transformation, the volume of documents that need to be processed daily is increasing daily, and the document formats and content are becoming increasingly complex, often mixing various document elements such as text paragraphs, tables, charts, stamps, signatures, and formulas. How to accurately and efficiently identify and extract valuable structured information from these complex large files is currently the main challenge facing this technology field.

[0003] Existing conventional processing methods typically rely on pre-defined fixed rules or templates for document analysis and element extraction. These methods first perform OCR processing on the document to obtain the text content and its coordinates. Then, based on the layout characteristics of specific document types, such as title positions and table borders, rules are set to divide regions and identify element types. Another common approach is to utilize traditional machine learning models to perform simple classification of document blocks based on manually designed features.

[0004] However, these conventional approaches have significant drawbacks. Methods relying on fixed rules or templates are highly dependent on the standardization and consistency of document layout, lacking flexibility. Once the document's layout changes, or if an unpredictable format is encountered, the applicability of the rules drops sharply, leading to misidentification or omission of elements, and making it difficult to guarantee the accuracy of the extraction results. While traditional machine learning-based methods possess a degree of adaptability, their reliance on feature engineering makes it difficult to fully capture the deep spatial layout relationships and semantic contextual associations between elements in complex documents. For example, the continuity of a multi-page table, or the correspondence between text and adjacent illustrations, is difficult to accurately determine based on local features alone, easily leading to broken element relationships or misclassification.

[0005] Therefore, existing technologies struggle to achieve robust and accurate multi-element intelligent recognition and structured extraction for large documents with diverse layouts and complex element compositions, and the completeness and accuracy of the extraction results need improvement. This limits the efficiency and reliability of subsequent automated document information processing workflows. Summary of the Invention

[0006] This invention provides a method and system for intelligent recognition and extraction of multiple elements in large files based on OCR, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a method for intelligent recognition and extraction of multiple elements in large files based on OCR, comprising:

[0008] Obtain the target document file to be processed, which contains various types of document elements;

[0009] The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined.

[0010] Based on the spatial relationships and semantic associations in the hierarchical structure representation, the document elements are classified into multiple predefined types, and a type identifier and boundary information are generated for each document element.

[0011] Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results;

[0012] The structured extraction results are validated for completeness based on the relationships between document elements. Validation feedback information is then generated, and the hierarchical structure representation is dynamically adjusted using this feedback information to correct the spatial relationships and semantic associations between document elements.

[0013] The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined, including:

[0014] The target document file is subjected to optical character recognition processing to obtain the original character sequence and the spatial position information and character attribute information of each character unit in the original character sequence;

[0015] Based on the spatial location information and character attribute information, the original character sequence is reconstructed into text lines to generate a preliminary set of text lines as the initial recognition result.

[0016] Semantic feature encoding is performed on the text content of each text line in the initial text line set to obtain the semantic representation vector of the text line;

[0017] A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix;

[0018] Based on the semantic association matrix, hierarchical clustering is performed on the preliminary set of text lines to identify text line clusters with the same hierarchical attributes, and the inclusion and parallel relationships between the text line clusters are determined. The hierarchical structure representation is determined based on the inclusion and parallel relationships.

[0019] A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix, including:

[0020] Based on the boundary information of the text lines in the preliminary text line set, local adjacency edges are established for text line pairs that satisfy the local proximity constraint conditions, thus obtaining the local adjacency relationship layer of the multi-scale spatial relationship graph;

[0021] The page space of the target document file is adaptively divided into multiple functional regions. Pairs of functional regions with cross-regional semantic association characteristics are identified among the multiple functional regions. Global association edges are established for the text lines in the functional region pairs to obtain the global regional relationship layer of the multi-scale spatial relationship graph.

[0022] On the local adjacency layer, the semantic representation vector is updated in the first round of propagation based on the local adjacency edges to generate a local semantic fusion vector;

[0023] On the global region relation layer, the local semantic fusion vector is updated in a second round based on the global association edge to generate a global semantic enhancement vector;

[0024] Calculate the similarity score between the global semantic enhancement vectors of text line pairs in the preliminary text line set, and perform path decay correction on the similarity score by combining the path connectivity of the text line pairs in the multi-scale spatial relationship graph to generate the semantic association matrix.

[0025] Based on the spatial relationships and semantic associations in the hierarchical structure representation, element types are classified, dividing the document elements into multiple predefined types, and generating type identifiers and boundary information for each document element, including:

[0026] The hierarchical depth information, parent node index information, and child node index information of the document elements are extracted from the hierarchical structure representation, and the spatial relationship features and semantic association features between the document elements are also extracted.

[0027] The context dependency path of a document element is determined based on the hierarchical depth information and the parent node index information, and the inherited semantic features are obtained based on the context dependency path.

[0028] The internal structural pattern of the document element is determined based on the child node index information and spatial relationship characteristics.

[0029] The inherited semantic features are fused with the internal composition structure pattern to generate a comprehensive classification feature vector for the document elements;

[0030] Based on the comprehensive classification feature vector, the document element is type-determined, mapped to one of the multiple predefined types, and a corresponding type identifier is assigned to the document element.

[0031] The boundary information of the document element is calculated based on the boundary determination rules corresponding to the type identifier and the child node index information of the document element in the hierarchical structure representation.

[0032] The context dependency path of a document element is determined based on the hierarchical depth information and parent node index information, and the internal composition structure pattern of the document element is determined based on the child node index information and spatial relationship features, including:

[0033] Based on the hierarchical depth information and the parent node index information, a reverse hierarchical tracing is performed, traversing upwards from the document element along the parent node pointer until the root node is reached, to obtain the context dependency path. The context dependency path records all ancestor nodes of the document element in an ordered node sequence.

[0034] For each ancestor node in the context-dependent path, the semantic association features and hierarchical position encoding of the ancestor node are extracted, and hierarchical decay weights are assigned according to the relative distance of the ancestor node in the context-dependent path;

[0035] The inherited semantic features are generated by weighted aggregation of the semantic association features of all ancestor nodes in the context-dependent path;

[0036] Based on the spatial relationship characteristics of the child nodes, the relative positional relationship matrix between the child nodes is calculated, and the structural pattern mining is performed on the relative positional relationship matrix to obtain the regular arrangement characteristics of the spatial layout of the child nodes.

[0037] Based on the regularity of the arrangement and the number of child nodes in the child node index information, a structural pattern encoding vector is generated to obtain the internal composition structural pattern.

[0038] Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results, including:

[0039] Based on the type identifier, determine the extraction rule corresponding to the type identifier from a predefined set of type extraction rules;

[0040] Based on the boundary information, the original text content and original visual features corresponding to the document elements are determined from the target document file.

[0041] The original text content is parsed according to the extraction rules to determine the semantic relationships between semantic units in the original text content, and the semantic units are mapped to fields based on the structured template corresponding to the type identifier to generate a structured content representation of the document element;

[0042] Based on the type identifier and the boundary information, when the document element contains non-text sub-elements, the non-text sub-elements are identified and their positions are marked according to the original visual features, and supplementary descriptive information of the non-text sub-elements is generated.

[0043] The structured content representation is associated and bound with the attached descriptive information, and according to the hierarchical and sequential relationships of the document elements in the hierarchical structure representation, the structured content representation and attached descriptive information of all document elements are organized into a tree data structure to generate the structured extraction result.

[0044] Based on the relationships between document elements in the structured extraction results, the integrity of the structured extraction results is verified, and verification feedback information is generated. This feedback information is then used to dynamically adjust the hierarchical structure representation to correct the spatial relationships and semantic associations between document elements, including:

[0045] The relationships between document elements are extracted from the structured extraction results, and the integrity of the relationships is checked based on these relationships to obtain the verification feedback information.

[0046] Based on the expected location information and missing type information in the verification feedback information, the candidate text region corresponding to the expected location information is located in the initial recognition result;

[0047] The candidate text region is verified based on the association context information between the missing type information and the verification feedback information.

[0048] For candidate text regions that pass the association verification, the spatial relationship features and semantic association features between the candidate text regions and existing document elements in the hierarchical structure representation are recalculated based on the association context information.

[0049] The recalculated spatial relationship features and semantic association features are updated in the hierarchical structure representation, and supplementary document element nodes corresponding to the candidate text regions are inserted into the hierarchical structure representation to correct the spatial relationships and semantic associations between the document elements.

[0050] A second aspect of this invention provides a large-file multi-element intelligent recognition and extraction system based on OCR, comprising:

[0051] The document acquisition unit is used to acquire the target document file to be processed, which contains various types of document elements.

[0052] The structure recognition unit is used to perform optical character recognition processing on the target document file to obtain an initial recognition result, and to determine the hierarchical structure representation of the document elements based on the spatial location information and semantic features of the text content in the initial recognition result.

[0053] The element classification unit is used to classify element types according to the spatial relationships and semantic associations in the hierarchical structure representation, divide the document elements into multiple predefined types, and generate type identifiers and boundary information for each document element.

[0054] The content extraction unit is used to extract the corresponding document element content from the target document file according to the type identifier and boundary information, and generate a structured extraction result;

[0055] The verification and adjustment unit is used to perform integrity verification on the structured extraction results based on the relationship between document elements in the structured extraction results, generate verification feedback information, and use the verification feedback information to dynamically adjust the hierarchical structure representation in order to correct the spatial relationship and semantic association between document elements.

[0056] A third aspect of the present invention provides an electronic device, comprising:

[0057] processor;

[0058] Memory used to store processor-executable instructions;

[0059] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0061] This invention can efficiently process complex document files containing multiple element types. By combining optical character recognition (OCR) with intelligent analysis, it achieves high-precision structured extraction of document content. This method not only identifies text content but also deeply analyzes the document's layout and semantic information, thereby accurately reconstructing the document's hierarchical logical structure and laying a solid foundation for subsequent information classification and extraction.

[0062] Based on hierarchical structural representation, this invention can perform refined classification of document elements according to spatial proximity and semantic coherence, accurately placing them into a predefined type system. Each element is assigned a clear type identifier and precise bounding box information, ensuring the accuracy of element positioning and content segmentation, and effectively avoiding the common problems of element misjudgment and boundary ambiguity in traditional methods.

[0063] By using type identifiers and boundary information, this invention can accurately extract the content of various target elements from the original document and automatically assemble them into a well-formatted, structured data result. This process achieves automated conversion from unstructured or semi-structured documents to machine-readable and processable data, greatly improving data utilization efficiency.

[0064] This invention introduces an integrity verification mechanism based on association relationships, which can automatically detect logical omissions or association errors in the extraction results. Utilizing the generated verification feedback information, the understanding of the document structure can be dynamically optimized and adjusted, correcting the spatial and semantic association model between elements, thereby forming a self-improving closed-loop processing flow. This continuously enhances the robustness of processing documents of different formats and complex layouts, and the reliability of the final extraction results. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the OCR-based intelligent recognition and extraction method for large files with multiple elements according to an embodiment of the present invention.

[0066] Figure 2 This is a flowchart illustrating the process of determining the boundary information of document elements according to an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0069] Figure 1 This is a flowchart illustrating the OCR-based intelligent recognition and extraction method for large files according to an embodiment of the present invention. Figure 1 As shown, the OCR-based intelligent recognition and extraction method for multiple elements in large files includes:

[0070] Obtain the target document file to be processed, which contains various types of document elements;

[0071] The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined.

[0072] Based on the spatial relationships and semantic associations in the hierarchical structure representation, the document elements are classified into multiple predefined types, and a type identifier and boundary information are generated for each document element.

[0073] Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results;

[0074] The structured extraction results are validated for completeness based on the relationships between document elements. Validation feedback information is then generated, and the hierarchical structure representation is dynamically adjusted using this feedback information to correct the spatial relationships and semantic associations between document elements.

[0075] The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined, including:

[0076] The target document file is subjected to optical character recognition processing to obtain the original character sequence and the spatial position information and character attribute information of each character unit in the original character sequence;

[0077] Based on the spatial location information and character attribute information, the original character sequence is reconstructed into text lines to generate a preliminary set of text lines as the initial recognition result.

[0078] Semantic feature encoding is performed on the text content of each text line in the initial text line set to obtain the semantic representation vector of the text line;

[0079] A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix;

[0080] Based on the semantic association matrix, hierarchical clustering is performed on the preliminary set of text lines to identify text line clusters with the same hierarchical attributes, and the inclusion and parallel relationships between the text line clusters are determined. The hierarchical structure representation is determined based on the inclusion and parallel relationships.

[0081] Before performing optical character recognition (OCR) on the target document file, the document image is preprocessed to improve recognition accuracy. Preprocessing operations include image denoising, tilt correction, and contrast enhancement. An adaptive threshold segmentation algorithm converts the document image into a binary image, making character edges clearer. After preprocessing, an OCR model based on a deep learning framework is used to scan and recognize the document image region by region. This model can simultaneously output the character content, the character's coordinate position in the image, and the character's font attribute information.

[0082] The optical character recognition (OCR) model divides the document image into multiple recognition units during processing, each corresponding to a character unit. The spatial location information of a character unit is represented by four coordinate values, recording the coordinates of the top-left and bottom-right corners of the character's bounding rectangle. These coordinate values ​​are labeled in pixels within the document image's coordinate system. Character attribute information includes descriptive parameters across multiple dimensions, such as font type, font size, font weight, whether italic, and character color. By traversing all regions of the entire document image, an original character sequence containing all character units is obtained. This sequence is arranged in a scanning order from left to right and top to bottom, with each character unit carrying its corresponding spatial location and character attribute information.

[0083] After obtaining the original character sequence, the scattered character units need to be organized into text lines with semantic integrity. The text line reconstruction process is based on the spatial relationship between character units, determining whether characters belong to the same text line by calculating the difference in vertical coordinates and the horizontal spacing between adjacent character units. Specifically, when the difference in the vertical center position of two character units is less than a preset line height threshold, and the horizontal spacing does not exceed three times the average width of the characters, these two character units are determined to belong to the same text line. A density-based clustering algorithm is used to scan the original character sequence, aggregating character units that meet the same-line condition together to form a text line object. The text line object not only contains the content of all characters in the line, but also records the overall bounding box coordinates, average font size, and dominant font attributes of the line. Through traversing and clustering the original character sequence one by one, a preliminary text line set containing multiple text line objects is generated, which serves as the basic data for subsequent semantic analysis.

[0084] To capture the semantic information of text lines, deep semantic encoding is performed on the text content of each line in the initial text line set. A pre-trained language model, trained on a large-scale text corpus, is used to vectorize the text line content, mapping natural language text to a high-dimensional semantic space. The string content of each text line is fed into the language model, which extracts the contextual semantic information of the text through a multi-layer transformation network, outputting a fixed-dimensional vector representation. This vector is the semantic representation vector of the text line, where the values ​​of different dimensions reflect the feature strength of the text in different semantic directions. By performing the same encoding operation on all text lines in the initial text line set, a set of semantic representation vectors equal to the number of text lines is obtained. The dimension of each vector is typically set to 512 or 768 dimensions to ensure sufficient expression of semantic information.

[0085] A multi-scale spatial relationship graph is constructed to characterize the spatial topology between text lines. Each text line in the initial set of text lines is considered a node in the graph structure, and nodes are connected by edges to represent spatial proximity. The multi-scale spatial relationship graph contains three levels of edge connections: the first level is adjacent row edges, connecting text line nodes that are adjacent in the vertical coordinate; the second level is column alignment edges, connecting text line nodes that are left-aligned or right-aligned in the horizontal coordinate; and the third level is region proximity edges, connecting text line nodes that are within the same local region in the two-dimensional plane. Each edge is assigned a weight value, which is calculated based on the spatial distance between the corresponding text lines of the two nodes, the similarity of font attributes, and the degree of overlap of the bounding boxes. The closer the text line nodes are, the more similar their attributes are, and the higher their overlap, the greater the weight of the edge, indicating a stronger spatial relationship between them.

[0086] Semantic propagation computation is performed on a multi-scale spatial relationship graph. Information interaction between nodes is achieved through a graph neural network. Each node's initial features are set as the semantic representation vector of the corresponding text line. The graph neural network employs a message-passing mechanism, where nodes send messages to their neighbors via edges and receive messages from neighbors. Messages are calculated by weighting the sending node's feature vector with edge weights. Receiving nodes aggregate all received messages; common aggregation functions include summation, averaging, or maximization. After aggregating neighbor messages, nodes fuse the aggregation result with their own features to update the node's feature representation, generating a new node feature representation through a nonlinear transformation function. This process is iterated multiple times, with each iteration incorporating the semantic information of other nodes within a larger neighborhood into the node's feature representation. After iterative propagation computation, the originally isolated text line semantic representation vectors gain spatial context enhancement, and the vector representations of adjacent or semantically related text lines are closer in the semantic space.

[0087] The semantic association matrix is ​​calculated based on the updated node feature vectors. The rows and columns of the matrix correspond to text lines in the initial text line set, and the numerical values ​​of the matrix elements represent the semantic association strength between two corresponding text lines. The association strength is obtained by calculating the cosine similarity of the updated semantic representation vectors of the two text lines, with a value ranging from 0 to 1. A higher value indicates a stronger semantic correlation between the two text lines. The semantic association matrix not only reflects the semantic similarity of the text content but also implicitly includes location proximity information obtained through spatial relationship graph propagation, achieving a fusion of semantic and spatial features.

[0088] A hierarchical clustering operation is performed on the initial set of text lines using a semantic association matrix. A bottom-up agglomerative hierarchical clustering algorithm is employed. Initially, each text line is treated as an independent cluster. Then, based on the similarity values ​​in the semantic association matrix, the clusters with the highest similarity are gradually merged. The merging operation continues until a preset stopping condition is met. The stopping condition can be set to the number of clusters reaching a desired value, or the similarity between clusters falling below a threshold. During the clustering process, text lines belonging to the same cluster are identified as text line clusters with the same hierarchical attributes. These text lines occupy the same hierarchical position in the document structure, such as paragraphs belonging to the same chapter, or cell content belonging to the same table.

[0089] Determining the inclusion and parallel relationships between text line clusters requires combining spatial location information and clustering results. Inclusion relationships are identified by whether the bounding box of one text line cluster completely encloses the bounding box of another. When the bounding box of cluster A covers the bounding box of cluster B in both the horizontal and vertical coordinate ranges, cluster A is considered to contain cluster B. Cluster A corresponds to a higher-level document element, and cluster B is its child element. Parallel relationships are identified by whether multiple text line clusters share the same parent cluster and are arranged horizontally or vertically in space. When multiple clusters are contained by the same higher-level cluster, and there is no inclusion relationship between the bounding boxes of these clusters, they are considered to be in a parallel relationship, corresponding to sibling elements at the same level in the document.

[0090] Based on the identified inclusion and parallel relationships, a hierarchical structural representation is constructed. This representation adopts a tree structure, where the root node represents the entire document, the first-level child nodes correspond to the top-level structural units of the document such as chapters or headings, and each node can further expand to include child nodes representing more granular document elements. The depth of the node hierarchy reflects the nesting level of document elements in the document structure. The parent-child connection relationship between nodes represents the inclusion relationship, and multiple child nodes under the same parent node represent the parallel relationship. Each node in the tree structure stores all information of the corresponding text line cluster, including the content of all text lines within the cluster, bounding box coordinates, semantic representation vectors, and similarity statistics from the clustering process. The hierarchical structural representation comprehensively depicts the organizational relationships between document elements, providing a structured data foundation for subsequent element type classification and content extraction.

[0091] A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix, including:

[0092] Based on the boundary information of the text lines in the preliminary text line set, local adjacency edges are established for text line pairs that satisfy the local proximity constraint conditions, thus obtaining the local adjacency relationship layer of the multi-scale spatial relationship graph;

[0093] The page space of the target document file is adaptively divided into multiple functional regions. Pairs of functional regions with cross-regional semantic association characteristics are identified among the multiple functional regions. Global association edges are established for the text lines in the functional region pairs to obtain the global regional relationship layer of the multi-scale spatial relationship graph.

[0094] On the local adjacency layer, the semantic representation vector is updated in the first round of propagation based on the local adjacency edges to generate a local semantic fusion vector;

[0095] On the global region relation layer, the local semantic fusion vector is updated in a second round based on the global association edge to generate a global semantic enhancement vector;

[0096] Calculate the similarity score between the global semantic enhancement vectors of text line pairs in the preliminary text line set, and perform path decay correction on the similarity score by combining the path connectivity of the text line pairs in the multi-scale spatial relationship graph to generate the semantic association matrix.

[0097] In the process of constructing a hierarchical structural representation of document elements, multi-scale spatial relationship modeling and semantic propagation computation of the initial text line set are key steps in accurately depicting the complex semantic relationships between document elements. This process requires comprehensive consideration of local proximity relationships and global regional relationships, and the effective fusion of multi-level semantic information through semantic propagation mechanisms on graph structures.

[0098] When constructing a multi-scale spatial relationship graph, the spatial positional relationships of each text line in the initial text line set are first analyzed in detail. For each text line, the coordinates of its four corner points and center point are extracted as the basis for subsequent spatial relationship calculations. In establishing the local adjacency relationship layer, local proximity constraints need to be defined to filter text line pairs with potential semantic associations. These constraints include horizontal distance thresholds, vertical distance thresholds, and projection overlap requirements. For text line i and text line j, the Euclidean distance between their center points is calculated. Simultaneously calculate the projection overlap ratio of two text lines in the horizontal and vertical directions. and When Euclidean distance Less than the preset neighbor threshold And the horizontal projection overlap ratio $ is greater than the preset overlap threshold If text line i and text line j satisfy the local proximity constraint, a local adjacency edge is established for this text line pair in the multi-scale spatial relationship graph. By performing pairwise judgments on all text line pairs in the initial text line set, the construction of the local adjacency relationship layer is completed. This layer mainly captures the local spatial dependencies between adjacent text lines in the document.

[0099] In constructing the global regional relationship layer, adaptive regional division of the target document file's page space is the primary task. Based on the density distribution characteristics of text lines, a spatial clustering method is used to divide the page space into multiple functional regions. Specifically, the density distribution of text line center points at various locations within the page space is statistically analyzed. A density peak detection algorithm is used to identify the core locations of dense text regions, and these core locations are used as seed points for region expansion, grouping spatially continuous text lines with similar densities into the same functional region. Each functional region corresponds to a relatively independent semantic unit in the document, such as a table region, paragraph region, or heading region. After obtaining multiple functional regions, it is necessary to further identify functional region pairs with cross-regional semantic association characteristics. This cross-regional semantic association is usually reflected in the inherent relationship between specific document element types, such as the association between table titles and table content, or the association between chapter titles and body paragraphs. By analyzing the text content characteristics and spatial location patterns of each functional region, logically related functional region pairs are identified. For the identified functional region pairs, global association edges are established between the text lines they contain. These edges can span large spatial distances and connect text lines that are semantically dependent but spatially separated, thereby forming a global regional relationship layer of a multi-scale spatial relationship graph.

[0100] After constructing the multi-scale spatial relationship graph, the semantic representation vectors of text lines are updated through multiple rounds of propagation based on this graph structure. In the first round of propagation, the focus is on the local adjacency layer, using local adjacency edges to achieve semantic information exchange between adjacent text lines. This involves considering text line i and the set of all neighboring text lines connected by local adjacency edges. A weighted aggregation strategy is employed to update the semantic representation of text line i. Specifically, the initial semantic representation vector of text line i is calculated as a weighted sum of the semantic representation vectors of its neighboring text lines. The weights are determined by the inverse of the spatial distance between the edges, with closer text lines contributing more to semantic propagation. By performing this aggregation operation in parallel on all text lines in the initial set, a local semantic fusion vector is generated. This vector integrates the semantic information of the text line itself as well as the semantic information of the text lines in its spatial neighborhood, enhancing the ability to model the local semantic context.

[0101] In the second round of propagation update, the system switches to the global region relation layer and uses global association edges to perform cross-region semantic enhancement. The local semantic fusion vector obtained in the first round of propagation is used as input for text line i and its set of remote associated text lines connected by global association edges. Similarly, a weighted aggregation mechanism is used for semantic updates. The difference lies in the design of the global relational edge weights, which needs to consider the semantic correlation strength between functional regions, assigning greater propagation weights to text lines from highly related functional regions. Through semantic propagation at the global region relation layer, the semantic representation of a text line can incorporate cross-regional contextual information, generating a global semantic enhancement vector. This vector not only contains the semantic features of the local neighborhood but also integrates the semantic information of distant text lines that are logically related to the current text line, thus providing a more comprehensive semantic characterization.

[0102] After obtaining the global semantic enhancement vector for each text line in the initial text line set, a semantic association matrix between text line pairs is constructed. For any text line pair (i,j), its global semantic enhancement vector is first calculated. and The cosine similarity between the two text lines is used to obtain a preliminary similarity score. This similarity reflects the proximity of the two text lines in the deep semantic space. To further incorporate spatial structural information, a path decay correction mechanism based on path connectivity is introduced. In the multi-scale spatial relationship graph, the shortest path length from text line i to text line j is calculated. Path length reflects the connectivity between two lines of text in a graph structure. The path decay factor is defined as... ,in The path decay rate parameter is used. The initial similarity score is multiplied by the path decay factor to obtain the final similarity score after path decay correction. This correction process ensures that, based on semantic similarity, text line pairs that are more spatially connected receive higher association scores, while spatially isolated text line pairs or those requiring multiple hops to reach are appropriately suppressed. This calculation is performed on all text line pairs in the initial text line set, and the resulting similarity scores are filled into the corresponding positions in the matrix, ultimately generating a complete semantic association matrix. This matrix integrates semantic similarity and spatial structure constraints, providing accurate semantic association basis for subsequent element type classification and boundary determination, effectively supporting the accurate parsing of complex document structures.

[0103] Figure 2 This is a schematic diagram illustrating the process of determining the boundary information of document elements according to an embodiment of the present invention. Figure 2 As shown, element types are classified according to the spatial relationships and semantic associations in the hierarchical structure representation. The document elements are divided into multiple predefined types, and type identifiers and boundary information are generated for each document element, including:

[0104] The hierarchical depth information, parent node index information, and child node index information of the document elements are extracted from the hierarchical structure representation, and the spatial relationship features and semantic association features between the document elements are also extracted.

[0105] The context dependency path of a document element is determined based on the hierarchical depth information and the parent node index information, and the inherited semantic features are obtained based on the context dependency path.

[0106] The internal structural pattern of the document element is determined based on the child node index information and spatial relationship characteristics.

[0107] The inherited semantic features are fused with the internal composition structure pattern to generate a comprehensive classification feature vector for the document elements;

[0108] Based on the comprehensive classification feature vector, the document element is type-determined, mapped to one of the multiple predefined types, and a corresponding type identifier is assigned to the document element.

[0109] The boundary information of the document element is calculated based on the boundary determination rules corresponding to the type identifier and the child node index information of the document element in the hierarchical structure representation.

[0110] Before classifying document elements, it is necessary to extract various feature information from the constructed hierarchical structure representation. For each document element node, its hierarchical depth information in the tree structure is extracted. This depth value reflects the nesting level of the element in the document organization structure. For example, top-level headings are usually located at a shallower level, while specific content items within paragraphs are located at a deeper level. Parent node index information records the position index of the current element's direct parent element in the structural representation. This index allows tracing back to the type and attributes of the parent element. Child node index information contains the set of position indices of all subordinate elements contained in the current element, used to characterize the internal composition relationship of the element. Spatial relationship feature extraction covers the relative positional relationship between adjacent elements, including horizontal left-right arrangement, vertical up-down arrangement, and the spacing between elements. Semantic association feature extraction is based on the lexical composition and syntactic structure of the text content. The text content of the elements is encoded using a pre-trained language model to obtain semantic vector representations. The similarity of semantic vectors between different elements is calculated as a semantic association metric.

[0111] The process of determining the context-dependent path starts from the current document element and traces upwards level by level along the parent node index information to the root node, forming a complete path sequence from the root node to the current element. Each node in this path sequence represents a level of document element, and the type identifiers and semantic information of these elements constitute the context of the current element. The acquisition of inherited semantic features employs a path aggregation mechanism, which weights and sums the semantic vectors of each node in the path according to their hierarchical weights. Specifically, suppose there are L nodes in the path, and the semantic vector of the i-th node is... Then inherit semantic features Calculated as ,in Let be the weight coefficient of the i-th node. The closer the node is to the current element, the greater its weight. The weight coefficient satisfies the following condition: This inherited semantic feature can capture the semantic influence of the current element on the parent element. For example, if a data item is located under a column in a financial table, its inherited semantic feature will include the financial account information expressed by the column header.

[0112] The determination of the internal composition structure pattern is based on the joint analysis of child node index information and spatial relationship characteristics. First, the number of child nodes contained in the current element is counted. If the number of child nodes is zero, it is determined to be a leaf node, and its internal composition structure pattern is marked as atomic. If it contains multiple child nodes, the spatial arrangement of these child nodes is analyzed. By calculating the bounding box coordinates of the child nodes, it is determined whether the child nodes are arranged linearly in the horizontal direction, linearly in the vertical direction, or in a grid-like distribution. For horizontal linear arrangement, the internal composition structure pattern is marked as horizontal list type; for vertical linear arrangement, it is marked as vertical list type; for grid-like distribution, it is marked as table type. Simultaneously, the uniformity of spacing between child nodes is examined, and the variance of the spacing between adjacent child nodes is calculated. If the variance is small, the child node arrangement is considered regular; if the variance is large, the arrangement is considered irregular. The internal composition structure pattern also includes the distribution statistics of child node types, recording the frequency of occurrence of each type of child node to form a child node type distribution histogram. This histogram serves as supplementary descriptive information for the structure pattern.

[0113] The feature fusion stage integrates inherited semantic features with internal structural patterns to form a comprehensive classification feature vector. Inherited semantic features are already represented as semantic vectors. The internal structural pattern needs to be converted into a numerical vector representation. The vectorization process of the structural pattern includes one-hot encoding of the pattern type label and concatenating numerical features such as the number of child nodes, spacing variance, and the histogram of child node type distribution to obtain the structural feature vector. Comprehensive classification feature vector It is obtained through splicing operations, that is The square brackets indicate vector concatenation. To enhance the discriminative power of features, cross features can be introduced, generating interaction vectors by element-wise multiplying inherited semantic features and structural features. ,in This represents element-wise multiplication, and the final comprehensive classification feature vector is expanded as follows: .

[0114] The type determination process employs a multi-classification model to map the comprehensive classification feature vector. The predefined type set includes various document element types such as headings, paragraphs, list items, table cells, image captions, and footnotes. The multi-classification model uses a fully connected neural network structure. The input layer receives the comprehensive classification feature vector, and after multiple non-linear transformations, the output layer generates the probability distribution for each predefined type. Let the total number of predefined types be K, and the probability vector generated by the output layer be... ,in This represents the probability that the current element belongs to the j-th class, satisfying... The type with the highest probability value is selected as the judgment result, i.e. ,in The type index is used for classification. A type identifier is assigned to each document element in string format; for example, "HEADI7G" is for headings, "PARAGRAPH" for paragraphs, and "TABLE_CELL" for table cells. The classification model is trained on manually labeled document samples. The model parameters are optimized using the cross-entropy loss function to make the probability distribution of the model's output approximate the one-hot encoded representation of the true type.

[0115] Boundary information is calculated based on the boundary determination rules corresponding to the type identifier and the child node index information. Different document elements follow different boundary determination rules. For atomic elements, such as a single text line or a single image, their boundary information is directly obtained from the OCR recognition bounding box coordinates, including the coordinates of the top-left corner. and the coordinates of the bottom right corner For composite elements, such as lists containing multiple child items or tables containing multiple cells, their boundaries need to be obtained by aggregating the boundary information of child nodes. Specifically, the calculation process involves obtaining the set of bounding box coordinates of all child nodes based on their index information. Where M is the total number of child nodes. The boundary of a composite element is calculated as the envelope rectangle of the boundaries of all child nodes, i.e. , , , Certain element boundaries require expansion processing. For example, paragraph elements have their boundaries expanded vertically upwards and downwards by a fixed number of pixels to accommodate paragraph spacing; table elements have their boundaries expanded around their edges by a fixed number of pixels to include table borders. In addition to the four coordinates of the rectangle, boundary information may also include rotation angle information to handle tilted documents. The rotation angle is estimated by fitting the direction of the text lines within the element. All document element type identifiers and boundary information are stored in a structured format, forming element type annotation data. This data provides precise location data for subsequent content extraction steps.

[0116] The context dependency path of a document element is determined based on the hierarchical depth information and parent node index information, and the internal composition structure pattern of the document element is determined based on the child node index information and spatial relationship features, including:

[0117] Based on the hierarchical depth information and the parent node index information, a reverse hierarchical tracing is performed, traversing upwards from the document element along the parent node pointer until the root node is reached, to obtain the context dependency path. The context dependency path records all ancestor nodes of the document element in an ordered node sequence.

[0118] For each ancestor node in the context-dependent path, the semantic association features and hierarchical position encoding of the ancestor node are extracted, and hierarchical decay weights are assigned according to the relative distance of the ancestor node in the context-dependent path;

[0119] The inherited semantic features are generated by weighted aggregation of the semantic association features of all ancestor nodes in the context-dependent path;

[0120] Based on the spatial relationship characteristics of the child nodes, the relative positional relationship matrix between the child nodes is calculated, and the structural pattern mining is performed on the relative positional relationship matrix to obtain the regular arrangement characteristics of the spatial layout of the child nodes.

[0121] Based on the regularity of the arrangement and the number of child nodes in the child node index information, a structural pattern encoding vector is generated to obtain the internal composition structural pattern.

[0122] After acquiring the target document file to be processed, an initial recognition result is obtained through optical character recognition (OCR). This initial recognition result contains the spatial location information and text content of each document element. During the process of determining the hierarchical structure representation of the document elements, each document element is assigned hierarchical depth information, parent node index information, and child node index information. To further clarify the position of document elements within the entire document structure and their relationships with other elements, it is necessary to determine the contextual dependency path and internal composition structure pattern of the document elements based on this index information.

[0123] For any document element, reverse hierarchical tracing is performed using its level depth information and parent node index information. Specifically, starting from the current document element, its direct parent node is located through the parent node index information, and then the process continues to traverse upwards along the parent node pointer, visiting the parent nodes' parent nodes in turn, until the root node is reached. During the traversal, all nodes visited are recorded in the order of access, forming an ordered node sequence, which is the context dependency path. For example, if a paragraph element has a level depth of 3, its parent node index information indicates that its parent node is a chapter element, and the parent node of the chapter element is the document root node. Therefore, the context dependency path of this paragraph element is "root node - chapter element - paragraph element". This path records the complete hierarchical relationship from the root node to the current element, reflecting the element's location in the document hierarchy.

[0124] After obtaining the context-dependent path, feature extraction is performed on each ancestor node in the path. For each ancestor node, its semantic association features are extracted. These semantic association features can be semantic vectors obtained by encoding the text content of the ancestor node through a natural language processing model, or type feature vectors generated based on the ancestor node type identifier. Simultaneously, the hierarchical position encoding of the ancestor nodes is extracted. This encoding reflects the depth information of the ancestor nodes in the document hierarchy. The hierarchical position encoding can employ a sine / cosine function encoding method to map the hierarchical depth values ​​into a high-dimensional vector representation.

[0125] After extracting the semantic association features and hierarchical position encoding of ancestor nodes, hierarchical decay weights are assigned based on the relative distance between ancestor nodes in the context-dependent path. The relative distance is defined as the hierarchical difference between the current document element and its ancestor nodes. Ancestor nodes closer to the current element have a greater semantic influence and should be assigned higher weights; conversely, ancestor nodes farther away have a gradually diminishing semantic influence and should be assigned lower weights. The hierarchical decay weights can be calculated using an exponential decay function, where the hierarchical depth of the current element is given by... The ancestor node's hierarchy depth is Then the level decay weight of the ancestor node It can be represented as ,in This is the decay coefficient, which controls the rate at which the weight decreases as distance increases. In this way, the parent node one level away from the current element has the highest weight, the grandparent node two levels away has the next highest weight, and so on.

[0126] After obtaining the semantic association features and hierarchical decay weights of each ancestor node, the semantic association features of all ancestor nodes in the context-dependent path are weighted and aggregated to generate inherited semantic features. Specifically, suppose there are K ancestor nodes in the context-dependent path, and the semantic association feature vector of the k-th ancestor node is... The corresponding hierarchical decay weight is Then inherit semantic features It can be obtained by weighted summation, that is To ensure numerical stability, the weights can be normalized, i.e. This inherited semantic feature integrates the semantic information of all ancestor nodes of the current document element and is weighted according to hierarchical distance, which can effectively reflect the background information of the current element in the semantic hierarchy of the document.

[0127] In addition to determining context-dependent paths and inherited semantic features, it is also necessary to determine the internal structural pattern of document elements, which describes the spatial layout relationships between the child nodes contained in the current document element. First, a list of all child nodes of the current document element is obtained based on the child node index information. For each child node, its spatial relationship features are obtained, including the spatial position information of the child node, such as the coordinates of the top-left corner of the bounding box, its width, and its height.

[0128] Utilizing the spatial relationship features of all child nodes, calculate the relative positional relationship matrix between child nodes. Let the current document element contain M child nodes, and the bounding box center coordinates of the i-th child node be... The relative positional relationship between the i-th child node and the j-th child node can be represented by the horizontal offset. With vertical offset This represents the process of integrating the relative positional relationships between all child node pairs to form a relative positional relationship matrix. The element in the i-th row and j-th column of the matrix is .

[0129] Structural pattern mining is performed on the relative position relationship matrix to discover the regularity of the spatial layout of child nodes. This regularity can be obtained through statistical analysis of the relative position distribution between child nodes. For example, if most child nodes have approximately equal intervals in the horizontal direction, it indicates a horizontally uniform arrangement; if most child nodes have approximately equal intervals in the vertical direction, it indicates a vertically uniform arrangement; and if child nodes exhibit regular intervals in both the horizontal and vertical directions, it indicates a grid-like arrangement. Specifically, cluster analysis can be performed on the horizontal and vertical offset sets in the relative position relationship matrix to identify the most frequently occurring offset values. If the frequency of a certain offset value exceeds a preset threshold, it is considered that a regular arrangement exists in that direction.

[0130] Based on the regularity of the arrangement and the number of child nodes in the child node index information, a structural pattern encoding vector is generated. This vector comprehensively reflects the arrangement and quantity of child nodes. For example, a multi-dimensional vector can be defined, where each dimension represents the degree of horizontal arrangement, vertical arrangement, grid arrangement, and the number of child nodes. The degree of horizontal arrangement can be obtained by calculating the proportion of regular offsets in the horizontal direction, and the degree of vertical arrangement is obtained similarly. The number of child nodes can be directly obtained from the child node index information or converted into numerical features through normalization. These features are then concatenated to obtain the structural pattern encoding vector. This encoded vector is a numerical representation of the internal structural pattern, and can serve as an important feature input for subsequent element type classification and extraction.

[0131] Through the above process, both the contextual dependency path and inherited semantic features of document elements were determined, as well as the internal compositional structure pattern of document elements. The contextual dependency path reflects the position and semantic background of document elements in the hierarchical structure, while the internal compositional structure pattern reflects the spatial organization rules of the sub-elements within the document element. These two aspects of information together constitute the core components of the hierarchical structural representation of document elements, providing sufficient feature basis for subsequent element type classification based on spatial relationships and semantic associations.

[0132] Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results, including:

[0133] Based on the type identifier, determine the extraction rule corresponding to the type identifier from a predefined set of type extraction rules;

[0134] Based on the boundary information, the original text content and original visual features corresponding to the document elements are determined from the target document file.

[0135] The original text content is parsed according to the extraction rules to determine the semantic relationships between semantic units in the original text content, and the semantic units are mapped to fields based on the structured template corresponding to the type identifier to generate a structured content representation of the document element;

[0136] Based on the type identifier and the boundary information, when the document element contains non-text sub-elements, the non-text sub-elements are identified and their positions are marked according to the original visual features, and supplementary descriptive information of the non-text sub-elements is generated.

[0137] The structured content representation is associated and bound with the attached descriptive information, and according to the hierarchical and sequential relationships of the document elements in the hierarchical structure representation, the structured content representation and attached descriptive information of all document elements are organized into a tree data structure to generate the structured extraction result.

[0138] After obtaining the type identifier and boundary information of each document element, the corresponding content needs to be accurately extracted from the target document file and organized in a structured manner based on this identifier information. For different types of document elements, a predefined set of type extraction rules stores the corresponding extraction rules. These extraction rules define the parsing mode, field mapping relationships, and special processing requirements for each type of element. During processing, based on the type identifier of the document element, the corresponding extraction rule is retrieved from this rule set. This extraction rule includes information such as regular expression templates, semantic annotation specifications, and field correspondences, which guide subsequent content parsing and structured transformation.

[0139] Boundary information is used to locate the specific extent of document elements in a target document file. This boundary information typically includes spatial parameters such as the element's starting coordinates, width, and height within the document. Based on these spatial parameters, the area occupied by the element can be precisely delineated, and the original text content within that area can be extracted from the initial recognition results. The original text content retains the text information output by optical character recognition (OCR), including attributes such as text sequence, character-level coordinates, and font style. Simultaneously, original visual features are obtained from this area. These visual features encompass non-textual features such as color distribution, line shape, and image texture, providing a basis for subsequent recognition of non-textual sub-elements.

[0140] For the extracted raw text content, content parsing is performed according to the corresponding extraction rules. This parsing process decomposes the raw text content into several semantic units, each representing an information fragment with independent semantic meaning. For example, in a table-type document element, the content of each cell can be considered a semantic unit; in a paragraph-type document element, each sentence or phrase can be considered a semantic unit. Natural language processing techniques are used to analyze the semantic relationships between semantic units, including parallel, subordinate, and causal relationships. After determining the semantic units and their relationships, field mapping is performed based on the structured template corresponding to the type identifier. The structured template predefines the field names and field types that a document element of this type should contain, and the mapping rules assign semantic units to the corresponding fields. For example, for an invoice-type document element, the structured template contains fields such as invoice number, invoice date, amount, and tax. The content parsing process maps the identified numerical sequences, date information, and amount values ​​to the corresponding fields, ultimately generating a structured content representation of the document element. The structured content representation is organized in key-value pair format, with each field name as the key and the corresponding semantic unit content as the value.

[0141] When a document element contains non-text child elements, these child elements require special processing. Non-text child elements include various types such as images, charts, signatures, barcodes, and QR codes. Type identification of non-text child elements is performed based on their original visual features, analyzing visual attributes such as color channel distribution, edge features, and shape features to determine the specific category of the child element. For example, barcodes typically present a regular arrangement of black and white stripes, signatures have specific color characteristics and circular or square outlines, and charts contain identifiable structural elements such as coordinate axes and data points. After type identification, the precise position of each non-text child element is determined within the boundaries of the document element, recording its coordinate offset relative to the parent element, size, and other positional parameters to complete position labeling. For different types of non-text child elements, corresponding supplementary descriptive information is generated. For images, the supplementary descriptive information includes attributes such as image format, resolution, and color mode; for barcodes and QR codes, the supplementary descriptive information includes the decoded data content; for signatures, the supplementary descriptive information includes the signature recognition result and verification status; for charts, the supplementary descriptive information includes metadata such as chart type and data series identifier.

[0142] After obtaining the structured content representation and accompanying descriptive information of document elements, it is necessary to associate and bind the two to establish a clear correspondence. This association and binding is achieved by adding a reference field to the structured content representation, which points to the corresponding accompanying descriptive information. This association method ensures that when querying structured content, related non-text sub-element information can be retrieved simultaneously. For example, if the structured content representation of a document element contains a product image field, the value of this field might be a reference identifier pointing to the specific image's accompanying descriptive information. Through this identifier, detailed information such as the image's storage path and format attributes can be located.

[0143] Based on the hierarchical and sequential relationships of document elements recorded in the hierarchical structure representation, the structured content representation and associated descriptive information of all document elements are organized into a tree data structure. The hierarchical structure representation reflects the inclusion relationships and order between document elements, such as chapters containing paragraphs, paragraphs containing sentences, tables containing rows, and rows containing cells. When constructing the tree data structure, the upper-level document elements are used as parent nodes, and the child elements contained within them are used as child nodes. Each node stores the structured content representation and associated descriptive information of the corresponding document element, and nodes are connected through parent-child pointers. Simultaneously, the order of sibling nodes is determined according to the sequential relationship, ensuring that the traversal order of nodes in the tree structure is consistent with the order in which document elements appear in the original document. The root node of the tree data structure represents the entire target document file, the first-level child nodes correspond to the main chapters or pages of the document, and deeper child nodes correspond to more granular document elements. This organization method preserves the original structural relationships of the document while facilitating quick access and retrieval of specific document element content through tree traversal algorithms.

[0144] The generated structured extraction results are output in the form of a tree-like data structure. This result contains complete information about all document elements in the target document file, including text content, semantic relationships, and non-textual sub-elements, among other multi-dimensional data. The structured extraction results support serialization and storage in JSO7 or XML format, facilitating subsequent data exchange and system integration. In this way, the originally complex document file is transformed into a clearly structured data format that is easy for machines to process, laying the foundation for subsequent information retrieval, data analysis, and knowledge extraction.

[0145] Based on the relationships between document elements in the structured extraction results, the integrity of the structured extraction results is verified, and verification feedback information is generated. This feedback information is then used to dynamically adjust the hierarchical structure representation to correct the spatial relationships and semantic associations between document elements, including:

[0146] The relationships between document elements are extracted from the structured extraction results, and the integrity of the relationships is checked based on these relationships to obtain the verification feedback information.

[0147] Based on the expected location information and missing type information in the verification feedback information, the candidate text region corresponding to the expected location information is located in the initial recognition result;

[0148] The candidate text region is verified based on the association context information between the missing type information and the verification feedback information.

[0149] For candidate text regions that pass the association verification, the spatial relationship features and semantic association features between the candidate text regions and existing document elements in the hierarchical structure representation are recalculated based on the association context information.

[0150] The recalculated spatial relationship features and semantic association features are updated in the hierarchical structure representation, and supplementary document element nodes corresponding to the candidate text regions are inserted into the hierarchical structure representation to correct the spatial relationships and semantic associations between the document elements.

[0151] After document element extraction is complete, a systematic verification of the structured extraction results is necessary to ensure the accuracy and completeness of the results. When extracting the relationships between document elements from the structured extraction results, the focus is on the logical dependencies, hierarchical relationships, and content continuity of the document elements. Specifically, for table elements, check whether the correspondence between the table header and table body cells is complete; for list elements, verify whether each list item appears in the expected order without omission; for chapter elements, confirm whether the hierarchical relationship between the heading level and paragraph content conforms to the overall structure of the document. During the extraction of relationships, a dependency graph structure of document elements is constructed. In this graph structure, nodes represent individual document elements, edges represent the relationships between elements, and the attributes of the edges include information such as the type of relationship, the strength of the relationship, and the expected number of edges.

[0152] When performing dependency graph-based integrity checks, a combination of rule-driven and pattern matching approaches is used. For common document types, integrity verification rules are predefined. For example, an invoice document should include necessary elements such as invoice number, invoice date, buyer information, seller information, product details, and total amount. These elements must also satisfy specific spatial distribution relationships and numerical consistency constraints. During the check, the nodes in the dependency graph are traversed one by one, and the number of incoming and outgoing edges of each node is checked against expectations. For nodes with missing associations, the missing association type, the expected location range of the missing element, and the context information that triggered the missing element detection are recorded. The expected location information is inferred by analyzing the spatial layout patterns of the extracted elements. For example, if a table is found to be missing a column of data, the approximate coordinate range of the missing column is calculated based on the position and spacing of the existing columns. The missing type information includes the element's category identifier, content feature template, and format requirements. For example, if a numeric amount field, a date timestamp, or a text name field is missing, the missing element is considered. The association context information covers the content of the extracted elements adjacent to the missing element, the role of the missing element in the document's logical structure, and its semantic association patterns with other elements.

[0153] When locating candidate text regions in the initial recognition results based on verification feedback, a combination of spatial matching and content filtering strategies is employed. In the spatial matching stage, the expected location information is converted into a rectangular region in the document coordinate system. The determination of this rectangular region considers the document's layout characteristics and the normal spacing between elements, typically setting a certain tolerance range to accommodate positional shifts that may occur during OCR recognition. All text blocks that spatially overlap with this rectangular region are retrieved from the initial recognition results. These text blocks were temporarily shelved or misclassified in the earlier element classification stage due to insufficient confidence, low feature matching, or ambiguous boundaries. In the content filtering stage, candidate text blocks are initially filtered based on the missing element type. For example, if the missing element is a date, text blocks matching the date format are selected; if the missing element is a table cell, text blocks with text length and font size similar to existing cells are selected.

[0154] When performing association verification on candidate text regions, a multi-dimensional verification mechanism is constructed. First, the consistency of spatial relationships between the candidate region and adjacent elements in the associated context is verified. This involves calculating the distance, alignment, and relative position between the candidate region and adjacent elements to determine if these spatial features conform to document layout rules. For example, in a table scenario, it verifies whether the candidate cell maintains horizontal alignment with other cells in the same row, vertical alignment with other cells in the same column, and whether the row height and column width are within reasonable ranges. Second, the degree of matching between the candidate region content and the expected content pattern is verified. This utilizes techniques such as regular expressions, keyword matching, or semantic similarity calculation to determine if the candidate text content meets the feature requirements of the missing type. Third, the semantic coherence between the candidate region and the associated context is verified. By analyzing the semantic association strength between the candidate content and the content of adjacent elements, it determines whether the insertion of the candidate region can form a reasonable semantic chain. For scenarios requiring numerical consistency verification, such as the summation relationship between subtotals and detailed items in financial statements, it verifies whether the candidate values ​​can maintain overall data consistency.

[0155] For candidate text regions that pass the association verification, the spatial relationship features and semantic association features between them and existing document elements in the hierarchical structure representation are recalculated. The calculation of spatial relationship features involves the absolute coordinates, relative position offset, bounding box size, and distance vectors from surrounding elements of the candidate region. These features quantify the distribution of the candidate region in the document space. The calculation of semantic association features is based on the semantic embedding vectors of the candidate region content and the content of related elements. By calculating the cosine similarity, Euclidean distance, or other similarity metrics between vectors, the correlation between the candidate region and other elements in the semantic space is evaluated. In addition, according to the role of the candidate region in the document's logical structure, it is assigned structured attributes such as hierarchical depth, subordinate relationship type, and dependency weight.

[0156] When updating the hierarchical structure representation with recalculated spatial relationship features and semantic association features, the inherent consistency of the structure representation must be maintained. Specific operations include creating new nodes at the corresponding levels of the hierarchical structure, storing the content, type identifier, boundary information, and association features of the candidate text region in these nodes, and establishing connections between these nodes and their parent, child, and sibling nodes. For nodes with missing associations, their edge sets are updated, adding edges pointing to the newly inserted nodes, and adjusting the edge attributes to reflect the corrected associations. During the insertion of supplementary document element nodes, local structural reorganization may be triggered. For example, text blocks that were previously misclassified as independent paragraphs need to be merged into complete list items or table rows after the insertion of new nodes. After dynamic adjustments, the hierarchical structure representation can more accurately reflect the true structure of the target document, and the spatial relationships and semantic associations between document elements are effectively corrected, providing a reliable data foundation for subsequent information extraction, content understanding, and knowledge construction.

[0157] A second aspect of this invention provides a large-file multi-element intelligent recognition and extraction system based on OCR, comprising:

[0158] The document acquisition unit is used to acquire the target document file to be processed, which contains various types of document elements.

[0159] The structure recognition unit is used to perform optical character recognition processing on the target document file to obtain an initial recognition result, and to determine the hierarchical structure representation of the document elements based on the spatial location information and semantic features of the text content in the initial recognition result.

[0160] The element classification unit is used to classify element types according to the spatial relationships and semantic associations in the hierarchical structure representation, divide the document elements into multiple predefined types, and generate type identifiers and boundary information for each document element.

[0161] The content extraction unit is used to extract the corresponding document element content from the target document file according to the type identifier and boundary information, and generate a structured extraction result;

[0162] The verification and adjustment unit is used to perform integrity verification on the structured extraction results based on the relationship between document elements in the structured extraction results, generate verification feedback information, and use the verification feedback information to dynamically adjust the hierarchical structure representation in order to correct the spatial relationship and semantic association between document elements.

[0163] A third aspect of the present invention provides an electronic device, comprising:

[0164] processor;

[0165] Memory used to store processor-executable instructions;

[0166] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0167] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0168] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent recognition and extraction of multiple elements in large files based on OCR, characterized in that, include: Obtain the target document file to be processed, which contains various types of document elements; The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined. Based on the spatial relationships and semantic associations in the hierarchical structure representation, the document elements are classified into multiple predefined types, and a type identifier and boundary information are generated for each document element. Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results; The structured extraction results are validated for completeness based on the relationships between document elements. Validation feedback information is then generated, and the hierarchical structure representation is dynamically adjusted using this feedback information to correct the spatial relationships and semantic associations between document elements.

2. The method according to claim 1, characterized in that, The target document file is subjected to optical character recognition (OCR) processing to obtain an initial recognition result. Based on the spatial location information and semantic features of the text content in the initial recognition result, the hierarchical structure representation of the document elements is determined, including: The target document file is subjected to optical character recognition processing to obtain the original character sequence and the spatial position information and character attribute information of each character unit in the original character sequence; Based on the spatial location information and character attribute information, the original character sequence is reconstructed into text lines to generate a preliminary set of text lines as the initial recognition result. Semantic feature encoding is performed on the text content of each text line in the initial text line set to obtain the semantic representation vector of the text line; A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix; Based on the semantic association matrix, hierarchical clustering is performed on the preliminary set of text lines to identify text line clusters with the same hierarchical attributes, and the inclusion and parallel relationships between the text line clusters are determined. The hierarchical structure representation is determined based on the inclusion and parallel relationships.

3. The method according to claim 2, characterized in that, A multi-scale spatial relationship graph is determined, and semantic propagation calculation is performed on the multi-scale spatial relationship graph based on the semantic representation vector to generate a fused semantic association matrix, including: Based on the boundary information of the text lines in the preliminary text line set, local adjacency edges are established for text line pairs that satisfy the local proximity constraint conditions, thus obtaining the local adjacency relationship layer of the multi-scale spatial relationship graph; The page space of the target document file is adaptively divided into multiple functional regions. Pairs of functional regions with cross-regional semantic association characteristics are identified among the multiple functional regions. Global association edges are established for the text lines in the functional region pairs to obtain the global regional relationship layer of the multi-scale spatial relationship graph. On the local adjacency layer, the semantic representation vector is updated in the first round of propagation based on the local adjacency edges to generate a local semantic fusion vector; On the global region relation layer, the local semantic fusion vector is updated in a second round based on the global association edge to generate a global semantic enhancement vector; Calculate the similarity score between the global semantic enhancement vectors of text line pairs in the preliminary text line set, and perform path decay correction on the similarity score by combining the path connectivity of the text line pairs in the multi-scale spatial relationship graph to generate the semantic association matrix.

4. The method according to claim 1, characterized in that, Based on the spatial relationships and semantic associations in the hierarchical structure representation, element types are classified, dividing the document elements into multiple predefined types, and generating type identifiers and boundary information for each document element, including: The hierarchical depth information, parent node index information, and child node index information of the document elements are extracted from the hierarchical structure representation, and the spatial relationship features and semantic association features between the document elements are also extracted. The context dependency path of a document element is determined based on the hierarchical depth information and the parent node index information, and the inherited semantic features are obtained based on the context dependency path. The internal structural pattern of the document element is determined based on the child node index information and spatial relationship characteristics. The inherited semantic features are fused with the internal composition structure pattern to generate a comprehensive classification feature vector for the document elements; Based on the comprehensive classification feature vector, the document element is typed, mapped to one of the multiple predefined types, and a corresponding type identifier is assigned to the document element. The boundary information of the document element is calculated based on the boundary determination rules corresponding to the type identifier and the child node index information of the document element in the hierarchical structure representation.

5. The method according to claim 4, characterized in that, The context dependency path of a document element is determined based on the hierarchical depth information and parent node index information, and the internal composition structure pattern of the document element is determined based on the child node index information and spatial relationship features, including: Based on the hierarchical depth information and the parent node index information, a reverse hierarchical tracing is performed, traversing upwards from the document element along the parent node pointer until the root node is reached, to obtain the context dependency path. The context dependency path records all ancestor nodes of the document element in an ordered node sequence. For each ancestor node in the context-dependent path, the semantic association features and hierarchical position encoding of the ancestor node are extracted, and hierarchical decay weights are assigned according to the relative distance of the ancestor node in the context-dependent path; The inherited semantic features are generated by weighted aggregation of the semantic association features of all ancestor nodes in the context-dependent path; Based on the spatial relationship characteristics of the child nodes, the relative positional relationship matrix between the child nodes is calculated, and the structural pattern mining is performed on the relative positional relationship matrix to obtain the regular arrangement characteristics of the spatial layout of the child nodes. Based on the regularity of the arrangement and the number of child nodes in the child node index information, a structural pattern encoding vector is generated to obtain the internal composition structural pattern.

6. The method according to claim 1, characterized in that, Based on the type identifier and boundary information, the corresponding document element content is extracted from the target document file to generate structured extraction results, including: Based on the type identifier, determine the extraction rule corresponding to the type identifier from a predefined set of type extraction rules; Based on the boundary information, the original text content and original visual features corresponding to the document elements are determined from the target document file. The original text content is parsed according to the extraction rules to determine the semantic relationships between semantic units in the original text content, and the semantic units are mapped to fields based on the structured template corresponding to the type identifier to generate a structured content representation of the document element; Based on the type identifier and the boundary information, when the document element contains non-text sub-elements, the non-text sub-elements are identified and their positions are marked according to the original visual features, and supplementary descriptive information of the non-text sub-elements is generated. The structured content representation is associated and bound with the attached descriptive information, and according to the hierarchical and sequential relationships of the document elements in the hierarchical structure representation, the structured content representation and attached descriptive information of all document elements are organized into a tree data structure to generate the structured extraction result.

7. The method according to claim 1, characterized in that, Based on the relationships between document elements in the structured extraction results, the integrity of the structured extraction results is verified, and verification feedback information is generated. This feedback information is then used to dynamically adjust the hierarchical structure representation to correct the spatial relationships and semantic associations between document elements, including: The relationships between document elements are extracted from the structured extraction results, and the integrity of the relationships is checked based on these relationships to obtain the verification feedback information. Based on the expected location information and missing type information in the verification feedback information, the candidate text region corresponding to the expected location information is located in the initial recognition result; The candidate text region is verified based on the association context information between the missing type information and the verification feedback information. For candidate text regions that pass the association verification, the spatial relationship features and semantic association features between the candidate text regions and existing document elements in the hierarchical structure representation are recalculated based on the association context information. The recalculated spatial relationship features and semantic association features are updated in the hierarchical structure representation, and supplementary document element nodes corresponding to the candidate text regions are inserted into the hierarchical structure representation to correct the spatial relationships and semantic associations between the document elements.

8. A large-file multi-element intelligent recognition and extraction system based on OCR, used to implement the method as described in any one of claims 1-7, characterized in that, include: The document acquisition unit is used to acquire the target document file to be processed, wherein the target document file contains various types of document elements; The structure recognition unit is used to perform optical character recognition processing on the target document file to obtain an initial recognition result, and to determine the hierarchical structure representation of the document elements based on the spatial location information and semantic features of the text content in the initial recognition result. The element classification unit is used to classify element types according to the spatial relationships and semantic associations in the hierarchical structure representation, divide the document elements into multiple predefined types, and generate type identifiers and boundary information for each document element. The content extraction unit is used to extract the corresponding document element content from the target document file according to the type identifier and boundary information, and generate a structured extraction result; The verification and adjustment unit is used to perform integrity verification on the structured extraction results based on the relationship between document elements in the structured extraction results, generate verification feedback information, and use the verification feedback information to dynamically adjust the hierarchical structure representation in order to correct the spatial relationship and semantic association between document elements.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images