High-performance document data reorganizing method based on persistent memory

Abstract

The invention relates to a high-performance document data reorganization method based on a persistent memory, which belongs to the technical field of data processing in computer software, and comprises the following steps of: initializing a persistent memory pool for storing original document blocks, intermediate analysis results and structured data; a to-be-reorganized document is mapped to a persistent memory pool, and multi-level copying is avoided; starting a multi-thread analyzer to read the document blocks in parallel, identifying document types and extracting content elements; an intermediate structured semantic block tree is constructed based on the extracted content elements, the semantic block tree stores specific content through leaf nodes, and a theme or a concept is represented through intermediate nodes; persistently storing the semantic chunk tree in a persistent memory pool by using transactional operation; an index structure is constructed in a persistent memory to support efficient retrieval. According to the method, the whole process of document analysis, semantic extraction and index construction is deployed in a persistent memory space, and high-performance, high-consistency and recoverable mass document reorganizing treatment is realized by utilizing the characteristics of high capacity and persistence of the persistent memory space.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology in computer software, specifically relating to a method for extracting content, performing structured transformation, and efficiently organizing and managing heterogeneous documents (such as PDF, Word, HTML, plain text, etc.) by utilizing the high-performance characteristics of persistent memory. It is applicable to application scenarios such as big data platforms, knowledge graph construction, and intelligent document management. Background Technology

[0002] With the acceleration of enterprise digital transformation, unstructured document data is experiencing explosive growth. Due to the typically large number of documents and cost considerations, document files are often stored on large-capacity mechanical hard drives with limited I / O capabilities, resulting in performance bottlenecks when processing massive amounts of documents. In recent years, Intel has introduced new storage media such as Optane persistent memory, which offers access speeds and non-volatility approaching DRAM, providing new possibilities for high-performance data processing.

[0003] Currently, because document files are stored on large-capacity mechanical hard drives, most data processing systems still run document parsing, feature extraction, and structure transformation processes on the traditional memory-disk architecture, failing to achieve an efficient end-to-end processing pipeline, which is particularly inefficient in scenarios requiring frequent backtracking, incremental updates, or fault-tolerant recovery.

[0004] In the traditional data compilation and governance process, the introduction of persistent memory can retain data even when power is off, while also providing read and write performance close to DRAM, offering a new optimization approach for document compilation.

[0005] The traditional document compilation and management process has the following problems: 1. Performance bottleneck: Traditional document editing processes are limited by disk I / O latency and DRAM capacity, making it difficult to meet the real-time editing requirements of high throughput and low latency; 2. Data consistency and fault tolerance issues: During long-term compilation tasks, system crashes may lead to the loss of intermediate states, requiring the entire document set to be reprocessed; 3. Inefficient structured processing: Existing methods lack memory-optimized data layouts and access patterns, making them unable to efficiently support the extraction, indexing, and association of massive amounts of document content; Summary of the Invention To address the aforementioned problems, this invention provides a high-performance document data assembling method based on persistent memory, improving the efficiency of document assembling and governance by leveraging the large capacity and persistence characteristics of persistent memory. Specifically, this invention aims to solve the performance bottlenecks, poor data consistency, and weak fault tolerance caused by disk I / O latency and DRAM capacity limitations in traditional document assembling methods. By deploying the entire process of document parsing, semantic extraction, and index construction in persistent memory space, and utilizing its large capacity and persistence characteristics, high-performance, highly consistent, and recoverable massive document assembling and governance is achieved.

[0006] To achieve the above objectives, the present invention provides a high-performance document data organization method based on persistent memory, comprising the following steps: Initialize a persistent memory pool to store raw document blocks, intermediate parsing results, and structured data; The document to be compiled is directly mapped to the persistent memory pool in the form of blocks to avoid multi-level copying; A multi-threaded parser is launched in persistent memory space to read document blocks in parallel, identify document types and extract content elements; An intermediate structured semantic block tree is constructed based on the extracted content elements. The semantic block tree stores specific content in leaf nodes and represents topics or concepts in intermediate nodes. Use transactional operations to persist the semantic block tree to a persistent memory pool; Build the index structure in persistent memory to support efficient retrieval.

[0007] Preferably, constructing the intermediate structured semantic block tree includes: Identify semantic boundaries and sub-blocks, extract semantic blocks from the document, and apply natural language processing techniques to identify key elements; Create an intermediate node for the current level; The semantic block content is aggregated using a recursive method and organized into a tree structure according to logical relationships.

[0008] Preferably, the key elements for identification include: Record the heading level depth of each semantic block in the original document. , including first-level headings =1, second-level heading =2, the body paragraphs inherit the depth of their nearest parent heading; Text of each semantic block The input to the pre-trained language model yields augmented semantic vectors. ; Calculate any two semantic blocks and The similarity between them is determined by a weighted average of cosine semantic similarity and positional distance.

[0009] Preferably, the similarity calculation formula is:

[0010] Where α∈[0,1] are semantic and positional weight hyperparameters. γ represents the sequential position of the semantic block in the document, and γ is the distance decay function.

[0011] Preferably, the tree structure is generated through bottom-up hierarchical clustering: Initialize each semantic block as a leaf node cluster; The inter-cluster similarity of legitimate cluster pairs is calculated using the average link method; Clusters with similarity exceeding a threshold τ are selected and merged to generate a new parent node, whose semantic vector is a weighted average of the sub-cluster vectors; Update the level of the new cluster to the minimum depth of the sub-cluster; Stop when there are no valid merge pairs or the preset maximum tree depth is reached.

[0012] Preferably, the method also includes a semantic tree post-processing step: performing structural verification on the clustering results; if there are hierarchical jumps or isolated text blocks in a subtree, then reattaching them to the nearest legal parent node based on location proximity and maximum similarity.

[0013] Preferably, constructing the index structure includes: Entity recognition, relation extraction, and topic clustering are performed based on semantic block trees; Utilize hash tables or B+ tree structures in persistent memory to construct inverted indexes and entity graphs.

[0014] Preferably, transactional operations ensure atomic writes to the semantic block tree, enabling recovery from the most recent consistent state after a system crash without having to reprocess the entire document set.

[0015] Preferably, when a new document is input, the new document block is mapped to persistent memory and a multi-threaded parser is started to process it, a new semantic block subtree is built and merged into the existing semantic block tree, and the index structure is incrementally updated at the same time.

[0016] Preferably, the distance decay function is γ(Δp)=e-λΔp, where λ is the decay coefficient and Δp is the positional distance between the two semantic blocks.

[0017] Compared with the prior art, the present invention has the following beneficial technical effects: This invention ensures extremely high processing performance. By deploying the entire process of document parsing, content extraction, structured mapping, and metadata indexing in persistent memory, it completely eliminates the multi-level data copy overhead of "disk → DRAM → processing" in traditional architectures. After document blocks are directly mapped to the persistent memory pool, parsing and assembly operations are completed in high-bandwidth, low-latency persistent memory, significantly reducing I / O latency and meeting the requirements for high-throughput, low-latency real-time assembly.

[0018] This invention provides strong consistency and reliable fault tolerance. It employs an intermediate structured semantic block tree construction mechanism based on persistent memory transactions, ensuring the atomicity of tree structure generation and update operations. Even if a system crash occurs during a long-running assembly task, data will not be lost, and the system can quickly recover from the most recent consistent state without reprocessing the entire document set, significantly improving system reliability and task continuity.

[0019] This invention boasts excellent high-capacity support capabilities. Persistent memory provides terabyte-level storage capacity, capable of simultaneously caching the original content, intermediate parsing results, and structured semantic trees of millions of documents. This avoids the frequent data swapping caused by the capacity limitations of traditional DRAM, enabling ultra-large-scale document compilation tasks to be completed efficiently in memory, supporting enterprise-level massive data processing scenarios.

[0020] This invention excels in scalability and incremental update capabilities. The multi-threaded parallel parser fully utilizes multi-core CPU resources to achieve parallel document processing. Simultaneously, the system supports streaming document ingestion and incremental updates; new documents can be dynamically mapped to persistent memory and semantic block subtrees constructed, then seamlessly merged into the existing structure. Furthermore, the index structure is updated incrementally and synchronously, eliminating the need for a full rebuild, making it suitable for continuously changing data environments.

[0021] The semantic tree construction algorithm of this invention is more intelligent and accurate. By combining enhanced semantic vectors generated by a pre-trained language model, heading level depth constraints, and a positional distance-based decay function, the algorithm can accurately capture the logical hierarchical relationships within a document. Bottom-up hierarchical clustering merges semantic blocks only between semantic blocks with a level difference of no more than 1, ensuring that the generated multi-branch tree structure strictly conforms to the original logic of the document, avoiding unreasonable level jumps, and laying a high-quality data foundation for subsequent entity recognition, relation extraction, and semantic retrieval. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the three stages in a high-performance document data comprehension method based on persistent memory according to the present invention; Figure 2 This is a schematic diagram illustrating the specific process of a high-performance document data compilation method based on persistent memory according to the present invention. Figure 3This is a schematic diagram illustrating the structured semantic block tree construction approach in a high-performance document data organization method based on persistent memory according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention proposes a high-performance document data assembling method based on persistent memory. Its core idea is to deploy the entire process of document parsing, content extraction, structured mapping and metadata indexing in persistent memory space, and design a dedicated algorithm to achieve efficient, recoverable and low-overhead assembling and governance.

[0025] This method mainly consists of three stages: 1. Multi-type document input serves as the starting point of the process, receiving documents from multiple sources or formats, including document 1, document 2 to document n. Multi-type document input includes documents of various types, including documents from multiple different sources or formats. 2. Load and store document content in persistent memory through persistent memory pool mapping, forming document1 mapping, document2 mapping to documentn mapping, so as to enable efficient access and processing; 3. Perform document logic processing and index building in persistent memory, including natural language understanding of document content to extract key information and meaning, constructing structured semantic blocks, organizing the identified semantic information into structured data blocks, and building an index structure to support efficient semantic retrieval.

[0026] The specific process is as follows: First, initialize the persistent memory pool. Then, determine if there are any new documents input. If there are new documents, map the document blocks to persistent memory, start the multi-threaded parser and call the corresponding parser according to the type, build the intermediate structured semantic block tree, use transactional operations to store it in persistent memory, and build or update the inverted index and entity graph at the same time. If there are no new documents, proceed directly to the next step and finally provide query services until the process ends.

[0027] The steps are as follows: Step 1: Persistent memory pool initialization and document loading When the system starts, a persistent memory pool is created to store raw document blocks, intermediate parsing results, and structured data.

[0028] The documents to be compiled are directly mapped to the persistent memory pool in the form of blocks, avoiding the multi-level copying of the traditional "disk → DRAM → processing".

[0029] Step 2: Parallel document parsing and content extraction Start a multi-threaded parser, with each thread reading one document block from the persistent memory pool; Determine the document type by the file extension and document header, and call the corresponding parser (such as a PDF or DOCX parser). Extract key content elements: text paragraphs, tables, image metadata, heading levels, keywords, etc., and generate intermediate structured semantic blocks.

[0030] Step 3: Persist intermediate structured semantic blocks The process of constructing an intermediate structured semantic block tree involves hierarchically parsing and representing document content. Leaf nodes contain specific content or sentences; intermediate nodes represent the themes or concepts of a set of related leaf nodes, or are further abstractions of these nodes.

[0031] The construction approach of structured semantic block tree is as follows: Figure 3 As shown. The algorithm flow is as follows: Identifying semantic boundaries and sub-blocks: Based on the document's structured representation in persistent memory, identification is performed to extract all semantic blocks (such as chapter titles, clauses, paragraphs, notes, etc.). Natural language processing techniques are applied to identify key elements in the current semantic block (such as topics, subtopics, important sentences, etc.) to clarify the semantic boundaries and sub-blocks of the current block.

[0032] Intermediate node creation: Creates an intermediate node for the current level (if multiple child blocks exist).

[0033] Intermediate structured semantic block tree construction: In this process, a recursive method is used to progressively aggregate the content of each semantic block until the finest-grained leaf node is reached. They are then organized into a tree structure according to logical relationships. The semantic tree construction process is shown below: S1: Semantic Vector Transformation Record the heading level depth of the semantic block in the original document. ∈N+ (e.g., first-level headings) =1, second-level heading =2, the body paragraph inherits the depth of its nearest parent heading). Each semantic block Forming tuples ,in Its sequential position within the document. The text of each semantic block. Inputting the pre-trained BERT model yields augmented semantic vectors. .

[0034] For any two semantic blocks and Calculate cosine semantic similarity:

[0035] Where cos is the cosine similarity, and γ is the distance decay function (γ(Δp)=e -λΔp ), α∈[0,1] are semantic and positional weight hyperparameters.

[0036] At the same time, only blocks with a level difference of no more than 1 are allowed to participate in clustering and merging, that is, the main text can only be assigned to the headings of adjacent levels to ensure that the tree structure conforms to the document logic.

[0037] S2: Bottom-up hierarchical clustering generates semantic trees Initialize each semantic block as a leaf node cluster ={ }, the corresponding vector is Repeat the following operations until the termination condition is met: Calculate all legal cluster pairs ( , The inter-cluster similarity was calculated using the UPGMA (Unified Linkage Method); the formula is as follows:

[0038] Clusters with the highest similarity exceeding a threshold τ are selected and merged to generate a new parent node Cnew, whose semantic vector is a weighted average of the sub-cluster vectors.

[0039] Update level: The level of the new cluster (i.e., taking the depth of a higher-level heading); If there are no valid merge pairs or the maximum tree depth reaches the preset value, the process will stop.

[0040] S3: Semantic Tree Post-processing and Output The clustering results are structurally validated: if a subtree contains hierarchical jumps or isolated text blocks, it is reattached to the nearest valid parent node based on proximity and maximum similarity. The final output is a multi-way tree, where each node contains the original semantic block content, hierarchical depth, and aggregated semantic vector.

[0041] Persistent storage: Once the intermediate structured semantic block tree is built, it is safely saved to persistent memory using transactional operations, ensuring that no data is lost even if the system crashes.

[0042] Step 4: Structured Data Compilation and Index Building Semantic assembly is performed based on the newly generated intermediate structured semantic block tree: entity recognition, relation extraction, and topic clustering; Utilize hash tables or B+ tree structures in persistent memory to construct inverted indexes and entity graphs; The index structure is also housed in a persistent memory pool, supporting fast querying of indexed content. Upon document update, the full-text search index and any existing entity graphs are updated synchronously to support fast querying.

[0043] This invention effectively combines the advantages of persistent memory with sophisticated text analysis techniques to achieve a highly efficient and stable document organization and management solution. Specifically, it offers the following advantages: High performance: Document parsing and compilation are completed entirely in persistent memory, avoiding DRAM to disk data migration and greatly reducing I / O latency; Strong consistency and fault tolerance: The intermediate structured semantic block tree construction mechanism based on persistent memory transactions ensures that the system can recover from the most recent consistent state after a crash without reprocessing; Large capacity support: Utilizing persistent memory with a capacity of terabytes, it can cache the intermediate structure of millions of documents at the same time, supporting ultra-large-scale compilation tasks. Highly scalable: Supports parallel parsing and incremental updates, suitable for streaming document ingestion scenarios.

[0044] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-performance document data organization method based on persistent memory, characterized in that, Includes the following steps: Initialize a persistent memory pool to store raw document blocks, intermediate parsing results, and structured data; The document to be compiled is directly mapped to the persistent memory pool in the form of blocks to avoid multi-level copying; A multi-threaded parser is launched in persistent memory space to read document blocks in parallel, identify document types and extract content elements; An intermediate structured semantic block tree is constructed based on the extracted content elements. The semantic block tree stores specific content in leaf nodes and represents topics or concepts in intermediate nodes. Use transactional operations to persist the semantic block tree to a persistent memory pool; Build the index structure in persistent memory to support efficient retrieval.

2. The high-performance document data organization method based on persistent memory according to claim 1, characterized in that, Constructing the intermediate structured semantic block tree includes: Identify semantic boundaries and sub-blocks, extract semantic blocks from the document, and apply natural language processing techniques to identify key elements; Create an intermediate node for the current level; The semantic block content is aggregated using a recursive method and organized into a tree structure according to logical relationships.

3. The high-performance document data organization method based on persistent memory according to claim 2, characterized in that, Key elements to identify include: Record the heading level depth of each semantic block in the original document. , including first-level headings =1, second-level heading =2, the body paragraphs inherit the depth of their nearest parent heading; Text of each semantic block The input to the pre-trained language model yields augmented semantic vectors. ; Calculate any two semantic blocks and The similarity between them is determined by a weighted average of cosine semantic similarity and positional distance.

4. The high-performance document data organization method based on persistent memory according to claim 3, characterized in that, The formula for calculating similarity is: Where α∈[0,1] are semantic and positional weight hyperparameters. γ represents the sequential position of the semantic block in the document, and γ is the distance decay function.

5. The high-performance document data organization method based on persistent memory according to claim 4, characterized in that, The tree structure is generated through bottom-up hierarchical clustering: Initialize each semantic block as a leaf node cluster; The inter-cluster similarity of legitimate cluster pairs is calculated using the average link method; Clusters with similarity exceeding a threshold τ are selected and merged to generate a new parent node, whose semantic vector is a weighted average of the sub-cluster vectors; Update the level of the new cluster to the minimum depth of the sub-cluster; Stop when there are no valid merge pairs or the preset maximum tree depth is reached.

6. The high-performance document data organization method based on persistent memory according to claim 5, characterized in that, It also includes a semantic tree post-processing step: performing structural verification on the clustering results; if there are hierarchical jumps or isolated text blocks in a subtree, then reattaching them to the nearest legal parent node based on location proximity and maximum similarity.

7. The high-performance document data organization method based on persistent memory according to claim 1, characterized in that, Building the index structure includes: Entity recognition, relation extraction, and topic clustering are performed based on semantic block trees; Utilize hash tables or B+ tree structures in persistent memory to construct inverted indexes and entity graphs.

8. The high-performance document data organization method based on persistent memory according to claim 1, characterized in that, Transactional operations ensure atomic writes to the semantic block tree, enabling recovery from the most recent consistent state after a system crash without having to reprocess the entire document set.

9. The high-performance document data organization method based on persistent memory according to claim 1, characterized in that, When a new document is input, the new document block is mapped to persistent memory and a multi-threaded parser is started to process it, a new semantic block subtree is built and merged into the existing semantic block tree, and the index structure is incrementally updated at the same time.

10. A high-performance document data organization method based on persistent memory according to claim 9, characterized in that, The distance decay function is γ(Δp) = e -λΔp , where λ is the attenuation coefficient and Δp is the positional distance between the two semantic blocks.

Images