Method and system based on NLP file analysis

Abstract

The invention provides a method and system based on NLP file analysis, and relates to the technical field of natural language processing. According to the method, time and identifier unification and format and character set standardization are carried out on the multi-source file, layout segmentation, table structure extraction, reference analysis, term standardization and anaphora resolution are combined, semantic representation is constructed, a hierarchical index and a unique traceability identifier are generated, intention recognition, retrieval sorting, incremental updating and consistency verification are supported, and the method is suitable for large-scale popularization and application. Unification, semantization and traceability of the file analysis process are achieved, and the processing efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system based on NLP document analysis. Background Technology

[0002] With the digitization of enterprise knowledge assets, document types are becoming increasingly diverse and voluminous, originating from collaborative platforms, email attachments, knowledge bases, and business systems. Traditional keyword retrieval has limited semantic understanding and insufficient support for cross-document citations, version evolution, and fragment-level location, leading to unstable relevance and usability of query results. Furthermore, frequent document updates and the mixing of historical and current content make it difficult to achieve traceability and audit compliance.

[0003] Document analysis for business decision-making and compliance auditing is shifting from "keyword indexing" to "semantic understanding and relationship modeling." Industry solutions are continuously combining named entity recognition, referential resolution, cross-document linking, and vector retrieval to build semantic indexes and knowledge graphs to support intent-based question answering, evidence backtracking, and incremental maintenance. Simultaneously, version chains, source credibility, and time sensitivity are gradually being incorporated into sorting and presentation logic, forming a collaborative retrieval and analysis framework based on the three elements of "semantics, relationships, and timeliness."

[0004] Existing technologies mostly rely on single-modality and single-layer indexing, lacking consistent constraints on internal structures such as titles, body text, and tables, leading to semantic drift in fragment-level search results. Most solutions lack a unified source identification system, making it difficult to stably map query results back to their original text positions to support auditing. Incremental updates often involve full reconstruction, which is costly and prone to introducing temporary index imbalances. Insufficient interpretation of intent and constraints during the query phase prevents scope, timeliness, and quality thresholds from being structurally incorporated into the ranking, affecting the controllability and reproducibility of the results. These issues limit the reliable implementation of document analysis in demanding scenarios. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method and system based on NLP document analysis. Through an innovative design of the entire process of document normalization, structure parsing, terminology resolution, semantic indexing and incremental maintenance, it overcomes the deficiencies of existing technologies in terms of uniformity, accuracy, traceability and update efficiency, and realizes an efficient, stable and traceable document semantic analysis method.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method based on NLP document analysis includes:

[0008] Collect files and metadata from multiple sources under a unified time base and identification rules, complete the format, character set and field specifications, and generate standardized objects;

[0009] Perform page segmentation and hierarchical modeling, divide the normalized object into sentence sequence and paragraph level, locate table candidate areas and citation candidate tags, and establish position index as the basis for structure extraction and relationship parsing;

[0010] The table structure is extracted based on the candidate table area, the table header, cell coordinates and merging relationships are determined, and cross-document reference relationships are parsed based on the candidate reference tags;

[0011] Perform terminology normalization and reference resolution on sentence elements, paragraph headings, table cell text, and referenced target names to generate a terminology table and entity mapping;

[0012] Semantic representations are generated based on semantic models, and consistency constraints are applied to titles, body text, and tables to obtain unified semantic vectors and backtracking indexes. Semantic indexes, keyword indexes, and relationship indexes are also established simultaneously, and unique traceability identifiers are generated and stored in association.

[0013] Perform intent identification and constraint interpretation, retrieve and sort results by relevance, confidence, and novelty, and output results with confidence scores;

[0014] Incremental updates and consistency checks are performed on add, modify, or delete events. If a failure occurs, the process is rolled back and an exception is marked.

[0015] Preferably, under a unified time base and identification rules, multi-source files and metadata are collected, and format, character set, and field specifications are completed to generate standardized objects, including:

[0016] Determine the data collection source and synchronization method, and adjust the timestamps of files from each source according to a unified time base to ensure that files from different sources are aligned on the same time scale;

[0017] Generate a globally unique identifier for each file and metadata collected according to the identification rules, and bind the source identifier and time information together;

[0018] The collected files are converted into a format according to their type and unified into a preset intermediate representation structure, while maintaining the paragraph hierarchy and resource reference relationship unchanged;

[0019] The system detects and unifies the character set of a file, converts it to the target character set encoding, and replaces or cleans up abnormal characters to ensure semantic consistency.

[0020] Standardize the mapping of metadata fields, aligning the title, author, creation time, version number, and source path to a unified field standard;

[0021] Using the globally unique identifier as the primary key, the intermediate representation structure is used as the content body. The target character set encoding and the normalized metadata fields are merged, and page number, paragraph number, position mapping, table cell coordinates and merge relationship mapping, as well as the reference pointer, source path and version information of the embedded resources are written to form the normalized object containing a unified format, a unified character set and unified fields.

[0022] Preferably, page layout segmentation and hierarchical modeling are performed, dividing the normalized object into sentence sequence and paragraph level, locating candidate table areas and candidate citation markers, and establishing a position index as the basis for structure extraction and relationship parsing, including:

[0023] Based on page boundaries and text connectivity, the normalized object is divided into pages, columns, blocks, and rows to determine the reading order;

[0024] Based on the rules of sentence segmentation and word boundary, the inline text is converted into a sentence element sequence, and the start and end offsets are marked for each sentence element in the sentence element sequence.

[0025] Generate a paragraph tree structure based on indentation depth, line spacing variations, heading styles, and numbering levels, and assign paragraph numbers to each paragraph.

[0026] Based on line segment continuity, cell alignment, grid periodicity, and header features, candidate regions for the table are determined, and the rectangular range and grid estimation parameters are output.

[0027] Based on citation number patterns, author year patterns, anchor text features, and chart number patterns, candidate citation markers are identified and bound to adjacent sentence elements.

[0028] Using sentence offset as the primary key, a mapping is established between page number, column number, block number, row number, paragraph number and table candidate coordinates to generate a traceable position index.

[0029] Preferably, the table structure is extracted based on the candidate table region, the table header, cell coordinates and merging relationships are determined, and cross-document reference relationships are parsed based on the candidate reference tags, including:

[0030] The location index is used to perform boundary refinement and skew correction on the candidate area of ​​the table to obtain the bounding rectangle of the table and the initial number of rows and columns of the grid;

[0031] Generate row and column separators based on line segment continuity and alignment constraints, output the top-left and bottom-right coordinates of each cell, and assign row and column numbers;

[0032] Based on the characteristics of missing adjacent cell boundaries and cross-domain text extension, row-direction merging and column-direction merging are determined to form a set of merging spans and target coordinates;

[0033] Based on the prior knowledge of the header position, style differences, and field dictionary, the header area is determined, and a field-to-column mapping relationship is established between the header cells and the data cells;

[0034] Summarize the number of rows and columns, coordinates, merged span, and field-to-column mapping to form a table structure model, and establish links with the corresponding page number and paragraph number;

[0035] For the aforementioned citation candidate tags, we perform number format unification, author year format unification, and anchor text cleaning to extract citation keys, target titles, and target numbers;

[0036] Candidate targets are retrieved in the hierarchical index based on the reference key, target title and target number. Globally unique identifiers and version information are matched first. If they are not unique, source credibility and time freshness are used for disambiguation.

[0037] Establish a one-to-many or one-to-one link between the source tag and the globally unique identifier of the referenced target, and record the source tag offset and the target position coordinates.

[0038] Preferably, terminology normalization and reference resolution are performed on sentence elements, paragraph headings, table cell text, and referenced target names to generate a terminology table and entity mapping, including:

[0039] Candidate terms are extracted from sentence elements, paragraph headings, table cell text, and reference target names, and the case, number notation, and character format are uniformly processed to provide standardized input for subsequent aggregation.

[0040] Based on the candidate terms, identify the relationship between aliases and abbreviations, match the abbreviations with the full names, and merge different spellings into a unified standard name;

[0041] After alias merging, cluster analysis is performed on the term set. By combining character similarity, contextual semantics, and header clues, terms with the same meaning are grouped into one cluster to obtain clustering results, and the types are further determined. The types include: technical terms, institution names, field names, and document entries.

[0042] After obtaining the clustering results, the referential resolution is performed. The same reference constraint is established by paragraph sequence, intra-sentence distance and intra-table column consistency. When crossing documents, the globally unique identifier and version information are given priority to ensure that the same entity is accurately tracked.

[0043] The terminology list is generated based on the resolution results. The main name, alias relationship, abbreviation and full name relationship, and type information of the normalized terms are output. At the same time, an entity identifier is assigned to each occurrence, and a mapping is established from sentence elements, paragraph titles, table cell text and reference target names to the entities.

[0044] Preferably, semantic representations are generated based on a semantic model, and consistency constraints are applied to the title, body text, and tables to obtain a unified semantic vector and a backtracking index. Simultaneously, a semantic index, a keyword index, and a relationship index are established, and a unique traceability identifier is generated and stored in association, including:

[0045] Taking sentence elements, paragraph titles, and table cell text that have undergone terminology normalization and dereference resolution as input, the semantic model is invoked to generate corresponding semantic vectors.

[0046] Consistency constraints are applied to the title, body text, and table content in the generated semantic vector to ensure the alignment and association of different structural units within the same file in the semantic space;

[0047] A unified semantic representation is constructed based on the semantic vectors after consistency constraints, and a traceable index is generated to correspond to the page number, paragraph number and table coordinates of the original text.

[0048] Based on the unified semantic representation, a three-tiered index structure of semantic index, keyword index and relation index is established to support multi-dimensional retrieval and reasoning;

[0049] A unique traceability identifier is generated based on the source information, version information, and reference relationship. The unique traceability identifier is then associated with the hierarchical index and the storage path of the original file to achieve integrated storage of semantic content and traceability information.

[0050] Preferably, the process involves performing intent recognition and constraint interpretation, retrieving and sorting results by relevance, confidence, and novelty, and outputting results with confidence scores, including:

[0051] Receive query requests, parse natural language expressions, extract query topics and core objectives, and obtain intent representations;

[0052] Based on the aforementioned intent representation, constraints are identified to form a set of constraints that includes scope, timeliness, and quality requirements;

[0053] The semantic index, keyword index, and relation index are invoked to retrieve the candidate result set based on the intent representation and the constraint set.

[0054] The candidate result set is sorted by a comprehensive weighted sort based on semantic relevance, source credibility, and version freshness to obtain the sorting result;

[0055] Calculate the confidence score for each candidate result in the ranking results and output the set of search results with confidence scores for users to call.

[0056] Preferably, incremental updates and consistency checks are performed under add, modify, or delete events, and a rollback and exception are marked when failure occurs, including:

[0057] When a file is added, modified, or deleted, an update process is triggered to identify the affected file objects and associated indexes.

[0058] Based on the affected file objects, extract the change fragments, generate incremental update records, and compare them with the existing index;

[0059] Perform a consistency check to verify that the semantic index, keyword index, and relationship index are consistent with the latest file content, version information, and reference relationships.

[0060] If the consistency check passes, the incremental update result is written to the index and storage system, and the unique traceability identifier is updated.

[0061] If the consistency check fails, the system will roll back to the most recent consistent state and generate an exception flag for the content that failed the check.

[0062] A system based on NLP document analysis includes:

[0063] The data acquisition and normalization unit is used to collect multi-source files and metadata under a unified time base and identification rules, complete the format, character set and field standardization, and generate normalized objects;

[0064] The page layout segmentation and hierarchical modeling unit is used to perform page layout segmentation and hierarchical modeling, dividing the normalized object into sentence sequence and paragraph level, locating table candidate areas and reference candidate markers, and establishing a position index as the basis for structure extraction and relationship parsing.

[0065] The table structure extraction and reference parsing unit is used to extract the table structure based on the table candidate area, determine the table header, cell coordinates and merging relationship, and parse the cross-document reference relationship based on the reference candidate mark;

[0066] The terminology normalization and reference resolution unit is used to perform terminology normalization and reference resolution on sentence elements, paragraph headings, table cell text and reference target names, and generate a terminology table and entity mapping;

[0067] The semantic representation and index building unit is used to generate semantic representations based on semantic models, apply consistency constraints to titles, body text and tables, obtain unified semantic vectors and backtracking indexes, and simultaneously build semantic indexes, keyword indexes and relationship indexes, generate unique traceability identifiers and store them in association;

[0068] The intent recognition and retrieval ranking unit is used to perform intent recognition and constraint interpretation, retrieve and rank results by relevance, confidence and novelty, and output results with confidence scores.

[0069] The incremental update and consistency maintenance unit is used to perform incremental updates and consistency checks under add, modify, or delete events, and roll back and mark the exception when it fails.

[0070] The present invention discloses the following technical effects:

[0071] This invention effectively solves the problems of content fragmentation and processing difficulties caused by diverse file sources and different formats in the prior art by collecting files from multiple sources under a unified time benchmark and identification rules and completing the format, character set and field specifications, thereby achieving standardized and unified management of cross-source files.

[0072] This invention performs page segmentation and hierarchical modeling, and on this basis, locates candidate table regions and candidate reference markers, thereby accurately extracting table structure and cross-document reference relationships. This overcomes the shortcomings of existing technologies in terms of insufficient page structure parsing and difficulty in tracing reference relationships, and improves document structuring and traceability capabilities.

[0073] This invention generates a glossary and entity mapping by performing terminology normalization and reference resolution on sentence elements, paragraph titles, table cell text and referenced target names. This ensures that the same entity is consistently represented in different documents and locations, and solves the semantic dispersion problem caused by terminology confusion and unclear reference in the prior art.

[0074] This invention generates semantic representations based on semantic models and applies consistency constraints among titles, body text, and tables. It also establishes semantic indexes, keyword indexes, and relationship indexes, and combines unique traceability identifiers to achieve highly relevant and traceable semantic retrieval, thus overcoming the shortcomings of existing technologies that rely solely on keywords or lack consistency constraints.

[0075] This invention ensures the stability and integrity of the index and traceability system during dynamic updates by performing incremental updates and consistency checks under new, modified, or deleted events, and by rolling back and marking exceptions when checks fail. This overcomes the problems of low efficiency and easy imbalance caused by existing technologies that rely on full reconstruction. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0078] Figure 2This is a schematic diagram of the system structure provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] The purpose of this invention is to provide a method and system for NLP-based document analysis, which optimizes the entire process from document collection and standardization to structure parsing, terminology resolution, semantic index construction and incremental updates. It has the advantages of strong uniformity, accurate parsing, semantic consistency, traceable retrieval and efficient updates.

[0081] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0082] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a method for NLP file analysis, including:

[0083] Step 100: Collect multi-source files and metadata under a unified time base and identification rules, complete the format, character set and field specifications, and generate a standardized object;

[0084] Step 200: Perform page segmentation and hierarchical modeling, divide the normalized object into sentence sequence and paragraph level, locate table candidate areas and reference candidate tags, and establish position indexes as the basis for structure extraction and relationship parsing;

[0085] Step 300: Extract the table structure based on the candidate table area, determine the table header, cell coordinates and merging relationships, and parse cross-document reference relationships based on the reference candidate tags;

[0086] Step 400: Perform terminology normalization and reference resolution on sentence elements, paragraph headings, table cell text, and reference target names to generate a terminology table and entity mapping;

[0087] Step 500: Generate semantic representation based on semantic model, apply consistency constraints to title, body text and table to obtain unified semantic vector and backtracking index, and at the same time establish semantic index, keyword index and relation index, generate unique traceability identifier and store them together;

[0088] Step 600: Perform intent recognition and constraint interpretation, retrieve and sort by relevance, credibility and novelty, and output results with confidence scores;

[0089] Step 700: Perform incremental updates and consistency checks under add, modify, or delete events. If the update fails, roll back and mark the exception.

[0090] In one specific embodiment, step 100 of the present invention preferably involves collecting files and metadata from multiple sources under a unified time base and identification rules. First, by setting the collection source and synchronization method, it is ensured that the time information of files from different sources can be aligned to a unified time scale. For example, a network time protocol server is used as a reference clock to adjust the generation and arrival times of each source file, resulting in a unified timestamp. Based on this, a globally unique identifier is generated for each collected file and its metadata according to the identification rules. This globally unique identifier is an encoding composed of a source identifier, corrected time information, and a content digest, used to ensure that the file maintains its uniqueness and traceability across sources and versions. In this way, the file can achieve source controllability and version differentiation when entering subsequent processing stages.

[0091] In another specific embodiment, step 100 of the present invention further completes the standardization of file format, character set, and fields. Specifically, the collected files are first converted into a unified intermediate representation structure according to their type (such as text documents, spreadsheet files, or presentations). This intermediate representation structure is an abstract data expression method used to preserve paragraph hierarchy, page numbering, and reference relationships of embedded resources. After format normalization, the system automatically detects the file character set and converts it to the target character set encoding, while replacing or cleaning up abnormal characters to ensure semantic continuity. Subsequently, normalized mapping is performed on the metadata fields, unifying the title, author, creation time, version number, and source path into a preset field set. Finally, using a globally unique identifier as the primary key, the intermediate representation structure is used as the content body. Combined with the unified character set encoding and normalized metadata fields, page number, paragraph number, position mapping, table cell coordinates and merge relationship mapping, as well as reference pointers and version information of embedded resources are written to form a normalized object containing a unified format, unified character set, and unified fields. The standardized object is a standardized data unit that can be directly called in subsequent semantic analysis, index construction and query retrieval stages, thereby achieving structural consistency and content traceability of the file.

[0092] In one specific embodiment, step 200 of the present invention preferably performs page segmentation and hierarchical modeling, dividing the normalized object into sentence sequence and paragraph hierarchy. Specifically, firstly, based on page boundaries and text connectivity, the normalized object is divided into pages, columns, blocks, and rows, and the reading order is determined, thereby maintaining the contextual relationship of the document in the original layout. On this basis, using sentence segmentation rules and word boundary rules, the inline text is further subdivided into sentence sequence, and the start and end offset positions are marked for each sentence. Here, a sentence is a custom minimum text unit, usually composed of words or phrases, whose function is to provide fine-grained content to support subsequent semantic processing and position mapping. At the same time, combined with indentation depth, line spacing variation, heading style, and numbering hierarchy, a hierarchical paragraph tree structure is generated, and each paragraph is assigned a paragraph number to ensure the complete reconstruction of the document's logical structure.

[0093] In another specific embodiment, step 200 of the present invention, while obtaining the sentence sequence and paragraph hierarchy, further performs positioning processing on tables and citations. Specifically, this includes: identifying candidate table regions through line segment continuity, cell alignment, grid periodicity, and table header features, and outputting the rectangular range of the table and grid estimation parameters for subsequent table structure extraction. Simultaneously, it identifies candidate citation markers in the document using citation number patterns, author year patterns, anchor text features, and chart number patterns, and binds them to adjacent sentence elements. Finally, using sentence offset as the primary key, a position index is established. This index maps page numbers, column numbers, block numbers, row numbers, paragraph numbers, and table candidate coordinates, generating a traceable position index. Here, the position index is a custom mapping table used to bind semantic fragments to their specific positions in the document layout. Its function is to provide accurate tracing basis for subsequent structure extraction and cross-document relationship parsing.

[0094] In one specific embodiment, step 300 of the present invention preferably utilizes a position index to perform boundary refinement and skew correction on the candidate table region. Specifically, the position index is a mapping table established with sentence offset as the primary key, which can correspond one-to-one with document content, page number, paragraph number, and table candidate coordinates, thereby providing a positional basis for subsequent structured processing. By calling this position index, the boundaries of the initially identified candidate table regions can be corrected, and skew errors in scanned documents or electronic layout files can be eliminated through geometric transformations, thereby obtaining the bounding rectangle range of the table and the initial number of rows and columns of the grid.

[0095] In another specific embodiment, step 300 of the present invention further generates a cell structure based on the candidate table region with boundary correction. Specifically, this includes: determining row and column dividing lines according to line segment continuity and alignment constraints, and outputting the top-left and bottom-right coordinates of each cell, while assigning row and column numbers to the cells. Based on this, according to the characteristics of missing boundaries between adjacent cells and cross-domain text extension, it is determined whether there is a merging relationship across rows or columns, thereby forming a merging span and target coordinate set. Subsequently, combining prior knowledge of the table header position, differences in format styles, and a preset field dictionary, the table header region is identified, and a field-to-column mapping relationship is established between the table header cells and data cells to achieve semantic organization of the table.

[0096] In another specific embodiment, step 300 of the present invention integrates the above extraction results to form a table structure model. The table structure model is a structured description oriented towards computer processing, including the number of rows and columns, cell coordinates, merge span, and the mapping relationship between fields and columns. It also establishes links with corresponding page numbers and paragraph numbers, thereby ensuring that the table's position information in the original document is traceable. Through the table structure model, the system can not only accurately reconstruct the geometric shape of the table but also combine the table header with the data content, supporting subsequent semantic analysis and index construction.

[0097] In another specific embodiment, step 300 of the present invention preferably parses cross-document citation relationships based on citation candidate tags. First, the citation candidate tags are standardized, including unifying the numbering format, unifying the author year format, and cleaning anchor text to extract the citation key, target title, and target number. Then, a search is performed in the hierarchical index, prioritizing matching globally unique identifiers and version information. If multiple candidates exist, disambiguation is performed based on source credibility and time freshness. Finally, a one-to-one or one-to-many link is established between the citation source sentence element and the globally unique identifier of the cited target, and the offset of the source sentence element and the target position coordinates are recorded, thereby realizing a traceable cross-document citation relationship. Here, citation candidate tags refer to reference clues identified in a document through specific patterns, such as serial numbers, document numbers, or figure / table numbers, whose function is to help the system automatically determine the citation and cited relationships between documents.

[0098] In one specific embodiment, step 400 of the present invention preferably involves term extraction and preprocessing of sentence elements, paragraph titles, table cell text, and referenced target names. Specifically, the system first identifies candidate terms from the above content using rules and word segmentation algorithms, and marks the start and end offset positions of each candidate term in the document. Subsequently, the extracted candidate terms undergo case unification, numerical notation standardization, and character format conversion to form standardized input for subsequent merging. Here, candidate terms refer to text fragments extracted from the document in the preliminary processing stage that may have independent semantics or represent specific entities, and their function is to provide basic corpus for subsequent merging, clustering, and resolution.

[0099] In another specific embodiment, step 400 of the present invention further performs merging and clustering analysis on the candidate term set. The system identifies alias relationships and abbreviation correspondences, establishing a one-to-one correspondence between abbreviations and full names, and merging different spellings into a unified standard name. After merging, the term set is clustered using character similarity, contextual semantics, and header clues, aggregating terms with the same meaning into a cluster, obtaining the clustering result, and further determining the type of each cluster. The type includes technical terms, institution names, field names, and document entries to ensure that terms of different categories can be accurately distinguished during semantic processing. Here, a cluster is a custom term set unit used to represent different forms of the same entity appearing in a document, its function being to ensure the consistency of entity representation.

[0100] In another specific embodiment, step 400 of the present invention performs referential resolution based on the obtained clustering results and generates a terminology list and entity mapping. Specifically, this includes: establishing referential constraints through paragraph order, intra-sentence distance, and intra-column consistency in the table; and prioritizing matching using globally unique identifiers and version information in cross-document scenarios to ensure that the same entity can be accurately tracked across documents. Finally, the system outputs a terminology list based on the resolution results. The terminology list includes the main name, alias relationships, abbreviation and full name relationships, and type information of standardized terms, and assigns an entity identifier to each occurrence. Here, entity mapping refers to the mapping relationship established between each term occurrence in the document and a unique entity identifier. Its function is to provide a traceable and unified identifier foundation for subsequent semantic modeling and retrieval.

[0101] In one specific embodiment, step 500 of the present invention preferably uses sentence elements, paragraph titles, and table cell text, after terminology normalization and referential resolution, as input to call a semantic model to generate corresponding semantic vectors. Here, the semantic model refers to a vectorized model trained through deep learning or other natural language processing algorithms, capable of transforming text content into a multi-dimensional semantic representation. Its function is to capture the semantic similarity and contextual logical relationships between texts. Through this step, the original document content is mapped to a unified semantic space, providing a foundation for subsequent consistency constraints and index establishment.

[0102] In another specific embodiment, step 500 of the present invention applies consistency constraints to the title, body text, and table content based on the generated semantic vectors. Consistency constraints refer to introducing rules into the semantic space to ensure that the representations of different structural units (such as titles and body text, body text and tables) maintain logical alignment and relevance. For example, titles and corresponding paragraphs under the same topic need to maintain a high degree of similarity in semantic vectors, thereby avoiding semantic drift between different parts of the document. This constraint mechanism ensures the semantic coherence and integrity of the overall document content.

[0103] In another specific embodiment, step 500 of the present invention further integrates the semantic vectors after consistency constraints to construct a unified semantic representation and generate a traceable index. The unified semantic representation is a holistic vector expression that integrates title, body text, and table information, used to support global semantic computation. The traceable index is a mapping mechanism used to map the vector positions in the unified semantic representation to the page numbers, paragraph numbers, and table coordinates of the original text, thereby enabling retrieval results to be traced back to specific document fragments. Through this mechanism, the system can directly locate the corresponding source position in the document when answering a query.

[0104] In another specific embodiment, step 500 of the present invention establishes a hierarchical index based on a unified semantic representation and generates a unique traceability identifier. Specifically, the system simultaneously establishes three types of structures: a semantic index, a keyword index, and a relationship index, to support semantic similarity retrieval, keyword matching, and cross-document relationship reasoning. Subsequently, a unique traceability identifier is generated based on the file's source information, version information, and citation relationships, and this identifier is associated with the hierarchical index and the original file storage path. This unique traceability identifier is a globally unique code composed of a source identifier, version number, and time information. Its function is to ensure that the retrieval results not only match correctly at the semantic level but also achieve traceable and verifiable integrated storage at the data level.

[0105] In one specific embodiment, step 600 of the present invention first receives a query request, performs syntactic parsing and semantic element extraction on the natural language expression, identifies the query topic, the object of interest, and the core objective, and generates an intent representation for subsequent retrieval. The intent representation is a structured expression comprising four parts: subject terms, task type, target attributes, and expected evidence. It is used to stably map the natural language question to the internal variable space required for retrieval and ranking. Subject terms are used to limit the object of discussion, task type is used to distinguish between search, comparison, or interpretation, target attributes are used to specify the required fields or fragments, and expected evidence is used to specify the required fragment-level backtracking or reference links.

[0106] In another specific embodiment, step 600 of the present invention identifies and standardizes constraints based on the intent representation, forming a constraint set. The constraint set consists of range constraints, timeliness constraints, and quality requirements. Range constraints limit the searchable set of sources and document types; timeliness constraints limit the time interval and version window; and quality requirements set a lower limit for source credibility and evidence tracing requirements. The system converts the user-defined time, source, and version preferences into internal thresholds, transforms fuzzy expressions into specific intervals, and aligns quality requirements with available tracing information so that they can be uniformly invoked during the retrieval and sorting stages.

[0107] In another specific embodiment, step 600 of the present invention calls the semantic index, keyword index, and relation index based on the intent representation and constraint set to generate a candidate result set. The semantic index is used to retrieve paragraphs and table units most similar to the intent in the vector space; the keyword index is used to match key terms with their synonym standard spellings; and the relation index is used to discover cross-document entity associations and reference chains. The results returned by the three types of indexes are deduplicated and merged using globally unique source identifiers; the system calculates a semantic relevance score, a source credibility score, and a version freshness score for each candidate. Source credibility is scored based on source category, publication channel, and historical consistency to measure the reliability of information; version freshness is scored based on document timestamp and version chain position to measure the recentity and timeliness of the content.

[0108] In another specific embodiment, step 600 of the present invention performs a comprehensive weighted ranking of candidate results and outputs a set of search results with confidence levels. The comprehensive weighted ranking dynamically allocates weights for semantic relevance, source credibility, and version freshness based on intent type and constraint set, ensuring that evidence most consistent with the query target is displayed first. The system calculates a confidence level for each result, which represents the degree to which the result satisfies the intent and constraints, comprehensively considering multi-index consistency, score stability, and source traceability completeness; the confidence level is increased when the result simultaneously satisfies range constraints and timeliness constraints and has a traceable location. The final output includes a set of results containing ranking position, confidence level value, source information, version information, and backtracking links, allowing users to directly locate the original text segment corresponding to the page number, paragraph number, and table coordinates, achieving integrated retrieval and verification.

[0109] In one specific implementation, step 700 of the present invention uses file event monitoring as the trigger source. When a file addition, modification, or deletion is detected, an update process is initiated to locate the affected file object and the semantic index, keyword index, and relationship index associated with that object. To avoid full reconstruction, this embodiment first locates the previous valid version of the target object based on the unique source identifier and version chain, and identifies the range of changes at the granularity of page number, paragraph number, and table coordinates, forming a candidate set of changed segments. Here, the unique source identifier is a global code composed of source identifier, version number, and time information, used to uniquely identify and backtrack the same object under cross-source and multi-version conditions.

[0110] In another specific embodiment, step 700 of the present invention performs difference calculation on the candidate set to obtain incremental update records. An incremental update record is a set of structured instructions describing the entries to be added, replaced, or deleted in the semantic index, keyword index, and relationship index, as well as the corresponding vector positions and their mappings to the original text positions. The specific steps are: extracting newly added or changed sentence elements, paragraph titles, and table cell text from the affected objects, regenerating semantic vectors, and matching them with existing indexes; for the relationship index, re-parseing link relationships based on the affected citation keys, target titles, and target numbers; for the keyword index, updating the inverted list of standard names according to the terminology and entity mapping. The above results are summarized into incremental update records, ready for the verification stage.

[0111] In another specific embodiment, step 700 of the present invention performs a consistency check on the incremental update record. The check includes three aspects: first, content consistency, checking whether the updated index and the latest file content are consistent in terms of page number, paragraph number, and table coordinates; second, version consistency, checking whether the version number referenced by the index is consistent with the latest node in the version chain; and third, relationship consistency, checking whether cross-document links point to a unique traceability identifier that exists and can be traced back. If the consistency check passes, the incremental update record is atomically committed to the index and storage system, and the version pointer of the unique traceability identifier is moved forward to the latest node. If the consistency check fails, it is rolled back to the rollback point that passed the check most recently, and an exception mark is generated for the content that failed the check. The exception mark records the reason for failure, the affected index category, and the position mapping, for subsequent review and correction. Here, the rollback point refers to the snapshot of the index and version pointer saved when the last consistency check passed, which is used to restore the system to a consistent state when the commit fails.

[0112] Corresponding to the above methods, such as Figure 2 As shown, this embodiment also provides a system based on NLP file analysis, including:

[0113] The data acquisition and normalization unit is used to collect multi-source files and metadata under a unified time base and identification rules, complete the format, character set and field standardization, and generate normalized objects;

[0114] The page layout segmentation and hierarchical modeling unit is used to perform page layout segmentation and hierarchical modeling, dividing the normalized object into sentence sequence and paragraph level, locating table candidate areas and reference candidate markers, and establishing a position index as the basis for structure extraction and relationship parsing.

[0115] The table structure extraction and reference parsing unit is used to extract the table structure based on the table candidate area, determine the table header, cell coordinates and merging relationship, and parse the cross-document reference relationship based on the reference candidate mark;

[0116] The terminology normalization and reference resolution unit is used to perform terminology normalization and reference resolution on sentence elements, paragraph headings, table cell text and reference target names, and generate a terminology table and entity mapping;

[0117] The semantic representation and index building unit is used to generate semantic representations based on semantic models, apply consistency constraints to titles, body text and tables, obtain unified semantic vectors and backtracking indexes, and simultaneously build semantic indexes, keyword indexes and relationship indexes, generate unique traceability identifiers and store them in association;

[0118] The intent recognition and retrieval ranking unit is used to perform intent recognition and constraint interpretation, retrieve and rank results by relevance, confidence and novelty, and output results with confidence scores.

[0119] The incremental update and consistency maintenance unit is used to perform incremental updates and consistency checks under add, modify, or delete events, and roll back and mark the exception when it fails.

[0120] The beneficial effects of this invention are as follows:

[0121] (1) By collecting and standardizing multi-source files and metadata under a unified time base and identification rules, this invention can eliminate parsing obstacles caused by format differences and inconsistent character sets, align heterogeneous documents under the same standard, reduce content fragmentation and repetitive processing from the source, and improve the stability and reusability of subsequent structuring and retrieval processes.

[0122] (2) This invention restores the document into a hierarchical structure of pages, columns, blocks, rows and paragraphs by dividing the page and hierarchical modeling. Based on this, it locates the candidate areas of the table and the candidate tags of the reference. It can accurately restore the organizational relationship of the document at the fragment level, and provide clear spatial anchors and order constraints for table structure extraction and cross-document reference parsing, thereby improving the accuracy and traceability of structure recognition.

[0123] (3) This invention establishes a glossary and entity mapping by performing terminology standardization and reference resolution on sentence elements, paragraph titles, table cell text and reference target names. It can unify aliases, abbreviations and multiple spellings into standard names and maintain the continuous representation of the same entity in the text and cross-document scenarios, significantly reducing the interference of semantic ambiguity on indexing and retrieval, and improving the consistency and interpretability of the results.

[0124] (4) This invention generates semantic representations based on semantic models and applies consistency constraints to titles, body text and tables. It further constructs a hierarchical structure of semantic index, keyword index and relation index, which can support multi-perspective retrieval and relation reasoning while maintaining global semantic coherence. With the unique source identifier and backtracking index, it realizes a one-to-one correspondence between the query results and the original text position, and enhances the retrieval accuracy and evidence traceability.

[0125] (5) In the query stage, the present invention performs intent recognition and constraint interpretation, structuring the topic, scope, timeliness and quality requirements into computable conditions, and performing comprehensive weighted sorting based on relevance, source credibility and version freshness. This enables the system to return a result set with reasonable priority, sufficient evidence and traceability under multiple constraints, thereby improving the usability and decision support effect in high-requirement scenarios.

[0126] (6) The present invention adopts an incremental update and consistency verification mechanism in the case of adding, modifying or deleting events. If the verification passes, it is atomically committed; if it fails, it is rolled back and marked as abnormal. It can maintain the consistency and reliability of the index and traceability system in a continuously changing document environment, significantly reduce the cost and risk of full reconstruction, and improve the system's operating efficiency and maintainability.

[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0128] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for NLP file analysis based, characterized in that, Comprise: Collecting multi-source files and metadata under unified time base and identification rules, completing format, character set and field specification, and generating normalized objects; Performing layout segmentation and hierarchical modeling, cutting the normalized objects into sentence element sequences and paragraph levels, positioning table candidate regions and reference candidate markers, and establishing position indexes as the basis for structure extraction and relationship analysis; Extracting table structures based on the table candidate regions, determining table headers, cell coordinates and merging relationships, and analyzing cross-document reference relationships based on the reference candidate markers; Performing term normalization and anaphora resolution on sentence elements, paragraph titles, table cell texts and reference target names, and generating a term table and entity mapping; Generating semantic representations based on semantic models, imposing consistency constraints on titles, main texts and tables, obtaining unified semantic vectors and backtracking indexes, and simultaneously establishing semantic indexes, keyword indexes and relationship indexes, generating unique traceable identifiers and associated storage; Performing intent recognition and constraint interpretation, retrieving and sorting by relevance, credibility and freshness, and outputting results with confidence; Performing incremental updates and consistency checks under addition, modification or deletion events, and rolling back and marking exceptions when failed.

2. The method for NLP file analysis based on claim 1, characterized in that, Collecting multi-source files and metadata under unified time base and identification rules, completing format, character set and field specification, and generating normalized objects, comprising: Determine the collection source and synchronization method, adjust the file timestamps of each source according to the unified time base, and ensure that different source files are aligned under the same time scale; Generate a globally unique identifier for each file and metadata collected according to the identification rules, and bind the source identifier and time information together; Convert the collected files according to their types to a preset intermediate representation structure, while keeping the paragraph hierarchy and resource reference relationship unchanged; Detect and unify the character set of the file, convert it to the target character set encoding, and replace or clean up abnormal characters to ensure semantic consistency; Normalize the metadata fields, align the title, author, creation time, version number and source path to the unified field standard; Take the globally unique identifier as the primary key, take the intermediate representation structure as the content body, merge the target character set encoding and the normalized metadata fields, write in the page number, paragraph number, position mapping, table cell coordinate and merging relationship mapping, and embed the reference pointer, source path and version information of the resource, forming the normalized object containing unified format, unified character set and unified field.

3. The method for NLP file analysis based on claim 1, characterized in that, Performing layout segmentation and hierarchical modeling, cutting the normalized objects into sentence element sequences and paragraph levels, positioning table candidate regions and reference candidate markers, and establishing position indexes as the basis for structure extraction and relationship analysis, comprising: Divide the normalized objects into pages, columns, blocks and lines according to page boundaries and text connectivity, and determine the reading order; Convert the text in the line into sentence element sequences according to punctuation and word boundary rules, and label the start offset and end offset for each sentence element in the sentence element sequence; According to the indentation depth, line spacing changes, title style and numbered levels to generate paragraph tree structure, and assign paragraph numbers to each paragraph; According to the line segment continuity, cell alignment, grid periodicity and table header characteristics to determine the table candidate area, output the rectangular range and grid estimation parameters; According to the reference number mode, author year mode, anchor text features and chart number mode to identify the reference candidate markers, and bind them with adjacent sentence elements; Take sentence element offset as the primary key, establish the mapping of page number, column number, block number, line number, paragraph number and table candidate coordinates, and generate the position index which can be traced back.

4. The method for NLP file analysis based on claim 1, characterized in that, Based on the table candidate area, extract the table structure, determine the table header, cell coordinates and merging relationship, and analyze the cross-document reference relationship based on the reference candidate markers, including: Use the position index to perform boundary refinement and tilt correction on the table candidate area, obtain the table bounding rectangle and the initial number of rows and columns of the grid; According to the line segment continuity and alignment constraints, generate row and column division lines, output the top-left and bottom-right coordinates of each cell, and assign row and column numbers; According to the adjacent cell boundary missing and text cross-domain extension features, determine the row and column merging, form the merging span and target coordinate set; According to the table header position priori and style difference and field dictionary to determine the table header area, establish the mapping relationship of field to column between the table header cell and data cell; Summarize the number of rows and columns, coordinates, merging span and field to column mapping to form the table structure model, and establish the link with the corresponding page number and paragraph number; For the reference candidate markers, perform number format unification, author year format unification and anchor text cleaning, extract the reference key, target title and target number; According to the reference key, target title and target number, search for candidate targets in the hierarchical index, preferentially match the globally unique identifier and version information, and disambiguate when not unique according to source credibility and time freshness; Establish one-to-many or one-to-one link between the reference source sentence element and the cited target global unique identifier, and record the source sentence element offset and target position coordinates.

5. The method for NLP file analysis based on claim 1, characterized in that, Perform term standardization and anaphora resolution on sentence elements, paragraph titles, table cell texts and reference target names, generate a mapping of terms and entities, including: Extract candidate terms from sentence elements, paragraph titles, table cell texts and reference target names, and uniformly process case, number notation and character format to provide standardized input for subsequent merging; Identify alias and abbreviation relationships based on candidate terms, correspond abbreviations to full names, and merge different notations into standard names; After completing alias merging, perform clustering analysis on the term set, combine character similarity, context semantics and table header clues to group co-referential terms into a cluster, obtain the clustering result, and further determine the type; Types include: technical terms, organization names, field names and literature items; After obtaining the clustering result, perform anaphora resolution, establish co-reference constraints through paragraph order, intra-sentence distance and table column consistency, and prioritize global unique identifier and version information when cross-document to ensure that the same entity is accurately tracked; The term table is generated according to the result of the resolution, and the main name of the standardized term, the alias relationship, the abbreviation and full name relationship, and the type information are output, and an entity identifier is assigned to each occurrence, and a mapping from the sentence element, paragraph title, table cell text, and reference target name to the entity is established.

6. The method for NLP file analysis based on claim 1, characterized in that, A semantic representation is generated based on a semantic model, consistency constraints are imposed on the title, body, and table, unified semantic vectors and backtracking indexes are obtained, and semantic indexes, keyword indexes, and relationship indexes are established at the same time, unique traceable identifiers are generated and stored in association, including: Taking the sentence element, paragraph title, and table cell text after term normalization and anaphora resolution as input, the corresponding semantic vector is generated by calling the semantic model; Consistency constraints are imposed on the title, body, and table content in the generated semantic vector to ensure alignment and association of different structural units in the same file in the semantic space; Based on the semantic vectors after consistency constraints, a unified semantic representation is constructed, and a backtracking index is generated to correspond to the page number, paragraph number, and table coordinates of the original text; Based on the unified semantic representation, three types of hierarchical index structures, semantic indexes, keyword indexes, and relationship indexes, are established to support multi-dimensional retrieval and reasoning; According to the source information, version information, and reference relationship, a unique traceable identifier is generated, and the unique traceable identifier is associated with the hierarchical index and the storage path of the original file to realize the integrated storage of semantic content and traceable information.

7. The method for NLP file analysis based on claim 1, characterized in that, Performing intent recognition and constraint interpretation, retrieving and sorting by relevance, credibility, and freshness, outputting results with confidence, including: Receiving a query request, parsing natural language expressions, extracting query topics and core targets, and obtaining intent representations; Identifying constraint conditions based on the intent representations to form a constraint set containing range, timeliness, and quality requirements; Calling the semantic indexes, keyword indexes, and relationship indexes, retrieving a candidate result set according to the intent representations and the constraint set; Comprehensively weighting and sorting the candidate result set according to semantic relevance, source credibility, and version freshness to obtain a sorted result; Calculating the confidence of each candidate result in the sorted result and outputting a set of retrieval results with confidence for user to call.

8. The method for NLP file analysis based on claim 1, characterized in that, Incremental updating and consistency checking are performed under the events of adding, modifying, or deleting, and rollback and exception marking are performed when failed, including: Triggering the updating process when a file has an adding, modifying, or deleting event, identifying the affected file objects and associated indexes; Extracting the changed segments based on the affected file objects, generating incremental update records, and comparing with existing indexes; Performing consistency checking to check whether the semantic indexes, keyword indexes, and relationship indexes are consistent with the latest file content, version information, and reference relationship; If the consistency check passes, the incremental update result is written into the index and storage system, and the unique traceable identifier is updated; If the consistency check fails, rollback to the last consistent state, and generate an exception mark for the content that fails to pass the check.

9. A system for NLP file analysis based, characterized by, Including: The data collection and normalization unit is configured to collect multi-source files and metadata under a unified time reference and identification rule, complete format, character set and field specification, and generate normalized objects; The layout segmentation and hierarchical modeling unit is configured to perform layout segmentation and hierarchical modeling, cut the normalized objects into sentence element sequences and paragraph levels, locate table candidate regions and reference candidate markers, and establish a position index as a basis for structure extraction and relationship analysis; The table structure extraction and reference analysis unit is configured to extract table structure based on the table candidate regions, determine table headers, cell coordinates and merging relationships, and analyze cross-document reference relationships based on the reference candidate markers; The term normalization and anaphora resolution unit is configured to perform term normalization and anaphora resolution on sentence elements, paragraph titles, table cell texts and reference target names, and generate a term table and a mapping of entities; The semantic representation and index construction unit is configured to generate semantic representations based on a semantic model, impose consistency constraints on titles, main texts and tables, obtain unified semantic vectors and backtracking indexes, and simultaneously establish semantic indexes, keyword indexes and relationship indexes, generate unique traceable identifiers and store them in association; The intent recognition and retrieval ranking unit is configured to perform intent recognition and constraint interpretation, retrieve and rank according to relevance, credibility and freshness, and output results with confidence; The incremental update and consistency maintenance unit is configured to perform incremental update and consistency check under the events of addition, modification or deletion, and roll back and mark exceptions when failed.

Images