A tobacco professional knowledge document processing and analysis method based on multi-level feature aggregation

Abstract

The application discloses a tobacco professional knowledge document processing and analysis method based on multi-level feature aggregation, and relates to the technical field of natural language processing. The method can adapt to tobacco professional knowledge documents with different degrees of structuring through intelligent document classification and differentiated preprocessing; can capture semantic and knowledge features of different granularities of the documents in layers through three-level feature progressive extraction and cross-level feature fusion; can realize accurate blocking in line with the knowledge semantic boundary based on the fused multi-level features through semantic segmentation matching the corresponding processing pipeline, so as to guarantee the semantic independence and integrity of each document block; can realize the uniformity of professional expression and the enrichment of information dimension of the document block content through term standardization, entity link processing and context enhanced information supplement; and can convert the standardized knowledge block into vector data which can be efficiently searched and accurately called through vectorization processing and writing of the vector library of the supporting hierarchical feature metadata.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation. Background Technology

[0002] In practical applications of information extraction, knowledge organization, and intelligent retrieval from unstructured and semi-structured professional knowledge documents in the tobacco industry, existing technologies exhibit the following shortcomings due to complex conditions such as diverse document formats, significant differences in layout styles, ambiguous semantic boundaries, inconsistent terminology, and strong logical connections between knowledge units: First, existing technologies in the pre-processing stage of tobacco professional documents mostly adopt generalized text extraction and cleaning schemes. These schemes fail to perform differentiated parsing and preprocessing operations based on the characteristic differences between structured and unstructured tobacco documents. They are ill-suited for various document formats such as PDF, Word, and handwritten notes, as well as complex layouts containing tables, images, and special symbols. Furthermore, they lack a systematic standardized mapping mechanism for tobacco professional terminology, failing to eliminate semantic ambiguity caused by abbreviations, colloquialisms, and synonyms. This leads to problems such as text parsing errors, omissions in key parameter extraction, and a significant drop in vector retrieval accuracy. Second, in the process of tobacco document content segmentation and knowledge aggregation, existing solutions either rely on manual reading and annotation for information extraction, resulting in low automation and high labor costs, or employ a crude segmentation method with a fixed number of characters / tokens. This lack of a multi-level feature extraction and fusion system to identify text semantic boundaries and capture logical connections between knowledge units easily leads to the fragmentation of complete sentences and paragraphs, damaging the semantic integrity and contextual relevance of tobacco professional knowledge. This results in incomplete information fragments and missing context retrieved by downstream RAG systems, failing to provide high-quality, high-precision structured data support for upper-level intelligent applications.

[0003] Therefore, there is an urgent need for a tobacco professional knowledge document processing and parsing method and system based on multi-level feature aggregation, which can achieve intelligent classification and differentiated preprocessing of tobacco documents, construct a three-level feature aggregation system to accurately capture semantic and logical relationships, drive intelligent semantic segmentation based on multi-level features, simultaneously complete terminology standardization and knowledge aggregation, and be equipped with a closed-loop quality verification and optimization mechanism. This would solve the above-mentioned technical problems, improve the processing efficiency and information utilization of tobacco professional knowledge documents, and ensure the operational effectiveness and practical value of downstream AI applications. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation. This method solves the problems of low utilization rate of unstructured and semi-structured tobacco professional knowledge documents, difficulty in accurately extracting key information and standardizing terminology, and the fact that traditional processing methods easily destroy semantic integrity, thus failing to provide high-quality structured data support for upper-level AI applications in the tobacco field.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation, comprising the following steps: Based on the metadata, format features, and preliminary text content of tobacco professional knowledge documents, tobacco professional knowledge documents are classified into structured documents and unstructured documents, and differentiated preprocessing is performed to obtain preprocessed text data. Based on the preprocessed text data, basic semantic layer features, logical association layer features, and knowledge system layer features are extracted, and feature fusion is performed to obtain document data with multi-level features. The document data, which incorporates multi-level features, is fed into the corresponding processing pipelines for semantic segmentation to obtain document block results. The document chunking results are then subjected to terminology standardization, entity linking, and contextual enhancement information to obtain standardized knowledge blocks. Standardized knowledge blocks are vectorized to generate corresponding vector data. The vector data is then combined with hierarchical feature metadata and written into a vector database for quality assessment. Based on the assessment results, the multi-level feature extraction parameters and semantic segmentation block parameters are optimized.

[0006] The present invention has the following beneficial effects: This invention, through intelligent document classification and differentiated preprocessing, can specifically adapt to tobacco professional knowledge documents with different levels of structure, outputting standardized and low-noise preprocessed text data. Through three-level progressive feature extraction and cross-level feature fusion, it can capture semantic and knowledge features of different granularities within the document, fully preserving information from basic terminology to logical connections and systematic knowledge, constructing a complete knowledge relationship network within the document. By matching semantic segmentation to corresponding processing pipelines, it can achieve precise segmentation that fits the semantic boundaries of knowledge based on the fused multi-level features, fully ensuring the semantic independence and integrity of each document segment. Through terminology standardization, entity linking, and contextual enhancement, it can achieve unified professional expression and enriched information dimensions for document segment content, forming standardized knowledge blocks with high standardization and complete information. Through vectorization processing and writing to a vector library with accompanying hierarchical feature metadata, standardized knowledge blocks can be transformed into vector data that can be efficiently retrieved and accurately invoked, improving the accuracy and stability of the entire document processing process.

[0007] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0008] Figure 1 This is a flowchart of the tobacco professional knowledge document processing and parsing method based on multi-level feature aggregation of the present invention. Detailed Implementation

[0009] Please see Figure 1 This invention provides a technical solution: a method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation, comprising the following steps: Based on the metadata, format features, and preliminary text content of tobacco professional knowledge documents, these documents are classified into structured and unstructured documents. Differentiated preprocessing is then performed to obtain preprocessed text data.

[0010] The specific process is as follows: The computer reads tobacco-related professional knowledge documents through a document parsing interface, compatible with PDF, DOCX, Markdown, and TXT formats, and also supports scanned images and handwritten notes. If the document is an editable native electronic document, the entire original character content is extracted directly through the native document parsing interface as the initial text content, along with the original metadata and layout features. If the document is a scanned or non-editable image, optical character recognition (OCR) technology is used to recognize characters line by line on each page, generating the recognized full character content as the initial text content. Simultaneously, the document's metadata and layout features are reconstructed based on the recognition results.

[0011] The extracted metadata, layout features, and preliminary text content are quantified to generate a quantitative feature set for classification decisions.

[0012] First, metadata quantification: Document format type, total number of paragraph tags, total number of page numbers, and number of title tags are extracted from the metadata. This data is then normalized to obtain a subset of metadata quantified features. Specifically, the document format type is assigned a value of 1 for natively editable PDF, Word, and Excel formats, and 0 for scanned PDFs, image formats, and plain text documents without formatting. The ratio of the total number of valid paragraph tags to the total number of page numbers is recorded as the paragraph density index. The ratio of the number of title tags to the total number of paragraph tags is recorded as the title level density index. The format type assignment, paragraph density index, and title level density index together form the metadata quantified feature subset. All indicators in this subset are normalized to a value range of 0 to 1 and participate in the weighted calculation for subsequent classification decisions, while also being used in the preprocessing stage for the corresponding document type.

[0013] Second, quantitative processing of layout features: From layout features, extract heading level format differentiation, paragraph indentation consistency, table and list element ratio, and font size differentiation. Normalize these features to obtain a subset of quantitative layout features. Specifically: For heading level format differentiation, count the number of text lines in the document with bold font, enlarged font size, and numbering prefixes, and calculate the ratio to the total number of text lines. Simultaneously, verify whether the font size gradient of adjacent heading lines conforms to a fixed decreasing rule. The ratio of the number of heading lines conforming to the rule to the total number of heading lines is recorded as the heading level compliance, and these are combined to generate the heading level format differentiation index. For paragraph indentation consistency, count the number of paragraphs with consistent first-line indentation to the total number of paragraphs, and record this as the paragraph indentation consistency index. For the proportion of table and list elements, count the total number of closed table border areas and the total number of ordered / unordered list numbering groups in the document, and calculate the ratio of the number of text lines occupied by table and list content to the total number of text lines, recording this as the standardized element proportion index. Regarding font and font size differentiation, the ratio of the number of text lines using a uniform font and font size in the main body of the document to the total number of text lines in the main body is recorded as the main body format consistency index. The heading level format differentiation, paragraph indentation consistency, proportion of standardized elements, and main body format consistency index together form a quantitative feature subset for layout features. All indicators in this quantitative feature subset are normalized to a numerical range of 0 to 1 and participate in the weighted calculation for subsequent classification decisions, while also being used simultaneously in the preprocessing stage of the corresponding document type.

[0014] Third, preliminary text content quantification: From the preliminary text content, the keyword matching degree of the first line of paragraphs, the proportion of fixed sentence structures, and the semantic integrity of paragraphs are extracted. Normalization is then performed to obtain a subset of quantitative text content features. Specifically, based on a pre-set structured title keyword library for tobacco professional knowledge documents, the number of matches between the first line of paragraph text and the keyword library is counted, and the ratio of the number of matches to the total number of paragraphs is calculated, denoted as the keyword matching degree index. Based on a pre-set structured fixed sentence structure library for tobacco professional knowledge documents, the ratio of the number of sentences matching fixed sentence structures within the document to the total number of sentences is counted, denoted as the fixed sentence structure proportion index. The ratio of the number of paragraphs with complete subject-verb-object structures and semantically complete meaning within the document to the total number of paragraphs is counted, denoted as the paragraph semantic integrity index. The keyword matching degree index, the fixed sentence structure proportion index, and the paragraph semantic integrity index are combined to form the text content quantification feature subset. All indicators in the subset are normalized to a numerical range of 0 to 1 and participate in the weighted calculation for subsequent classification decisions, while also being used simultaneously in the preprocessing stage of the corresponding document type.

[0015] The subsets of metadata quantification features, layout quantification features, and text content quantification features are merged to generate a quantification feature set for classification decisions.

[0016] Based on the quantified feature set, a binary classification decision is performed between structured and unstructured documents. The specific execution process is as follows: The total threshold for determining structured documents is preset to 0.7. Simultaneously, a fixed weight is assigned to each indicator in the quantified feature set. Specifically, the total weight of the metadata quantified feature subset is 0.3, the total weight of the layout feature quantified feature subset is 0.5, and the total weight of the text content quantified feature subset is 0.2. Within the metadata quantified feature subset, the weights for format type, paragraph density, and heading level density are all 0.3. Within the layout feature quantified feature subset, the weights for heading level format differentiation are 0.4, paragraph indentation consistency is 0.2, the proportion of standardized elements is 0.2, and the body text format consistency is 0.2. Within the text content quantified feature subset, the weights of each indicator are evenly distributed.

[0017] Based on the weighted indicators allocated above, the weighted sum of all indicators in the quantitative feature set is calculated to obtain the comprehensive document structure score, with a score range of 0 to 1. If the comprehensive document structure score is greater than or equal to the preset total threshold of 0.7 for determining structured documents, it is determined to be a structured document; otherwise, it is determined to be an unstructured document.

[0018] The process of differential preprocessing is as follows: It performs text and layout extraction, redundant content cleaning, native text validation, table content structure conversion, list hierarchy restoration, and paragraph boundary marking on structured documents.

[0019] Specifically: Text and layout extraction: Using the PyMuPDF or pdfplumber library, based on the restored layout features, extract the text content, table positions, image positions, and heading level format information of the document page by page, restore the original paragraph structure and hierarchical relationship of the text, and output the text content after layout restoration.

[0020] Redundant content cleaning: Traverse the text content after the layout is restored, identify and remove headers, footers, watermarks, and duplicate navigation bar content in full, while removing invalid garbled characters and unprintable control characters. Perform deduplication processing on consecutively repeated invalid symbols, and output the deredundant text content.

[0021] Raw text verification: The deduplicated text content is compared line by line with the original content of the document. Missing or incorrectly identified characters are marked and identified. The text is then completed and corrected based on the document context and a tobacco terminology database. The verified text content is then output.

[0022] Table content structuring transformation: Based on the quantitative table and list element ratio indicators, determine the number and distribution of tables in the document, extract the row and column information, header information and cell content of all tobacco-related tables in the document, establish a key-value pair mapping relationship between the header and cell for each table, convert the tobacco leaf grade indicators, process parameters, chemical composition limits and other contents in the table into structured data in JSON format, while retaining the table title hierarchy information, and output the structured mapping text of the table.

[0023] List hierarchy restoration: Identify the numbering and indentation levels of all ordered and unordered lists in a document, restore the parent-child relationships of the lists, mark the corresponding level number and parent node for each list item, and output the text content after restoring the list hierarchy.

[0024] Paragraph boundary marking: Based on the quantified heading level density index, combined with the document's heading level, paragraph marking, and list level, the start position, end position, heading level, and parent node of each paragraph are marked to complete the solidification of paragraph boundaries. Finally, the full text data of the structured document is output after preprocessing.

[0025] It performs plain text base extraction, optical character recognition error correction, global text normalization, garbled and invalid symbol filtering, sentence-level segmentation and sentence completion, intelligent segmentation of continuous text, contextual semantic coherence verification, and paragraph boundary and meta-information marking for unstructured documents.

[0026] Specifically: Plain Text Base Extraction: Based on the extracted preliminary text content and the quantified document format type assignment results, targeted text extraction is performed. For native unstructured Word / PDF documents, the python-docx or pypdf library is used to extract the full plain text, removing irrelevant style tags, annotations, parenthetical phrases, and handwritten annotations. For speech-to-text, regular expressions are used to match and clean up colloquialisms, speaker tags, invalid pause markers, and repetitive colloquial expressions. For unstructured documents such as images / scanned documents, based on the optical character recognition results, invalid characters generated by image background noise recognition and redundant content from handwritten signature recognition are removed, outputting the plain text base content. The output of this step will be used in all subsequent unstructured preprocessing steps and reused in subsequent feature extraction steps.

[0027] Optical Character Recognition (OCR) Error Correction: Based on a quantified paragraph semantic integrity index, this step, targeting the plain text base content and combining a tobacco professional terminology database and a modern Chinese general vocabulary database, corrects errors in the recognition of similar-looking characters, homophones, missing characters, extra characters, and misspelled characters word by word. It also reassembles and restores text fragments caused by line breaks, page breaks, and sentence breaks, and calibrates tobacco professional terms with handwriting recognition deviations. The corrected text content is then output. The output of this step will be used for subsequent text standardization processing and reused in subsequent terminology standardization steps.

[0028] Global text normalization: This process standardizes full-width / half-width characters, Chinese and English punctuation marks, uppercase and lowercase English letters, and tobacco-related unit symbols within the text. It replaces special spaces, tabs, and line breaks with standard format symbols. It removes duplicate consecutive spaces and line breaks, retaining only spaces between sentences and standard line breaks between paragraphs. The output is the normalized text. The results of this step will be used for subsequent garbled character filtering and segmentation, and will also be reused in subsequent basic semantic layer feature extraction steps.

[0029] Garbled text and invalid symbol filtering: Based on the Unicode encoding range and a whitelist of tobacco-specific symbols, invalid garbled characters, non-printable control characters, meaningless special symbols, and rare symbols not used in tobacco-related fields are filtered out from the text. A full removal operation is performed, retaining only commonly used modern Chinese characters, special symbols necessary for tobacco-related professional terminology, unit of measurement symbols, and numerical symbols. The filtered clean text content is output. The output of this step will be used for subsequent continuous text segmentation processing and will also be reused in subsequent word segmentation and basic feature extraction steps.

[0030] Sentence-level segmentation and sentence completion: Based on the rules of modern Chinese punctuation, this step identifies sentence-ending punctuation marks such as periods, question marks, exclamation marks, and semicolons in the text. Combined with a quantified paragraph semantic integrity index, it performs sentence-level segmentation on the clean text content, splitting continuous text into independent sentence units. It then iterates through all segmented sentences, identifying incomplete sentences caused by line breaks, page breaks, missing information, and colloquial omissions. Based on contextual semantics and tobacco professional knowledge expression norms, it completes the sentence components, ensuring that each sentence has complete semantics and syntactic structure. The completed sentence-level text content is then output. The output of this step will be used for subsequent paragraph segmentation and semantic coherence verification, and will also be reused in subsequent sentence-level feature extraction and logical association similarity calculation steps.

[0031] Intelligent segmentation of continuous text: Based on the sentence-level text content after sentence completion, and combined with a quantified paragraph density index, the semantic topic relevance between adjacent sentences is calculated. Continuous sentences with consistent themes and close logical connections are grouped into the same paragraph. Simultaneously, based on the expression logic of tobacco professional knowledge, continuous sentences of process, cause-and-effect, and parallel categories are aggregated into paragraphs to avoid the fragmentation problem of single-sentence paragraphs. This completes the segmentation of continuous text and outputs the segmented text content. The output of this step will be used for subsequent semantic coherence verification and reused in subsequent paragraph-level feature extraction and semantic jump analysis between paragraphs.

[0032] Contextual Semantic Coherence Verification: For the text content after paragraph segmentation, the semantic coherence and logical connection between adjacent paragraphs and sentences are verified. Abrupt semantic jumps, logical breaks, and disordered word order are identified. Based on the standard expression logic of the tobacco professional knowledge system, the content order is fine-tuned, and all adjustment records are marked to ensure that the adjustments do not change the original semantics of the text, but only optimize semantic coherence. The final output is the semantically coherent text content. The output of this step will be used for subsequent paragraph boundary marking and reused in the subsequent logical connection layer feature extraction step.

[0033] Paragraph boundary and meta-information marking: Mark the start and end positions, document source, core semantic topic, and tobacco professional knowledge related categories of each paragraph after verification. At the same time, bind the generated pipeline identifier to complete the solidification marking of paragraph boundaries of unstructured documents. Finally, output the full text data of unstructured documents after preprocessing.

[0034] Based on the preprocessed text data, basic semantic layer features, logical association layer features, and knowledge system layer features are extracted, and feature fusion is performed to obtain document data with multi-level features.

[0035] Specifically: Character-level features, vocabulary-level features, basic information, and sentence-level features are extracted from the preprocessed text data to obtain basic semantic layer features.

[0036] The specific execution steps are as follows: Character-level feature extraction: Traverse all characters in the preprocessed text data, distinguish between structured and unstructured document text, assign a unique character code to each character, map each character to a fixed-dimensional character vector based on a preset character embedding table, and record the position information, vocabulary, sentence, and paragraph information of each character in the text to generate full character-level features for structured and unstructured documents respectively.

[0037] Lexical-level feature extraction: Based on a pre-defined modern Chinese word segmentation dictionary and a tobacco-specific terminology dictionary, and combined with the positional information of character-level features, forward maximum matching word segmentation is performed on the pre-processed structured and unstructured texts to obtain full word segmentation results. For each word segmentation result, the corresponding part-of-speech tag is labeled, generating part-of-speech tagging results. Based on a pre-defined lexical embedding table, each word segmentation result is mapped to a fixed-dimensional word vector, while recording the positional information, sentence, and paragraph information of each word in the text, generating full lexical-level features for both structured and unstructured documents.

[0038] Basic Information Extraction: Through part-of-speech tagging and keyword matching, three core basic information categories—tobacco-related basic terms, discrete technical parameters, and simple entities—are extracted from the word segmentation results of structured and unstructured documents, respectively, and compiled into a basic feature library. This basic feature library is divided into a structured document basic feature library and an unstructured document basic feature library, which will be continuously reused in subsequent terminology standardization and entity linking stages.

[0039] Sentence-level feature extraction: Based on punctuation rules, all sentences in the preprocessed text data are divided into structured and unstructured segments. For each sentence, a weighted average of the word vectors from the lexical-level features is calculated to obtain the sentence vector. Simultaneously, based on syntactic dependency rules, the subject-verb, verb-object, attributive-head, adverbial-head, and complement relationships among the words within the sentence are analyzed to generate the corresponding syntactic dependency relation results. The position information and paragraph information of each sentence in the text are recorded, generating full sentence-level features for both structured and unstructured documents.

[0040] By summarizing the above character-level features, vocabulary-level features, basic information, sentence-level features, as well as word segmentation results, part-of-speech tagging results, sentence vectors, syntactic dependency relations, and basic feature library, basic semantic layer features are generated and output.

[0041] The preprocessed text data is used to extract logical relationships between sentences, semantic jumps between paragraphs, mid-level logical feature networks, and global contextual logical relationships to obtain logical relationship layer features.

[0042] The specific execution steps are as follows: Sentence-to-sentence logical relationship extraction: For adjacent sentences within the same paragraph, distinguishing between structured and unstructured document text, calculate the cosine similarity between the sentence vectors of the two sentences based on sentence vectors in the basic semantic layer features, obtaining the semantic similarity value between sentences. Based on a pre-defined logical relationship discrimination rule base, and combining the logical connectors of the two sentences and the syntactic dependency results in the basic semantic layer features, determine the type of logical relationship between the two sentences. The logical relationship types include causal relationship, progressive relationship, parallel relationship, adversative relationship, sequential relationship, and explanatory relationship. Bind the corresponding semantic similarity value and logical relationship type to each pair of adjacent sentences, generating sentence-to-sentence logical relationship results for both structured and unstructured documents.

[0043] Semantic jump relationship extraction between paragraphs: For adjacent paragraphs within a document, distinguishing between structured and unstructured document text, and based on sentence vectors from the basic semantic layer features, calculate the weighted average of the sentence vectors of all sentences within a paragraph to obtain the paragraph vector corresponding to that paragraph. Calculate the cosine similarity between the paragraph vectors of two adjacent paragraphs to obtain the semantic similarity value between paragraphs. Based on the magnitude of the semantic similarity value, determine the degree of semantic jump between adjacent paragraphs; a higher semantic similarity value indicates a lower degree of semantic jump, and vice versa. Assign a corresponding semantic similarity value and semantic jump degree to each pair of adjacent paragraphs, generating semantic jump relationship results between paragraphs for both structured and unstructured documents.

[0044] Construction of the intermediate-level logical feature network: Based on the basic feature library in the basic semantic layer, and combined with the results of logical relationships between sentences, this method distinguishes between structured and unstructured document texts, capturing the logical relationships of tobacco professional knowledge within the text. Through relationship mapping, intermediate-level logical feature networks are constructed for structured and unstructured documents respectively, clarifying the relationships between different basic features.

[0045] Global contextual logical relationship extraction: Taking the entire document as the scope, distinguishing between structured and unstructured documents, and constructing sentence-to-sentence semantic relationship matrices and paragraph-to-paragraph semantic relationship matrices based on sentence-to-sentence semantic similarity values ​​and paragraph-to-paragraph semantic similarity values, respectively. Each element in the matrix corresponds to the semantic similarity value between two sentences or two paragraphs. Based on the relationship matrices, identifying semantically related sentence pairs and paragraph pairs across paragraphs and pages, marking their corresponding long-distance logical relationships, and generating global contextual logical relationship results for both structured and unstructured documents.

[0046] By summarizing the logical relationships between sentences, semantic jumps between paragraphs, the mid-level logical feature network, and the global logical relationships of the context, the logical relationship layer features are generated and output.

[0047] The preprocessed text data is analyzed to extract the knowledge category of professional terms, the hierarchical relationship of professional concepts, the aggregation result of complete knowledge units, the domain of knowledge nodes, the upstream and downstream relationships of the knowledge system, and the cross-document knowledge relationships, thus obtaining the knowledge system layer features.

[0048] The specific execution steps are as follows: A pre-defined tobacco professional knowledge system ontology library is established, which includes a hierarchical tree of tobacco professional knowledge categories, a set of standard terms corresponding to each category, a definition of knowledge nodes corresponding to each category, and the upstream and downstream relationships between knowledge nodes. It is also linked to the enterprise's internal tobacco leaf database, process standard library, and product database.

[0049] For each text segment in the preprocessed text data, structured and unstructured documents are distinguished, and the corresponding basic semantic layer features and logical association layer features are integrated. The weight of the core knowledge units in the text is strengthened through an attention mechanism. Based on the tobacco professional knowledge system ontology, combined with the basic feature library in the basic semantic layer features, entity matching and concept mapping processing between the text and knowledge categories are performed. The number of matching hits between the terms in the text and the standard terms under each category in the ontology is counted, and the ratio of the number of matching hits to the total number of terms in the text is calculated to obtain the matching degree value between the text and each knowledge category.

[0050] The knowledge category with the highest matching score is selected as the knowledge category to which the professional term in the text belongs. At the same time, the hierarchical position, superior category, and subordinate category of this knowledge category in the category hierarchy tree of the tobacco professional knowledge system are recorded to generate the professional concept hierarchy relationship results.

[0051] Based on the basic feature library in the basic semantic layer features, the mid-level logical feature network in the logical association layer features, and the knowledge category of the text, this method distinguishes between structured and unstructured documents, integrates scattered basic information and logical relationships into complete knowledge units, generates complete knowledge unit aggregation results, and records the core components, boundary range, and upstream and downstream related knowledge of the knowledge units.

[0052] Based on the tobacco professional knowledge system ontology, the industry fields and application scenarios corresponding to knowledge nodes in the text are determined, and the results of the knowledge node's belonging field and the upstream and downstream association relationship of the knowledge system are generated.

[0053] Based on the tobacco professional knowledge system ontology and the enterprise's internal relational database, this study distinguishes between structured and unstructured documents, identifies the relationships between knowledge units within a text and knowledge units in other documents, and generates cross-document knowledge relational results.

[0054] The system summarizes the knowledge categories to which the above-mentioned professional terms belong, the hierarchical relationship of professional concepts, the aggregation results of complete knowledge units, the domain to which knowledge nodes belong, the upstream and downstream relationships of the knowledge system, and the cross-document knowledge relationships, and generates and outputs the knowledge system layer features.

[0055] The process of performing feature fusion to obtain document data with fused multi-level features is as follows: For structured documents and unstructured documents, vector data is extracted from the basic semantic layer features, logical association layer features, and knowledge system layer features, respectively. Through linear transformation, the vector data is uniformly mapped to the same vector dimension, thus completing the dimension alignment process.

[0056] For dimension-aligned vector data, min-max normalization is performed to map all feature values ​​to the range of 0 to 1, eliminating the difference in numerical units between features at different levels.

[0057] We initialize weights for the vector data of basic semantic layer features, logical association layer features, and knowledge system layer features respectively, with the weight of basic semantic layer features being 0.3, the weight of logical association layer features being 0.3, and the weight of knowledge system layer features being 0.4.

[0058] Based on the assigned weights, the vector data of basic semantic layer features, logical association layer features, and knowledge system layer features are respectively weighted and scaled to obtain weighted vector data.

[0059] The weighted vector data is multiplied pairwise to generate cross-interaction features.

[0060] The weighted vector data and cross-interaction features are concatenated end to end to generate the fused full feature data for structured documents and unstructured documents, respectively.

[0061] The full set of feature data is bound one-to-one with each segment of text in the preprocessed text data. At the same time, each segment of text is bound with the corresponding original features, feature weight information, feature metadata, and pipeline identifier at each level, distinguishing between structured and unstructured documents and generating document data that integrates multi-level features.

[0062] The document data, which incorporates multi-level features, is fed into the corresponding processing pipelines for semantic segmentation to obtain document block results.

[0063] Specifically: The structured data from the document data that integrates multi-level features is fed into the structured document processing pipeline to perform title structure recognition and hierarchical node extraction, title jurisdiction delineation, knowledge boundary matching and verification based on knowledge system layer features, block rule execution and boundary correction, and document block division and verification to obtain structured document blocks.

[0064] The specific execution process is as follows: Title structure recognition and hierarchy node extraction: Based on the title hierarchy format marked during preprocessing, paragraph boundary markers, and list hierarchy restoration results, the entire document content is traversed to identify all H1-H3 level title lines. Based on the font size, number, and format differentiation of the titles, the titles are divided into first-level, second-level, and third-level levels. A unique title hierarchy number is assigned to each title line, and the title text, title start position, and title end position corresponding to each title line are determined, generating all title hierarchy nodes.

[0065] Defining the scope of headings: Based on the hierarchical relationship of headings, with the next higher-level heading as the parent node and the next lower-level adjacent heading as the child nodes, the initial scope of each heading level node is defined. Specifically, the initial scope of a first-level heading is all content from the current heading's start position to the start position of the next first-level heading. The initial scope of a second-level heading is all content from the current heading's start position to the start position of the next second-level heading at the same level, and all content belonging to the same parent first-level heading. This process continues for third-level headings, completing the initial scope definition for all heading level nodes.

[0066] Knowledge boundary matching and verification based on knowledge system layer features: For the text content within the initial jurisdiction of each title-level node, extract the knowledge system layer features corresponding to that content to obtain the tobacco professional knowledge category to which the content belongs, the boundary of the complete knowledge unit, and the upstream and downstream relationships of the knowledge system. Match the knowledge category corresponding to the title text with the knowledge category corresponding to the text content within the jurisdiction, and calculate the category matching degree between the two.

[0067] Segmentation Rule Execution and Boundary Correction: A core segmentation rule is preset, treating each H2 title's corresponding high-level knowledge system as a main knowledge block. If the main knowledge block contains multiple mid-level logical units and has more than 800 tokens, secondary knowledge blocks are split at logical breakpoints. Based on this rule, if the knowledge category of the text content within the title's jurisdiction matches the title text's knowledge category with a match greater than or equal to a preset category matching threshold of 0.8, the initial jurisdiction of that title-level node is retained as the segmentation boundary. If the match is less than the preset category matching threshold of 0.8, or the main knowledge block has more than 800 tokens, the initial jurisdiction is split based on the content's corresponding knowledge category switching node and the logical breakpoints of the mid-level logical units. Continuous content belonging to the same knowledge category and the same complete mid-level logical unit are placed within the same segmentation boundary, while content belonging to other knowledge categories is removed and placed within the jurisdiction of the corresponding title-level node, thus correcting the segmentation boundary.

[0068] Document block segmentation and validation: The segmentation operation is performed collaboratively by LangChain's MarkdownHeaderTextSplitter and a custom RecursiveCharacterTextSplitter. Based on the corrected segmentation boundaries, continuous text content within the same title and belonging to the same knowledge system category is divided into the same document block. Each document block is marked with the corresponding title level node, the knowledge category it belongs to, the block start position, the block end position, the number of tokens, and the pipeline identifier to ensure that the segmented knowledge blocks retain complete hierarchical features. This completes the segmentation of the structured document and outputs the structured document blocks.

[0069] Unstructured data from document data that incorporates multi-level features is fed into an unstructured document processing pipeline. The pipeline performs the following steps: determining candidate semantic nodes based on the distribution of basic semantic layer features; filtering candidate block boundaries based on logical association similarity; verifying and correcting block boundaries based on knowledge system keywords; and dividing and performing fallback verification on document blocks to obtain unstructured document blocks.

[0070] The specific execution process is as follows: Candidate semantic nodes are determined based on the distribution of basic semantic layer features: Basic semantic layer features corresponding to the content of unstructured documents are extracted. Taking sentences as the smallest unit, the cosine similarity between the sentence vector of each sentence and the average sentence vector of the preceding five consecutive sentences is calculated based on the sentence vector in the basic semantic layer features to obtain the semantic density change value of the sentence. A semantic switching threshold of 0.5 is preset. If the semantic density change value of the current sentence is less than the semantic switching threshold of 0.5, and the current sentence ends with a sentence punctuation mark, then the end position of the sentence is marked as a semantic topic switching node and also as a candidate block boundary node, generating a full set of candidate block boundary nodes specific to unstructured documents.

[0071] Candidate block boundary selection based on logical association similarity: Logical association layer features corresponding to the unstructured document content are extracted. For each pair of adjacent candidate block boundary nodes, the average logical association similarity between adjacent sentences within the paragraph is calculated to obtain the semantic tightness value within the paragraph. Simultaneously, based on the paragraph vectors in the logical association layer features, the cosine similarity of the paragraph vectors of the current paragraph and its preceding adjacent paragraphs is calculated to obtain the logical association similarity value between paragraphs. Preset thresholds for paragraph tightness are 0.6 and for inter-paragraph similarity are 0.4. If the semantic tightness value within a paragraph is less than the paragraph tightness threshold of 0.6, and the logical association similarity value between paragraphs is less than the inter-paragraph similarity threshold of 0.4, then the candidate block boundary node at that position is retained. Otherwise, the candidate block boundary node at that position is removed, completing the first round of selection for candidate block boundary nodes.

[0072] Based on knowledge system keyword verification and correction of block boundaries: A pre-set keyword library for tobacco professional knowledge system is used, where each keyword is bound to a corresponding tobacco professional knowledge category. The entire content of the unstructured document is traversed, and combined with the knowledge category information from the knowledge system layer features, all knowledge system keywords matching the keyword library are identified, and the location information and corresponding knowledge category of each keyword are marked. For the candidate block boundary nodes retained after the first round of screening, the core knowledge category corresponding to the text content before and after the node is verified. If the core knowledge category of the text before and after the node changes, the candidate block boundary node is retained. If the core knowledge category of the text before and after the node does not change, the candidate block boundary node is removed. Simultaneously, at the location where a knowledge system keyword is detected and the core knowledge category changes, a new candidate block boundary node is added, completing the final verification and correction of the candidate block boundary nodes and determining the final block boundaries of the unstructured document.

[0073] Document Block Segmentation and Last-Cut Verification: Based on the final determined block boundaries, continuous text content between two adjacent block boundaries is divided into the same document block. Simultaneously, a length constraint last-cut verification is performed. If the number of tokens in a single document block exceeds 512, secondary segmentation is performed at logical breakpoints and sentence-ending punctuation within that document block to ensure that the number of tokens in a single document block does not exceed 512. Each document block is labeled with its corresponding core knowledge category, block start position, block end position, token count, and pipeline identifier, completing the unstructured document block processing and outputting the unstructured document blocks.

[0074] The system aggregates structured and unstructured document blocks, assigns a globally unique block identifier to each document block, and binds corresponding document source information, block boundary location information, hierarchical feature information, knowledge category information, token quantity information, and pipeline identifier to each document block, generating and outputting the document block segmentation results.

[0075] The document chunking results are processed sequentially with terminology standardization and entity linking, and contextual enhancement information is added to obtain standardized knowledge blocks.

[0076] A pre-set standard dictionary of tobacco professional terminology is provided, which stores all standard terms in the tobacco professional field, as well as a set of variant terms corresponding to each standard term. The set of variant terms includes abbreviations, aliases, foreign language translations, colloquial names, production area-specific terms, non-standard expressions within the industry, and handwritten colloquial expressions.

[0077] The entire document block in the document segmentation result is read. Structured document blocks and unstructured document blocks are distinguished according to the pipeline identifier. Taking a single document block as the smallest processing unit, based on the standard dictionary of tobacco professional terminology, combined with the generated basic semantic layer features, basic feature library, and word segmentation results, the forward maximum matching algorithm is used to traverse the entire text content in the current document block. For unstructured document blocks, additional special matching and recognition of handwritten variants, colloquial expressions, and abbreviations are added. Non-standard terms that hit the variant term set in the text are identified, and the position, text content, and corresponding standard terms of each non-standard term are marked.

[0078] Based on the tagging results, all identified non-standard terms within the document block are uniformly replaced with the corresponding standard terms from the tobacco professional terminology standard dictionary. Simultaneously, a replacement mapping relationship is established for each replacement operation, recording the original non-standard term, the replaced standard term, and the replacement location information. For unstructured document blocks, additional standardized mapping relationships for colloquial and handwritten expressions are recorded.

[0079] After completing the full replacement of non-standard terms, the semantic coherence of the text content within the document block is checked to ensure that the term replacement does not change the original semantics of the text. The terminology-standardized document block content corresponding to the structured document block and the unstructured document block is generated respectively, and the corresponding term replacement mapping relationship is stored.

[0080] Based on the standardized document block content, entity linking is performed by combining basic semantic layer features, knowledge system layer features, and basic feature library to generate entity-linked document block content. The specific execution process is as follows: The system includes a pre-defined entity library for tobacco professional knowledge graphs and an internal product database for enterprises. The entity library stores all standard entities in the tobacco professional field. Each standard entity is bound to a unique identifier in the enterprise's internal product database. It also stores the entity name, alias, attribute information, and relationship information for each standard entity, covering standard entities in structured documents and corresponding entities in unstructured documents (including colloquial and handwritten text).

[0081] Based on named entity recognition rules, and combined with the tobacco professional knowledge graph entity library and the basic feature library of the corresponding document blocks generated in step S2, entity recognition is performed on the content of terminology-standardized document blocks. The entire text content within the document block is traversed, and all text fragments conforming to the tobacco professional entity definition are extracted as candidate entities, generating a candidate entity set. These candidate entities include tobacco leaf varieties, production areas, cigarette categories, equipment models, processing steps, chemical components, and names of pests and diseases, among other tobacco-related entities. For unstructured document blocks, additional specialized recognition is added for colloquial field terms and smoking-related descriptive entity expressions.

[0082] For each candidate entity in the candidate entity set, entity matching and disambiguation are performed based on the knowledge system layer features of the corresponding document block: First, the textual and semantic similarity between the candidate entity and each standard entity in the tobacco professional knowledge graph entity library are calculated to obtain a matching score for each standard entity. The top three standard entities with the highest matching scores are selected as candidate matching entities. Second, based on the knowledge system layer features and contextual semantic information corresponding to the current document block, disambiguation is performed on the candidate matching entities. The semantic fit between each candidate matching entity and the context of the current document block is calculated, and the candidate matching entity with the highest fit is selected as the standard entity corresponding to that candidate entity. For unstructured document blocks, a second disambiguation is performed by combining entity association information from other paragraphs in the same document.

[0083] After matching and disambiguating all candidate entities, each professional entity within a document block is bound to a corresponding standard entity unique identifier, an internal product database association ID, entity attribute information, and entity association information, establishing an entity link mapping relationship. The document block content with entity links for both structured and unstructured document blocks is generated, and the corresponding entity link mapping relationships are stored.

[0084] Based on the document block content linked by entities, combined with document source information, basic semantic layer features, logical association layer features, knowledge system layer features, and document segmentation results, context enhancement information is added to generate and output standardized knowledge blocks. The specific execution process is as follows: Extract the basic contextual information corresponding to the current document block, including the original document title, document type, position information within the document to which the current document block belongs, text content and semantic topic information of the preceding and following document blocks, logical relationship information between the current document block and other document blocks within the same document, and knowledge unit type information. For unstructured document blocks, additional contextual metadata such as the core document scenario, recorder information, and record time is added.

[0085] Extract the overall knowledge system affiliation information of the document to which the generated current document block belongs, including the tobacco professional knowledge category to which the document belongs, the core knowledge nodes corresponding to the document, and the knowledge system hierarchy information of the document.

[0086] Extract the upstream and downstream association information of the knowledge graph corresponding to the entity within the generated current document block, including the entity attribute information bound in the entity link mapping relationship, the upstream and downstream associated entity information, and the detailed information of the knowledge category corresponding to the entity.

[0087] The extracted basic contextual information, overall document knowledge system attribution information, and upstream and downstream knowledge graph association information are integrated to generate contextual enhancement information corresponding to the current document block. This contextual enhancement information is then bound one-to-one with the text content, terminology substitution mapping, and entity link mapping of the current document block to complete the addition of contextual enhancement information.

[0088] Each document block is assigned a globally unique knowledge block identifier. At the same time, each document block is bound with a corresponding knowledge block identifier, document source information, term substitution mapping relationship, entity link mapping relationship, context enhancement information, hierarchical feature information, knowledge category information, token quantity information, and pipeline identifier. This distinguishes between structured and unstructured document sources and generates and outputs standardized knowledge blocks.

[0089] Standardized knowledge blocks are vectorized to generate corresponding vector data. The vector data is then combined with hierarchical feature metadata and written into a vector database for quality assessment. Based on the assessment results, the multi-level feature extraction parameters and semantic segmentation block parameters are optimized.

[0090] The system reads all standardized knowledge blocks, distinguishes between knowledge blocks from structured and unstructured document sources according to pipeline identifiers, and uses a single standardized knowledge block as the smallest processing unit. It extracts the standardized core text content of the terms within the current standardized knowledge block, along with the bound contextual enhancement information. These two parts are then concatenated to generate the text to be encoded. For knowledge blocks from unstructured sources, additional weights are applied to the scene information and record information in the contextual enhancement information to ensure that the vectorized result closely matches the core semantics of the unstructured document.

[0091] The Qwen3-Embedding-4B model deployed on the Xinference platform is invoked to perform vectorized encoding on the text to be encoded. Batch encoding is performed using a batch processing approach, with a batch size of 16. Structured and unstructured knowledge blocks are encoded in batches to improve document processing throughput.

[0092] After encoding is completed, dense vector data with a dimension of 2560 is generated, uniquely corresponding to each standardized knowledge block. Simultaneously, model parameters, batch information, encoding time, and document source type are recorded during the vector generation process, completing the vectorization process.

[0093] The vector data, combined with hierarchical feature metadata, is written into the vector database. The specific execution process is as follows: Hierarchical feature metadata is extracted, which originates from core information in document type, basic semantic layer features, logical association layer features, and knowledge system layer features. This includes tobacco production area, tobacco variety, processing steps, knowledge level, core feature tags, weight parameters during feature extraction, feature dimension information, knowledge category information, feature matching degree information, and semantic association information. For knowledge blocks from unstructured sources, additional metadata tags such as scene type and recording subject are added.

[0094] The system associates and binds the unique identifier, vector data, hierarchical feature metadata, original text content, term substitution mapping, entity link mapping, and context enhancement information of the current standardized knowledge block to establish a one-to-one index relationship. Simultaneously, it establishes a mapping relationship between the block ID and the original document source file path, storing this information in a local SQLite metadata database to ensure knowledge traceability.

[0095] Based on the pipeline identifiers corresponding to standardized knowledge blocks and the original document types, the full set of associated and bound data is written in batches to different collections in the ChromaDB vector database. Specifically, standardized knowledge blocks corresponding to structured documents are written to the `technical_guide_collection` collection, while standardized knowledge blocks corresponding to unstructured documents are written to the `case_study_collection` collection. During the writing process, a vector index is simultaneously built based on the vector data, and a multi-dimensional structured index is built based on hierarchical feature metadata, completing the persistent storage of the data and the construction of a searchable index.

[0096] A multi-dimensional quality assessment system is constructed to perform quality assessment on all data written to the vector database. The specific execution process is as follows: A multi-dimensional quality assessment system is pre-designed, comprising five core assessment dimensions: knowledge block integrity assessment, terminology standardization accuracy assessment, entity link accuracy assessment, block boundary rationality assessment, and vector representation effectiveness assessment. Each dimension is assigned equal weight, and the score range for each dimension is 0 to 100. For knowledge blocks from both structured and unstructured sources, scores are calculated for each dimension. The overall quality assessment result is the arithmetic mean of the scores across the five dimensions for both types of documents, with an overall score range of 0 to 100.

[0097] Knowledge Block Integrity Assessment: By matching core keywords in the tobacco industry and combining them with the generated basic feature library, we verify whether key information such as tobacco leaf grade indicators, process parameter values, field diagnostic schemes, and chemical component limits are completely preserved in each standardized knowledge block from both structured and unstructured sources, without any cross-block breaks. The ratio of the number of knowledge blocks meeting the integrity requirements to the total number of knowledge blocks of the corresponding type is calculated. Multiplying this ratio by 100 yields the assessment score for this dimension for both structured and unstructured documents. These scores are then combined to calculate the overall score for this dimension.

[0098] Terminology standardization accuracy assessment: For each standardized knowledge block, based on the tobacco professional terminology standard dictionary and combined with the generated terminology substitution mapping relationship, the accuracy of terminology substitution is cross-validated. The ratio of the number of correct terminology substitutions to the total number of terminology substitutions is calculated. The ratio is multiplied by 100 to obtain the assessment score for this dimension of structured and unstructured documents. The overall terminology standardization rate is calculated simultaneously, and the overall score for this dimension is calculated by combining them.

[0099] Entity link accuracy assessment: For each standardized knowledge block, based on the generated entity link mapping relationship, verify the accuracy of entity matching and disambiguation, the accuracy of association with the enterprise's internal database ID, and calculate the ratio of the number of correct entity links to the total number of entity links. Multiply the ratio by 100 to obtain the assessment score for this dimension for both structured and unstructured documents, and then combine them to calculate the overall score for this dimension.

[0100] Block boundary rationality assessment: For each standardized knowledge block, combined with the generated document block segmentation results and the generated knowledge system layer features, verify the consistency of semantic topics and knowledge categories within the block, and calculate the ratio of the number of knowledge blocks with consistent core semantic topics and knowledge categories to the total number of knowledge blocks of the corresponding type. At the same time, verify the conformity of the block boundaries without semantic fragmentation and knowledge fragmentation. Multiply the arithmetic mean of the two ratios by 100 to obtain the evaluation scores for this dimension of structured and unstructured documents, and combine them to calculate the overall score for this dimension.

[0101] Vector Representation Validity Evaluation: For the vector data corresponding to each knowledge block, combined with the constructed vector index of the corresponding set, small-scale retrieval tests are conducted on samples of both types of documents to evaluate the reasonableness of their vector similarity with other related blocks. The similarity between knowledge block vectors of the same knowledge category and the same semantic topic is verified, as well as the distinguishability between knowledge block vectors of different knowledge categories and different semantic topics. The recall and precision of vector retrieval are calculated, and the arithmetic mean of the recall and precision is multiplied by 100 to obtain the evaluation scores for this dimension of structured and unstructured documents, respectively. The overall score for this dimension is then calculated by combining the scores.

[0102] The overall evaluation scores across the five dimensions are aggregated, the arithmetic mean is calculated, and an overall quality evaluation result is generated. At the same time, detailed evaluation reports by dimension and document type are generated, marking the score deficiencies, corresponding defect data, error type, and document type for each dimension.

[0103] Based on the quality assessment results, the multi-level feature extraction parameters and semantic segmentation block parameters are optimized. The specific execution process is as follows: The preset acceptable quality threshold is 80 points. The overall quality assessment results are compared with the preset acceptable quality threshold. At the same time, a single acceptable threshold of 80 points is set for both structured and unstructured documents. When the score of a single type is lower than the threshold, the corresponding type's dedicated parameter optimization is triggered simultaneously.

[0104] For handling quality defects in a single block: If a block fails the quality assessment for the corresponding dimension, the system automatically triggers a retry mechanism, re-executing the corresponding processing steps for that block. A single block can be retried a maximum of 3 times. Blocks that fail to retry are marked as "awaiting manual review," and detailed error logs are recorded, including the error type, error location, reason for failure, and document type.

[0105] Parameter optimization for overall quality deficiencies: If the overall quality assessment result is below the preset acceptable quality threshold of 80 points, or if a single document type score is below 80 points, the core processing steps corresponding to the quality defects will be located based on the detailed assessment reports by dimension and document type. If the accuracy of terminology standardization, entity linking, and vector representation effectiveness assessment dimensions are not up to standard, the multi-level feature extraction step corresponding to the defect will be located. If the rationality of block boundaries and the completeness of knowledge blocks assessment dimensions are not up to standard, the semantic segmentation block step corresponding to the defect will be located. Simultaneously, the document type to which the defect belongs will be clearly identified, and the specific processing parameters for structured or unstructured documents will be optimized accordingly.

[0106] To address the shortcomings of the multi-level feature extraction process, multi-level feature extraction parameters were optimized: Based on the defective data and its document type, the word segmentation weights, part-of-speech tagging rules, and tobacco terminology matching weights in the basic semantic layer feature extraction process were adjusted accordingly. For unstructured documents, the matching weights for colloquial and handwritten terms were further optimized. The similarity calculation weights, logical relationship discrimination rules, and logical connector matching library in the logical association layer feature extraction process were adjusted. For unstructured documents, the recognition rules for colloquial logical connectors were further optimized. The category matching rules, terminology matching weights, and attention mechanism weight allocation coefficients in the knowledge system layer feature extraction process were adjusted. For unstructured documents, the error tolerance threshold for knowledge category matching was further optimized. The optimized multi-level feature extraction parameters were then updated to the feature extraction processing stages for both structured and unstructured documents.

[0107] To address the deficiencies in the semantic segmentation and chunking process, the semantic segmentation and chunking parameters were optimized: Based on the defective data and the document types, the category matching threshold, title level jurisdiction definition rules, and chunk token quantity threshold were adjusted in the structured document chunking process. The semantic switching threshold, paragraph density threshold, paragraph similarity threshold, knowledge system keyword database, and maximum token fallback threshold were adjusted in the unstructured document chunking process. The optimized semantic segmentation and chunking parameters were then updated in the semantic segmentation and chunking processing stages for both structured and unstructured documents.

[0108] After completing the parameter optimization and update, the tobacco professional knowledge documents processed in this instance are used as the processing objects. The structured and unstructured documents are distinguished, and the entire processing process is repeated until the overall quality assessment result and the score of each type of document are not lower than the preset qualified quality threshold of 80 points.

[0109] If both the overall quality assessment result and the score of a single document type are not lower than the preset qualified quality threshold of 80 points, the entire process of processing and parsing tobacco professional knowledge documents is completed. The current multi-level feature extraction parameters and semantic segmentation block parameters are locked, and the exclusive processing parameters for structured documents and unstructured documents are solidified as the default parameters for subsequent document processing. The system regularly calculates the success rate and average processing time for each module and document type, and dynamically adjusts the processing parameters of each stage based on historical processing data to achieve a dynamic balance between processing speed and processing quality.

[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0115] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation, characterized in that, Includes the following steps: Based on the metadata, format features, and preliminary text content of tobacco professional knowledge documents, tobacco professional knowledge documents are classified into structured documents and unstructured documents, and differentiated preprocessing is performed to obtain preprocessed text data. Based on the preprocessed text data, basic semantic layer features, logical association layer features, and knowledge system layer features are extracted, and feature fusion is performed to obtain document data with multi-level features. The process of extracting basic semantic layer features, logical association layer features, and knowledge system layer features is as follows: Character-level features, vocabulary-level features, basic information, and sentence-level features are extracted from the preprocessed text data to obtain basic semantic layer features; The preprocessed text data is used to extract logical relationships between sentences, semantic jump relationships between paragraphs, mid-level logical feature networks, and global contextual logical relationships to obtain logical relationship layer features. The preprocessed text data is analyzed to extract the knowledge category of professional terms, the hierarchical relationship of professional concepts, the aggregation result of complete knowledge units, the domain of knowledge nodes, the upstream and downstream relationships of the knowledge system, and the cross-document knowledge relationships to obtain the knowledge system layer features. The document data, which incorporates multi-level features, is fed into the corresponding processing pipelines for semantic segmentation to obtain document block results. The document chunking results are then subjected to terminology standardization, entity linking, and contextual enhancement information to obtain standardized knowledge blocks. Standardized knowledge blocks are vectorized to generate corresponding vector data. The vector data is then combined with hierarchical feature metadata and written into a vector database for quality assessment. Based on the assessment results, the multi-level feature extraction parameters and semantic segmentation block parameters are optimized.

2. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of classifying tobacco professional knowledge documents into structured and unstructured documents is as follows: Extract document format type, total number of document paragraph tags, total number of document page numbers, and number of document title tags from the metadata, and normalize them to obtain a subset of metadata quantified features; Extract heading level format differentiation, paragraph indentation consistency, table and list element ratio, and font size differentiation from layout features, and normalize them to obtain a quantitative subset of layout features. Extract the keyword matching degree of the first line of paragraphs, the proportion of fixed sentence patterns in the text structure, and the semantic integrity of paragraphs from the preliminary text content, and obtain a subset of quantitative features of text content by normalization; Merge the subsets of metadata quantified features, the subsets of layout quantified features, and the subsets of text content quantified features to generate a quantified feature set for classification decisions; Calculate the weighted sum of all indicators in the quantitative feature set to obtain the comprehensive score of document structure; If the overall structure score of a document is greater than or equal to the preset total threshold for determining a structured document, it is determined to be a structured document; otherwise, it is determined to be an unstructured document.

3. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of performing differentiated preprocessing is as follows: Extract text and layout from structured documents, clean up redundant content, validate raw text, transform table content into a structured form, restore list hierarchy, and mark paragraph boundaries. It performs plain text base extraction, optical character recognition error correction, global text normalization, garbled and invalid symbol filtering, sentence-level segmentation and sentence completion, intelligent segmentation of continuous text, contextual semantic coherence verification, and paragraph boundary and meta-information marking for unstructured documents.

4. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of performing feature fusion to obtain document data with fused multi-level features is as follows: For structured documents and unstructured documents, vector data is extracted from the basic semantic layer features, logical association layer features, and knowledge system layer features, respectively. Through linear transformation, the vector data is uniformly mapped to the same vector dimension to complete the dimension alignment process. For dimension-aligned vector data, min-max normalization is performed to map all feature values ​​to the range of 0 to 1. Initialize weights for the vector data of basic semantic layer features, logical association layer features, and knowledge system layer features respectively; Based on the assigned weights, the vector data of basic semantic layer features, logical association layer features and knowledge system layer features are respectively weighted and scaled to obtain weighted vector data; The weighted vector data is multiplied pairwise to generate cross-interaction features; The weighted vector data and cross-interaction features are concatenated end to end to generate the fused full feature data for structured documents and unstructured documents, respectively. The full set of feature data is bound one-to-one with each segment of text in the preprocessed text data. At the same time, each segment of text is bound with the corresponding original features, feature weight information, feature metadata, and pipeline identifier at each level, distinguishing between structured and unstructured documents and generating document data that integrates multi-level features.

5. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of feeding document data with fused multi-level features into corresponding processing pipelines for semantic segmentation to obtain document block results is as follows: The structured data in the document data that integrates multi-level features is sent into the structured document processing pipeline to perform title structure recognition and hierarchical node extraction, title jurisdiction delineation, knowledge boundary matching and verification based on knowledge system layer features, block rule execution and boundary correction, and document block division and verification to obtain structured document blocks; Unstructured data from document data that incorporates multi-level features is fed into an unstructured document processing pipeline. The pipeline performs the following steps: determining candidate semantic nodes based on the distribution of basic semantic layer features; filtering candidate block boundaries based on logical association similarity; verifying and correcting block boundaries based on knowledge system keywords; and dividing and verifying document blocks to obtain unstructured document blocks. The system aggregates structured and unstructured document blocks, assigns a globally unique block identifier to each document block, and binds corresponding document source information, block boundary location information, hierarchical feature information, knowledge category information, token quantity information, and pipeline identifier to each document block, generating and outputting the document block segmentation results.

6. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of sequentially performing terminology standardization, entity linking, and adding contextual enhancement information on the document segmentation results to obtain standardized knowledge blocks is as follows: Based on the standard dictionary of tobacco professional terminology, the positive maximum matching algorithm is used to standardize the terminology of the document block results and generate terminology-standardized document block content; Based on the standardized document block content, entity linking is performed by combining basic semantic layer features, knowledge system layer features, and basic feature library to generate document block content with linked entities. Based on the document block content linked by entities, combined with document source information, basic semantic layer features, logical association layer features, knowledge system layer features, and document block segmentation results, context enhancement information is added to generate and output standardized knowledge blocks.

7. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of vectorizing standardized knowledge blocks to generate corresponding vector data, and then writing the vector data, combined with hierarchical feature metadata, into a vector database is as follows: The text to be encoded is generated by using a single standardized knowledge block as the smallest processing unit; The text to be encoded is vectorized into dense vector data; the dense vector data is then combined with hierarchical feature metadata and written into the vector database.

8. The method for processing and parsing tobacco professional knowledge documents based on multi-level feature aggregation according to claim 1, characterized in that, The process of performing quality assessment and optimizing multi-level feature extraction parameters and semantic segmentation block parameters based on the assessment results is as follows: A quality assessment is performed on all data written into the vector database to obtain scores for knowledge block completeness, terminology standardization accuracy, entity link accuracy, block boundary rationality, and vector representation effectiveness. The arithmetic mean is then calculated to obtain the quality assessment result. Based on the quality assessment results, optimize the multi-level feature extraction parameters and semantic segmentation block parameters.

Images