Adaptive text chunking and summarization vectorization method for enterprise long documents

Abstract

This invention discloses an adaptive text segmentation and summary-based vectorization method and system for long enterprise documents. The method includes: acquiring long enterprise documents, parsing their format type, extracting structural and industry features, filtering redundant information, and forming standardized data. Based on the extracted features, a segmentation decision model is constructed. The granularity is dynamically adjusted through semantic correlation analysis, merging consecutive paragraphs with the same technical topic, and splitting text blocks when the topic changes, ensuring that each text block retains core information. The core technical points of the text blocks are extracted using a large local enterprise model, generating structured summaries and associating them with source and permission identifiers. Semantic encoding is used to convert the structured summaries into high-dimensional vectors. The representation is optimized by combining an industry-specific word vector dictionary. After establishing a mapping relationship between vectors and text blocks and the original document, the vectors are stored in the enterprise knowledge base vector database. This improves the utilization efficiency of the enterprise's core knowledge.

Description

Technical Field

[0001] This invention belongs to the field of enterprise knowledge management technology, and more specifically, relates to an adaptive text segmentation and summary-based vectorization method and system for long enterprise documents. Background Technology

[0002] In the daily operations of an enterprise, long documents serve as important carriers of core knowledge such as technological achievements, project plans, and business data, including technical R&D reports, project feasibility analysis documents, and industry research summaries. These documents are typically characterized by their length, multiple information dimensions, and complex nesting of technical topics. They contain both specific implementation details and key technical conclusions and industry experience, making them a core resource for enterprise knowledge accumulation and reuse.

[0003] Currently, enterprises often rely on manual chapter division or simple keyword segmentation to break down long documents, making it difficult to accurately identify the technical themes within the document. Manual segmentation is inefficient and heavily influenced by subjective judgment, easily leading to uneven segmentation granularity and insufficient thematic focus. Keyword segmentation, on the other hand, only matches surface-level words and cannot capture the deep semantic connections and structural logic of the text, resulting in thematically mixed or fragmented text blocks.

[0004] The deficiencies in the segmentation process directly affect the effectiveness of subsequent knowledge extraction and utilization. Existing summary generation methods mostly involve full-text summarization or paragraph extraction, making it difficult to generate structured summaries focusing on specific topics from the segmented text blocks, resulting in inaccurate extraction of core technical information. During the vectorization process, due to the lack of deep integration with industry attributes, the generated vectors fail to accurately represent the industry characteristics of the technical topics, leading to low accuracy in knowledge retrieval and reuse in vector databases.

[0005] The aforementioned problems prevent the efficient extraction and utilization of a large amount of core knowledge within enterprise long documents. This not only increases the cost of knowledge management but also hinders the efficiency of knowledge sharing within the enterprise, impacting the advancement of business activities such as technology research and development and project decision-making. Therefore, achieving accurate segmentation, efficient summary extraction, and high-dimensional vector transformation of enterprise long documents has become a key requirement for improving enterprise knowledge management and promoting knowledge reuse, and has significant practical implications. Summary of the Invention

[0006] This invention aims to address the practical problems of inaccurate text segmentation, inefficient summary extraction, and insufficient vectorized representation in long enterprise documents. By employing adaptive text segmentation, structured summary generation, and semantic coding technology that integrates industry attributes, it achieves accurate segmentation of long documents according to technical topics, structured extraction of core knowledge, and efficient vector-level transformation. This provides reliable technical support for enterprise knowledge management, intelligent retrieval, and reuse, and improves the utilization efficiency of enterprise core knowledge.

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, as a first aspect of this invention, the present invention provides an adaptive text segmentation and summary-based vectorization method for long enterprise documents, comprising: S1. Obtain the enterprise's long document, parse the format type of the long document and extract the document's structural features and industry features, while filtering out redundant information in the document to form standardized document data; S2. Based on the extracted structural and industry features, a block-segmentation decision model is constructed, and the block granularity is dynamically adjusted through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text blocks are automatically split, and each text block retains the core information of the corresponding technical topic. S3. For each text block obtained, call the large model deployed locally by the enterprise to extract the core technical points within each text block and generate a structured summary; at the same time, associate the source information and permission identifier of the original document with the structured summary; S4. The generated structured summary is converted into a high-dimensional vector using a semantic encoding method, and the vector representation is optimized by combining it with a professional word vector dictionary for the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorized result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

[0008] Furthermore, the enterprise long document in S1 includes professional documents corresponding to the enterprise's field; the professional documents include technical standard documents, project result documents, and design-related documents; the format types include document formats commonly used in the enterprise's daily office work and professional production processes; the structural features include the document's inherent hierarchical division, paragraph division, and format identification information; and the industry features include professional terminology, topic identifiers, and compliance clause identifiers corresponding to the application field.

[0009] Furthermore, the standardized document data in S1 is stored in a structured data format, including hierarchical structural feature data, key-value pair industry feature data, and paragraph-level deredundant text content. The hierarchical structural feature data is set up according to a three-level structure: overall document hierarchy, chapter / paragraph division, and format identifier content type. The features at each level establish a mapping relationship through associated fields. In the industry feature data in the form of key-value pairs, the key is the feature type identifier corresponding to professional terms, topic identifiers, and compliance clause identifiers, and the value is a set of feature content with frequency of occurrence and location index. In the paragraph-level deduplication of text content, each paragraph is accompanied by a semantic topic tag and a positional mapping relationship with the original document, thus completing the structured retention and traceability of core information.

[0010] Furthermore, the block-based decision model in S2 is specifically as follows: Based on the extracted structural and industry features, a decision-making model with multi-dimensional quantification as its core is constructed to determine the boundaries of text segments; among them, the correlation degree of industry features is determined by the formula. Calculate, where , These are sets of industry-specific content from adjacent paragraphs. For set and The number of elements in the intersection. For set and The number of elements in the union is used to quantify the degree of overlap of core industry information in adjacent paragraphs; Technical topic coherence is expressed by formula Calculate, where Vectors of adjacent paragraphs after semantic encoding and The distance is used to characterize the semantic relevance of the technical topics in adjacent paragraphs; Structural continuity is expressed by the formula Judgment, among which This indicates that adjacent paragraphs have the same hierarchical structure. This indicates that the formatting tags of adjacent paragraphs are consecutive, and is used to capture the thematic continuity at the document structure level. The logical NOT operator is represented. This represents the logical AND operator. Represents the logical "OR" operator; when , and When consecutive paragraphs revolving around the same technical topic are determined, they are merged into a single text block. , For the preset threshold; when , If any of the conditions are met or When a chapter jump or technical topic switch occurs, the text block is automatically split.

[0011] Furthermore, the specific process of dynamically adjusting the block granularity through semantic correlation analysis in S2 is as follows: With the information density and semantic integrity of the technical topics within the text block as the core adjustment target, a granular adjustment mechanism is established. By quantifying the topic fit and information complementarity between the current text block and the paragraph to be merged, the granularity of the block is adaptively adjusted. The theme relevance is expressed by the formula. Calculate, where This is the set of industry features for the current text block. The set of industry characteristics for the paragraphs to be merged. Let be the number of elements in the intersection of the two sets. The number of elements in the union of the two platforms quantifies the degree of overlap in their technical themes; Information complementarity is expressed by the formula Calculate, where The number of industry-specific features unique to the paragraphs to be merged. The total number of elements in the industry feature set of the paragraphs to be merged quantifies the degree to which the paragraphs to be merged supplement the current text block with new information; When the calculation results of topic relevance and information complementarity satisfy When it is determined that the paragraph to be merged matches the theme of the current text block and can supplement effective information, the granularity of the block is expanded and it is incorporated into the current text block; when If the paragraph to be merged is determined to have a weak connection with the topic of the current text block or to have duplicate information, the current block granularity is maintained and the construction of a new text block is started.

[0012] Furthermore, the structured summary in S3 is a standardized information carrier formed by integrating key information within a text block, with the core theme of the text block as its core; its structure matches the adaptation requirements of the subsequent semantic vectorization step and is directly related to the block segmentation results of the text block. The structured summary includes at least three core fields: topic identifier, core information set, and feature association description. The topic identifier is determined based on the feature information of the text block and the semantic association analysis results, and is consistent with the core topic determined during the block segmentation process. The core information set extracts key content including the core principles, implementation logic, and key conclusions within the text block, corresponding to the core topic information retained in the text block after segmentation. The feature association description records the correspondence between the core information and the relevant features of the text block, providing a basis for vector dimension mapping during subsequent semantic encoding. The structured summary adopts a standardized organizational form, and its associated original information source identifier and access permission information are embedded as independent fields. This ensures the traceability of the summary information and the adaptability of access control, while also improving the efficiency and accuracy of subsequent semantic encoding through the standardized structure.

[0013] Furthermore, the semantic encoding method in S4 is specifically as follows: First, symbolic mapping is performed on each field of the structured summary, including the subject identifier, core information set, and feature association description, to obtain a field symbol sequence. Then through the formula Calculate the fundamental vector for each field, where This is the original semantic vector of a single symbol in the field symbol sequence. The number of symbols in the corresponding field is used to complete the normalized representation of the field-level semantics; Then through the formula Calculate the initial fusion vector of the structured summary, where These are the base vectors for the topic identifier, core information set, and feature association description fields, respectively. The magnitude of the summation of the three vectors is used to complete the unbiased fusion of multi-field semantics; Finally, combining industry-specific word vector dictionaries, through formulas... Optimize vector representation, where These are industry-specific word vectors from the dictionary that match the core information of the structured summary. To determine the number of matched professional terms, the high-dimensional vector is integrated with industry attributes.

[0014] Furthermore, the vectorized results in S4 provide data support for applications including RAG-enhanced retrieval and intelligent question answering, and the vectorized results support real-time invocation of enterprise multi-terminal application plugins.

[0015] As a second aspect of the present invention, an adaptive text segmentation and summary-based vectorization system for enterprise long documents is also provided, comprising: The document preprocessing unit is used to acquire long enterprise documents, parse the format type of the long documents and extract the structural and industry features of the documents, while filtering out redundant information in the documents to form standardized document data. The adaptive text segmentation unit is used to build a segmentation decision model based on extracted structural and industry features, and dynamically adjust the segmentation granularity through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text block is automatically split, and each text block retains the core information of the corresponding technical topic. The segmented structured summary generation unit is used to extract the core technical points within each text block from the large model deployed locally by the enterprise, and generate a structured summary; at the same time, it associates the source information and permission identifier of the original document with the structured summary. The summary-based vectorization and storage unit is used to convert the generated structured summary into a high-dimensional vector using semantic encoding methods, and to optimize the vector representation by combining it with a professional word vector dictionary of the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorization result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

[0016] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor as described in any one of the claims, an adaptive text segmentation and summary vectorization method for enterprise long documents.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The adaptive text segmentation and summary-based vectorization method for long enterprise documents of this invention dynamically adjusts the granularity of text segmentation by constructing a multi-dimensional quantitative segmentation decision model based on industry characteristics, semantic relationships, and structural features. This model calculates the overlap of industry feature sets between adjacent paragraphs, performs spatial distance analysis on semantic vectors, and determines the continuity of hierarchical structure and format identifiers, thereby accurately identifying the boundaries of technical topics and achieving adaptive text block division. This technical feature enables long enterprise documents to be broken down into topic-focused and information-complete text blocks according to technical topics, avoiding both topic mixing caused by overly coarse segmentation and information fragmentation caused by overly fine segmentation, laying a structured foundation for subsequent knowledge extraction.

[0018] 2. The adaptive text segmentation and summary-based vectorization method for long enterprise documents of this invention generates a structured summary directly related to the segmentation results by extracting key information such as technical topic identifiers, core technical element sets, and feature association descriptions from text blocks. This summary is organized in a field-based format, with each field corresponding to the core content of the text block and associated with the original document source and access control, ensuring information traceability and secure management. This technical feature allows the core knowledge of long enterprise documents to be presented in a standardized structure, adapting to the subsequent semantic encoding vector generation logic and meeting the storage and retrieval needs of the enterprise knowledge base, thus achieving efficient conversion of knowledge from documents to structured summaries.

[0019] 3. The adaptive text segmentation and summary-based vectorization method for enterprise long documents of this invention transforms structured summaries into high-dimensional vectors that integrate semantic information and industry attributes through a multi-stage semantic encoding method using pure mathematical operations. This method first sums and normalizes the symbol sequences of each field of the summary using semantic vectors, then performs unbiased fusion of multi-field vectors, and finally optimizes the vector representation by combining an industry-specific word vector dictionary. The generated high-dimensional vectors are then mapped to text blocks and the original document and stored in a vector database. This technical feature enables efficient storage and retrieval of knowledge from enterprise long documents in vector form, preserving semantic richness while incorporating industry-specific professional attributes, providing precise vector-level data support for enterprise applications such as knowledge reuse and intelligent retrieval based on long documents. Attached Figure Description

[0020] Figure 1 This is a flowchart of an adaptive text segmentation and summary-based vectorization method for long enterprise documents according to an embodiment of the present invention; Figure 2 This is a knowledge management ecosystem architecture diagram according to an embodiment of the present invention; Figure 3 This is a system unit diagram of an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 Please refer to Figure 1 This embodiment 1 provides an adaptive text segmentation and summary-based vectorization method for long enterprise documents, including: S1. Obtain the enterprise's long document, parse the format type of the long document and extract the document's structural features and industry features, while filtering out redundant information in the document to form standardized document data; S2. Based on the extracted structural and industry features, a block-segmentation decision model is constructed, and the block granularity is dynamically adjusted through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text blocks are automatically split, and each text block retains the core information of the corresponding technical topic. S3. For each text block obtained, call the large model deployed locally by the enterprise to extract the core technical points within each text block and generate a structured summary; at the same time, associate the source information and permission identifier of the original document with the structured summary; S4. The generated structured summary is converted into a high-dimensional vector using a semantic encoding method, and the vector representation is optimized by combining it with a professional word vector dictionary for the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorized result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

[0023] Please refer to Figure 2 The method in Example 1 constructs a complete enterprise knowledge management ecosystem covering knowledge acquisition, topic accumulation, and scenario application. Through precise segmentation, structured summary generation, and semantic vectorization processing of long enterprise documents, including technical standards and research projects, various long documents are transformed into efficiently searchable and reusable knowledge assets. These assets are then implemented in multiple scenarios such as updates to safety supervision regulations, intelligent Q&A, and reuse of historical data. Furthermore, through multi-terminal plugins such as CAD and WPS, and RAG-enhanced retrieval and generative Q&A capabilities, collaborative sharing and intelligent application of enterprise knowledge across project and personal spaces are achieved, providing a full-chain solution for enterprise digital knowledge management and intelligent business support.

[0024] Specifically, this embodiment 1 further elaborates on the above steps.

[0025] (1) Document preprocessing In enterprise knowledge management systems, long documents serve as crucial carriers of core technological achievements, business process specifications, and project practical experience. However, their heterogeneous formats, redundant content, and complex information hierarchies severely hinder the efficient extraction and reuse of knowledge. To overcome this challenge, it is necessary to first collect long documents from the enterprise. These collected long documents should focus on the professional document categories within the enterprise's field, specifically covering high-value-density core materials such as technical standards documents, project outcome documents, and design-related documents. These documents are directly related to key business scenarios such as enterprise technology research and development, project progress, and compliance management.

[0026] Subsequently, the document format parsing process is initiated. The parsed formats cover a wide range of document formats widely used in daily office scenarios and professional production processes, ensuring comprehensive compatibility and adaptation for long documents from different sources and in different formats. Simultaneously, the system performs document feature extraction: on the one hand, it extracts the document's structural features, including the document's inherent hierarchical structure, paragraph break boundaries, and format identification information. The hierarchical structure is reflected in the overall chapter hierarchy, paragraph break boundaries clearly define the natural dividing points of text content, and format identification information covers visual distinguishing elements such as heading styles, list symbols, and chart annotations, providing a structural basis for subsequent text segmentation. On the other hand, it extracts the document's industry characteristics, encompassing the professional terminology system, topic identification information, and compliance clause identification in the corresponding application field. The professional terminology system reflects the technical specifications and expression habits within the field, topic identification information clarifies the core technical direction or business theme of the document, and compliance clause identification corresponds to industry regulatory requirements and relevant internal company regulations, highlighting the document's professional and compliant attributes.

[0027] After feature extraction, a systematic filtering process is implemented to remove redundant information in the documents, focusing on eliminating interfering elements that do not have substantial information value, such as duplicate statements, invalid placeholder content, and redundant format data, and finally constructing a standardized document data system. This standardized document data is stored in a structured data format and consists of three core parts: First, hierarchical structural feature data, set up according to a three-level structure of overall document hierarchy, chapter / paragraph division, and format identifier content type. Each level of feature establishes a precise mapping relationship through preset association fields, fully restoring the document's organizational structure and logical context. Second, industry feature data in key-value pair format, where the key field is defined as the feature type identifier corresponding to professional terms, topic identifiers, and compliance clause identifiers, and the value field is a set of feature content with accompanying frequency statistics and original text location index, supporting rapid retrieval and quantitative analysis of industry features. Third, redundant text content at the paragraph level, with each paragraph accompanied by topic tags generated by semantic analysis and establishing a precise location mapping relationship with the original document. This achieves both the structured retention of the document's core information and ensures the traceability of the original information in any subsequent stage, providing standardized and high-quality data support for subsequent processing stages such as text segmentation, structured summary generation, and semantic vectorization.

[0028] (2) Adaptive text chunking Following the standardization process described above, structured document features, industry characteristics, and redundant text content have been obtained, providing high-quality data support for accurate segmentation. Based on these extracted features, a segmentation decision model centered on multi-dimensional quantitative analysis is constructed. Through comprehensive evaluation, the boundaries of text segmentation are scientifically determined. At the same time, semantic relevance analysis is combined to dynamically adjust the segmentation granularity, ensuring that the segmentation results not only conform to the document logic but also fully preserve the core information of the technical topic.

[0029] The decision-making logic of the block-based model encompasses three key dimensions: First, industry feature relevance assessment, which quantifies the degree of consistency between adjacent paragraphs in core information such as professional terminology, topic identifiers, and compliance clauses by calculating the overlap ratio of industry feature sets. Second, technical topic coherence analysis, which infers the topical relevance of adjacent paragraphs through vector space distance after semantic encoding; the closer the distance, the stronger the topical consistency. Third, structural feature continuity determination, which determines that adjacent paragraphs have topical continuity at the structural level when their hierarchical structure and format identifiers are consistent; otherwise, the structure is considered broken. Only when the industry feature relevance and technical topic coherence both reach preset thresholds, and the structural features remain continuous, are consecutive paragraphs determined to revolve around the same technical topic and merged into a single text block. If any dimension fails to meet the requirements, it is determined that a chapter jump or technical topic switch has occurred, and the text block is automatically split to ensure the topical focus of each text block.

[0030] Specifically, the mathematical description of the block decision model is as follows: Based on the extracted structural and industry features, a decision-making model with multi-dimensional quantification as its core is constructed to determine the boundaries of text segments; among them, the correlation degree of industry features is determined by the formula. Calculate, where , These are sets of industry-specific content from adjacent paragraphs. For set and The number of elements in the intersection. For set and The number of elements in the union is used to quantify the degree of overlap of core industry information in adjacent paragraphs; Technical topic coherence is expressed by formula Calculate, where Vectors of adjacent paragraphs after semantic encoding and The distance is used to characterize the semantic relevance of the technical topics in adjacent paragraphs; Structural continuity is expressed by the formula Judgment, among which This indicates that adjacent paragraphs have the same hierarchical structure. This indicates that the formatting tags of adjacent paragraphs are consecutive, and is used to capture the thematic continuity at the document structure level. The logical NOT operator is represented. This represents the logical AND operator. Represents the logical "OR" operator; when , and When consecutive paragraphs revolving around the same technical topic are determined, they are merged into a single text block. , For the preset threshold; when , If any of the conditions are met or When a chapter jump or technical topic switch occurs, the text block is automatically split.

[0031] Regarding the adjustment of block granularity, an adaptive adjustment mechanism is established with the information density and semantic integrity of the technical topics within the text block as the core objectives. The suitability of the current text block and the paragraph to be merged is quantified by two indicators: topic fit and information complementarity. Topic fit is determined by calculating the overlap ratio of the industry feature sets of the two to clarify the degree of matching of technical topics; information complementarity is assessed by statistically analyzing the proportion of unique industry features of the paragraph to be merged to evaluate its value in supplementing the current text block with new information.

[0032] Specifically, the process of dynamically adjusting the granularity of segmentation through semantic relevance analysis is as follows: With the information density and semantic integrity of the technical topics within the text block as the core adjustment target, a granular adjustment mechanism is established. By quantifying the topic fit and information complementarity between the current text block and the paragraph to be merged, the granularity of the block is adaptively adjusted. The theme relevance is expressed by the formula. Calculate, where This is the set of industry features for the current text block. The set of industry characteristics for the paragraphs to be merged. Let be the number of elements in the intersection of the two sets. The number of elements in the union of the two platforms quantifies the degree of overlap in their technical themes; Information complementarity is expressed by the formula Calculate, where The number of industry-specific features unique to the paragraphs to be merged. The total number of elements in the industry feature set of the paragraphs to be merged quantifies the degree to which the paragraphs to be merged supplement the current text block with new information; When the calculation results of topic relevance and information complementarity satisfy When it is determined that the paragraph to be merged matches the theme of the current text block and can supplement effective information, the granularity of the block is expanded and it is incorporated into the current text block; when If the paragraph to be merged is determined to have a weak connection with the topic of the current text block or to have duplicate information, the current block granularity is maintained and the construction of a new text block is started.

[0033] When the combined evaluation results show that the paragraph to be merged is highly consistent with the theme of the current text block and can supplement effective information, the granularity of the segmentation is expanded and it is incorporated into the current text block to improve the information richness of the text block; if the evaluation results show that the themes of the two are weakly related or that there is a lot of information duplication, the current granularity of the segmentation is maintained and the construction process of a new text block is started to avoid the theme mixing due to the segmentation being too coarse, and to ensure that each text block not only fully retains the core information of the corresponding technical theme, but also has a reasonable granularity.

[0034] (3) Generation of block-based structured summaries After text segmentation, each text block becomes a focused and complete independent unit, accurately carrying the core content of the corresponding technical topic, laying the foundation for subsequent core information extraction and structured presentation. For each text block, a large-scale model deployed locally by the enterprise is invoked, leveraging its deep understanding of professional domain knowledge to accurately extract core technical points from the text block. The extraction process focuses on key content such as core principles, implementation logic, and key conclusions within the text block, ensuring that no core information of the technical topic is omitted, while eliminating irrelevant and redundant expressions to guarantee the accuracy and value density of the extracted information.

[0035] Based on the extracted core technical points, a standardized structured summary is generated. This structured summary, centered on the core theme of the text block, is a standardized information carrier integrating key information. Its structure matches the adaptation requirements of subsequent semantic vectorization and is directly related to the text block segmentation results. The structured summary contains three core fields: a topic identifier determined based on the text block's feature information and semantic association analysis results, consistent with the core theme identified during segmentation, ensuring the accuracy of the summary topic; a core information set system that includes the extracted core principles, implementation logic, and key conclusions, completely corresponding to the core theme information retained in the segmented text blocks; and a feature association description that details the correspondence between core information and relevant features of the text block, providing a clear basis for vector dimension mapping during subsequent semantic encoding, ensuring a smooth and accurate encoding process.

[0036] To meet the traceability and security requirements of enterprise knowledge management, the structured summary is generated and associated with the source information and access identifier of the original document, embedded as an independent field within the summary. The source information ensures that any structured summary can be traced back to the specific location of the original document, facilitating subsequent information verification and tracing. The access identifier clearly defines the access permission level of the summary, adapting to the enterprise's internal access management system and preventing the leakage of core knowledge. The standardized organizational form adopted by the structured summary improves the efficiency and accuracy of subsequent semantic encoding and supports the standardized management of the enterprise knowledge base, achieving efficient transformation of core information from text blocks to standardized summaries.

[0037] (4) Summary-based vectorization and storage After the structured summary generation process, a standardized, high-value-density core information carrier has been obtained, and its field-based organization provides a clear data foundation for semantic vectorization. To achieve efficient storage and intelligent application of core knowledge, a semantic encoding method is used to transform the structured summary into a high-dimensional vector, while incorporating industry attributes to ensure that the vector can accurately represent semantic information and reflect the characteristics of the professional field.

[0038] The semantic encoding process proceeds step by step according to the field level, and the main body is as follows: First, symbolic mapping is performed on each field of the structured summary, including the subject identifier, core information set, and feature association description, to obtain a field symbol sequence. Then through the formula Calculate the fundamental vector for each field, where This is the original semantic vector of a single symbol in the field symbol sequence. The number of symbols in the corresponding field is used to complete the normalized representation of the field-level semantics; Then through the formula Calculate the initial fusion vector of the structured summary, where These are the base vectors for the topic identifier, core information set, and feature association description fields, respectively. The magnitude of the summation of the three vectors is used to complete the unbiased fusion of multi-field semantics; Finally, combining industry-specific word vector dictionaries, through formulas... Optimize vector representation, where These are industry-specific word vectors from the dictionary that match the core information of the structured summary. To determine the number of matched professional terms, the high-dimensional vector is integrated with industry attributes.

[0039] In other words, it first performs symbolic mapping on three core fields of the structured summary: topic identifier, core information set, and feature association description, transforming the textual information of each field into an ordered sequence of field symbols. For each field symbol sequence, the original semantic vectors of individual symbols in the sequence are summed, and then normalized based on the number of field symbols to obtain the basic vectors of each field, achieving accurate representation of field-level semantics. Subsequently, the basic vectors of the three types of fields are summed, and then standardized using the magnitude of the summed vectors, completing the unbiased fusion of multi-field semantics to generate the initial fusion vector of the structured summary, comprehensively integrating the semantic information in the summary.

[0040] To enhance the industry attributes of the vectors, the initial fusion vectors are further optimized by incorporating a dictionary of industry-specific term vectors. Industry-specific terms related to the core information of the structured summary are matched from the dictionary, and the semantic vectors corresponding to these terms are extracted and averaged. This average value is then fused with the initial fusion vector, resulting in a high-dimensional vector that simultaneously carries semantic information and industry attributes, thus improving the accuracy of the vector representation in professional scenarios.

[0041] After vectorization, establish a precise mapping relationship between high-dimensional vectors and corresponding text blocks and original documents, clarify the association path between vectors and original information, and ensure the traceability of information in subsequent applications.

[0042] Finally, the optimized high-dimensional vector results are stored in a dedicated vector database according to the standardized data protocol of the enterprise knowledge base. During the storage process, a multi-level association index between vectors and corresponding text blocks and original documents is constructed simultaneously. The index includes key metadata such as unique vector identifiers, text block numbers, original document storage paths, and structured summary field mapping relationships to ensure full-link traceability of vectors and original information.

[0043] This vector data possesses high-dimensional semantic representation capabilities and industry attribute recognition, and can be directly used as the core data foundation for RAG enhanced retrieval. Through vector space similarity calculation, it can accurately recall the core knowledge of enterprise long documents and supplement contextual associations. At the same time, it provides structured semantic support for intelligent question and answer applications, helping the system to quickly match user query intent with corresponding knowledge vectors and generate accurate and professionally relevant answers.

[0044] Vector Database supports high-concurrency read and write operations and low-latency queries. It adopts a distributed storage architecture to ensure data security and availability, and is compatible with the enterprise's existing technical architecture. It can support real-time invocation of multi-terminal application plugins through standardized interfaces, including desktop office software, mobile business applications, enterprise-level intelligent platforms and other scenarios. It provides stable and efficient technical support for the rapid reuse of core knowledge in internal technology research and development, project decision-making and business collaboration, as well as intelligent interaction in external services.

[0045] This embodiment 1 has broad application prospects in the field of enterprise knowledge management, and can be deeply adapted to the long document processing needs of various industries such as scientific and technological research and development, engineering design, financial compliance, and healthcare. Whether it is patent documents and R&D reports of technology-intensive enterprises, or industry standards and project archives with high compliance requirements, the method can achieve structured extraction and vectorized storage of core knowledge, solving the pain points of low efficiency and difficulty in knowledge reuse in traditional long document retrieval. Its adaptive block division and industry attribute fusion vector representation capabilities can transform the massive long documents accumulated by enterprises from "static storage" to "dynamically usable" knowledge assets, providing precise knowledge support for internal training, technology transfer, and project collaboration, and helping enterprises build an efficient knowledge management system.

[0046] Under the trend of digital transformation and intelligent upgrading, the application scenario of Example 1 can be further extended to the construction of an enterprise intelligent application ecosystem. Vectorized knowledge data can be seamlessly connected to terminal applications such as RAG enhanced retrieval, intelligent customer service, and decision support systems, creating a full-link intelligent solution for enterprises from knowledge acquisition, processing, storage to application. In cross-departmental collaboration, it can realize the rapid flow and accurate matching of core knowledge; in customer service scenarios, it can support intelligent question-and-answer systems to efficiently respond to professional inquiries; in R&D and innovation scenarios, it helps R&D personnel quickly retrieve relevant technical literature and project experience. Its compatibility with multi-terminal calls and existing technical architecture can lower the threshold for enterprise intelligent transformation, inject continuous momentum into enterprise digital transformation, and has significant practical value and market promotion potential.

[0047] Example 2 Please refer to Figure 3 This embodiment 2 provides an adaptive text segmentation and summary-based vectorization system for long enterprise documents, including: The document preprocessing unit is used to acquire long enterprise documents, parse the format type of the long documents and extract the structural and industry features of the documents, while filtering out redundant information in the documents to form standardized document data. The adaptive text segmentation unit is used to build a segmentation decision model based on extracted structural and industry features, and dynamically adjust the segmentation granularity through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text block is automatically split, and each text block retains the core information of the corresponding technical topic. The segmented structured summary generation unit is used to extract the core technical points within each text block from the large model deployed locally by the enterprise, and generate a structured summary; at the same time, it associates the source information and permission identifier of the original document with the structured summary. The summary-based vectorization and storage unit is used to convert the generated structured summary into a high-dimensional vector using semantic encoding methods, and to optimize the vector representation by combining it with a professional word vector dictionary of the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorization result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

[0048] Example 3 This embodiment 3 also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement any step of an adaptive text segmentation and summary vectorization method for enterprise long documents.

[0049] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0051] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive text segmentation and summary-based vectorization method for long enterprise documents, characterized in that, include: S1. Obtain the enterprise's long document, parse the format type of the long document and extract the document's structural features and industry features, while filtering out redundant information in the document to form standardized document data; S2. Based on the extracted structural and industry features, a block-segmentation decision model is constructed, and the block granularity is dynamically adjusted through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text blocks are automatically split, and each text block retains the core information of the corresponding technical topic. S3. For each text block obtained, call the large model deployed locally by the enterprise to extract the core technical points within each text block and generate a structured summary; at the same time, associate the source information and permission identifier of the original document with the structured summary; S4. The generated structured summary is converted into a high-dimensional vector using a semantic encoding method, and the vector representation is optimized by combining it with a professional word vector dictionary for the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorized result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

2. The adaptive text segmentation and summary-based vectorization method for enterprise long documents according to claim 1, characterized in that, The enterprise long document in S1 includes professional documents corresponding to the enterprise's field; the professional documents include technical standard documents, project result documents, and design-related documents; the format types include document formats commonly used in the enterprise's daily office work and professional production processes; the structural features include the document's inherent hierarchical division, paragraph division, and format identification information; the industry features include professional terms, theme identifiers, and compliance clause identifiers in the corresponding application field.

3. The adaptive text segmentation and summary-based vectorization method for long enterprise documents according to claim 1, characterized in that, The standardized document data in S1 is stored in a structured data format, including hierarchical structure feature data, key-value pair industry feature data, and paragraph-level deredundant text content. The hierarchical structural feature data is set up according to a three-level structure: overall document hierarchy, chapter / paragraph division, and format identifier content type. The features at each level establish a mapping relationship through associated fields. In the industry feature data in the form of key-value pairs, the key is the feature type identifier corresponding to professional terms, topic identifiers, and compliance clause identifiers, and the value is a set of feature content with frequency of occurrence and location index. In the paragraph-level deduplication of text content, each paragraph is accompanied by a semantic topic tag and a positional mapping relationship with the original document, thus completing the structured retention and traceability of core information.

4. The adaptive text segmentation and summary-based vectorization method for long enterprise documents according to claim 1, characterized in that, The block-based decision model in S2 is specifically as follows: Based on the extracted structural and industry features, a decision-making model with multi-dimensional quantification as its core is constructed to determine the boundaries of text segments; among them, the correlation degree of industry features is determined by the formula. Calculate, where , These are sets of industry-specific content from adjacent paragraphs. For set and The number of elements in the intersection. For set and The number of elements in the union is used to quantify the degree of overlap of core industry information in adjacent paragraphs; Technical topic coherence is expressed by formula Calculate, where Vectors of adjacent paragraphs after semantic encoding and The distance is used to characterize the semantic relevance of the technical topics in adjacent paragraphs; Structural continuity is expressed by the formula Judgment, among which This indicates that adjacent paragraphs have the same hierarchical structure. This indicates that the formatting tags of adjacent paragraphs are consecutive, and is used to capture the thematic continuity at the document structure level. The logical "NOT" operator is represented. This represents the logical "AND" operator. This represents the logical "OR" operator; when , and When consecutive paragraphs revolving around the same technical topic are determined, they are merged into a single text block. , For the preset threshold; when , If any of the conditions are met or When a chapter jump or technical topic switch occurs, the text block is automatically split.

5. The adaptive text segmentation and summary-based vectorization method for enterprise long documents according to claim 1, characterized in that, The specific process of dynamically adjusting the block granularity through semantic correlation analysis in S2 is as follows: With the information density and semantic integrity of the technical topics within the text block as the core adjustment target, a granular adjustment mechanism is established. By quantifying the topic fit and information complementarity between the current text block and the paragraph to be merged, the granularity of the block is adaptively adjusted. The theme relevance is expressed by the formula. Calculate, where This is the set of industry features for the current text block. The set of industry characteristics for the paragraphs to be merged. Let be the number of elements in the intersection of the two sets. The number of elements in the union of the two platforms quantifies the degree of overlap in their technical themes; Information complementarity is expressed by the formula Calculate, where The number of industry-specific features unique to the paragraphs to be merged. The total number of elements in the industry feature set of the paragraphs to be merged quantifies the degree to which the paragraphs to be merged supplement the current text block with new information; When the calculation results of topic relevance and information complementarity satisfy When it is determined that the paragraph to be merged matches the theme of the current text block and can supplement effective information, the granularity of the block is expanded and it is incorporated into the current text block; when If the paragraph to be merged is determined to have a weak connection with the topic of the current text block or to have duplicate information, the current block granularity is maintained and the construction of a new text block is started.

6. The adaptive text segmentation and summary-based vectorization method for enterprise long documents according to claim 1, characterized in that, The structured summary in S3 is a standardized information carrier that integrates key information within a text block, with the core theme of the text block as its core. Its structure matches the adaptation requirements of the subsequent semantic vectorization process and is directly related to the block segmentation results of the text block. The structured summary includes at least three core fields: topic identifier, core information set, and feature association description. The topic identifier is determined based on the feature information of the text block and the semantic association analysis results, and is consistent with the core topic determined during the block segmentation process. The core information set extracts key content including the core principles, implementation logic, and key conclusions within the text block, corresponding to the core topic information retained in the text block after segmentation. The feature association description records the correspondence between the core information and the relevant features of the text block, providing a basis for vector dimension mapping during subsequent semantic encoding. The structured summary adopts a standardized organizational form, and its associated original information source identifier and access permission information are embedded as independent fields. This ensures the traceability of the summary information and the adaptability of access control, while also improving the efficiency and accuracy of subsequent semantic encoding through the standardized structure.

7. The adaptive text segmentation and summary-based vectorization method for enterprise long documents according to claim 1, characterized in that, The semantic encoding method in S4 is specifically as follows: First, symbolic mapping is performed on each field of the structured summary, including the subject identifier, core information set, and feature association description, to obtain a field symbol sequence. Then through the formula Calculate the fundamental vector for each field, where This is the original semantic vector of a single symbol in the field symbol sequence. The number of symbols in the corresponding field is used to complete the normalized representation of the field-level semantics; Then through the formula Calculate the initial fusion vector of the structured summary, where These are the base vectors for the topic identifier, core information set, and feature association description fields, respectively. The magnitude of the summation of the three vectors is used to complete the unbiased fusion of multi-field semantics; Finally, combining industry-specific word vector dictionaries, through formulas... Optimize vector representation, where These are industry-specific word vectors from the dictionary that match the core information of the structured summary. To determine the number of matched professional terms, the high-dimensional vector is integrated with industry attributes.

8. The adaptive text segmentation and summary vectorization method for long enterprise documents according to claim 1, characterized in that, The vectorized results in S4 provide data support for applications including RAG-enhanced retrieval and intelligent question answering, and the vectorized results support real-time invocation of enterprise multi-terminal application plugins.

9. An adaptive text segmentation and summarization vectorization system for long enterprise documents, characterized in that, include: The document preprocessing unit is used to acquire long enterprise documents, parse the format type of the long documents and extract the structural and industry features of the documents, while filtering out redundant information in the documents to form standardized document data. The adaptive text segmentation unit is used to build a segmentation decision model based on extracted structural and industry features, and dynamically adjust the segmentation granularity through semantic correlation analysis. When continuous paragraphs are detected to revolve around the same technical topic, the continuous paragraphs are merged into a text block. When a chapter jump or technical topic switch is detected, the text block is automatically split, and each text block retains the core information of the corresponding technical topic. The segmented structured summary generation unit is used to extract the core technical points within each text block from the large model deployed locally by the enterprise, and generate a structured summary; at the same time, it associates the source information and permission identifier of the original document with the structured summary. The summary-based vectorization and storage unit is used to convert the generated structured summary into a high-dimensional vector using semantic encoding methods, and to optimize the vector representation by combining it with a professional word vector dictionary of the corresponding industry, so that the high-dimensional vector contains both semantic information and industry attributes. After establishing a mapping relationship between the vectorization result and the corresponding text block and the original document, it is stored in the vector database of the enterprise knowledge base.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8: an adaptive text segmentation and summary vectorization method for enterprise long documents.

Images