Method and storage medium for enterprise-level unstructured knowledge governance

Abstract

The application provides a method and a storage medium for enterprise-level unstructured knowledge governance. The method comprises the following steps: inputting an initial document into a visual language model for layout analysis to obtain document structure information of a target document; performing segmentation processing on the document structure information to generate a text slice to be vectorized; extracting metadata based on the text slice to generate a structured knowledge content vector; and configuring an index according to the knowledge content vector to form an enterprise-level knowledge governance standard. Through the visual language model for analyzing the document structure, the intelligent segmentation algorithm for generating the text slice, the metadata extraction for constructing the knowledge vector, and the hybrid index configuration, the problems of inaccurate layout analysis, logical structure damage, and low retrieval efficiency in the traditional technology are effectively solved, and the method has the advantages of improving the knowledge governance precision and usability.

Description

Technical Field

[0001] This application relates to the field of enterprise knowledge management technology, specifically to a method and storage medium for enterprise-level unstructured knowledge governance. Background Technology

[0002] As artificial intelligence and enterprise knowledge management systems become increasingly integrated, more and more organizations are building internal knowledge bases to support business scenarios such as intelligent question answering, knowledge retrieval, and process automation. However, current mainstream enterprise knowledge base systems still face significant technical bottlenecks in the automated governance of unstructured documents. Enterprise knowledge sources are complex and diverse in format, including a large number of portable document formats (PDF), Microsoft Word, scanned documents, images, tables, and even web page content. This data often has complex layout structures, such as two-column layouts, mixed text and images, nested tables, and multi-level directories. Traditional optical character recognition (OCR) or text parsing technologies struggle to recover their true structural semantics, resulting in insufficient parsing accuracy. In double-column papers and reports commonly found in industries like medicine and scientific research, OCR typically parses documents line by line, failing to accurately reconstruct the reading order. For documents containing images, flowcharts, or charts, such as instruction manuals and reports, the system cannot recognize semantic relationships or logical structures within the images. Traditional parsing and segmentation of table content cannot guarantee the table's structural information. The lack of structured hierarchy after parsing titles and subheadings leads to inaccurate subsequent knowledge slicing and indexing. These problems frequently result in phenomena such as "overly fragmented segments," "missing paragraphs," and "content misalignment" during knowledge base construction, directly impacting the accuracy and usability of knowledge-based question answering. Existing knowledge base governance systems generally suffer from complex operational processes, insufficient visualization capabilities, and a lack of clear process guidance and intelligent decision-making mechanisms. Users must repeatedly experiment with different parameters to obtain satisfactory results, increasing learning costs and significantly reducing governance efficiency. Furthermore, systems often lack batch governance templates or reusable pipeline processes for similar documents, requiring repeated processing of the same type of files and hindering scalable governance. Current automated segmentation algorithms slice text based solely on statistical features or character count, ignoring the document's visual layout and semantic context. This disrupts the logical boundaries of the document content, resulting in a lack of complete semantic support for the question-answering recall, leading to illusions or ambiguities in the generated results. Summary of the Invention

[0003] The embodiments of this application provide a method and storage medium for enterprise-level unstructured knowledge governance, which has the advantages of improving the parsing accuracy of unstructured documents, maintaining the logical integrity of content, and improving the efficiency of knowledge governance.

[0004] In a first aspect, embodiments of this application provide a method for enterprise-level unstructured knowledge governance, the method comprising:

[0005] The initial document is input into the visual language model for layout parsing to obtain the document structure information of the target document;

[0006] The document structure information is segmented to generate text slices to be vectorized;

[0007] Metadata is extracted based on the text slices to generate a structured knowledge content vector;

[0008] Based on the knowledge content vector, an index configuration is performed to form an enterprise-level knowledge governance standard.

[0009] In some embodiments, the step of inputting the initial document into a visual language model for layout parsing to obtain the document structure information of the target document includes:

[0010] The initial document is input into the visual language model for initialization processing to obtain the target document;

[0011] Extract the initial structured layout information of the target document;

[0012] The initial structured layout information is formatted to obtain the target structured layout information;

[0013] The target structured layout information is processed to obtain the document structure information.

[0014] In some embodiments, segmenting the document structure information to generate text slices to be vectorized includes:

[0015] Based on the document structure information, configure segmentation constraint parameters and construct an initial segmentation algorithm;

[0016] The content of the target document is preprocessed to generate iterable content stream information;

[0017] A preset greedy strategy is used to fill the content stream information into the initial segmentation algorithm to determine the segmentation boundary information of the document structure information;

[0018] The initial segmentation algorithm is optimized based on the segmentation boundary information to obtain the target segmentation algorithm;

[0019] The text slices are generated based on the target segmentation algorithm.

[0020] In some embodiments, the step of using a preset greedy strategy to fill the content stream information into the initial segmentation algorithm and determining the segmentation boundary information of the document structure information includes:

[0021] The greedy strategy is used to fill the content stream information into the initial segmentation algorithm to obtain the title level information, content type information and semantic boundary information of the target document;

[0022] The segmentation boundary information is determined based on the title level information, the content type information, and the semantic boundary information.

[0023] In some embodiments, optimizing the initial segmentation algorithm based on the segmentation boundary information to obtain the target segmentation algorithm includes:

[0024] Based on the segmentation boundary information, the slices with fewer than a preset threshold in the initial segmentation algorithm are merged with adjacent slices to obtain the target segmentation algorithm.

[0025] In some embodiments, the step of extracting metadata based on the text slices to generate a structured knowledge content vector includes:

[0026] The text slices are semantically compressed and feature-annotated to generate meta-hint information for the text slices;

[0027] The knowledge content vector is generated based on the meta-hint information.

[0028] In some embodiments, the meta-hint information includes summary information, tag information, and contextual reference information; the step of semantically compressing and characterizing the text slice to generate the meta-hint information for the text slice includes:

[0029] Semantic compression is performed on the text slices to extract and retain the summary information that preserves the core semantics of the text slices;

[0030] The text slice is encapsulated using a context window to generate the context reference information; the context window includes key information about the preceding and subsequent slices of the text slice.

[0031] The feature label generation algorithm is applied to the text slice to extract the topic words and keywords from the text slice;

[0032] The tag information is determined based on the topic words and keywords.

[0033] In some embodiments, the step of configuring the index based on the knowledge content vector to form an enterprise-level knowledge governance standard includes:

[0034] The knowledge content vector is transformed to obtain a multi-dimensional vector;

[0035] The multi-dimensional vector is indexed to generate hybrid retrieval information that supports both structured query language and vector query language; the index construction includes at least one of quality pattern index construction and economic pattern index construction.

[0036] Based on the hybrid retrieval information, a retrieval strategy is configured to generate the knowledge governance standard.

[0037] In some embodiments, the index configuration includes drag-and-drop metadata configuration; the method further includes:

[0038] Construct the database for the drag-and-drop metadata configuration; the database includes text components, table components, image components, and formula components;

[0039] At least one of the text component, table component, image component, and formula component is combined to construct a reusable structured information extraction template for the knowledge content vector;

[0040] The knowledge governance standard is determined based on the structured information extraction template.

[0041] Secondly, embodiments of this application provide an apparatus for enterprise-level unstructured knowledge governance, the apparatus comprising:

[0042] The parsing module is used to input the initial document into the visual language model for layout parsing to obtain the document structure information of the target document;

[0043] The processing module is used to segment the document structure information to generate text slices to be vectorized;

[0044] The extraction module is used to extract metadata based on the text slices and generate a structured knowledge content vector.

[0045] The configuration module is used to configure the index based on the knowledge content vector to form an enterprise-level knowledge governance standard.

[0046] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a computer program configured to be executed by the processor to implement the enterprise-level unstructured knowledge governance method as described in any of the preceding claims.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program configured to be executed by a processor to implement the enterprise-level unstructured knowledge governance method as described in any of the preceding claims.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, which are executed by a processor to implement the enterprise-level unstructured knowledge governance method as described in any of the preceding claims.

[0049] This application provides a method and computer-readable storage medium for enterprise-level unstructured knowledge governance. By parsing document structure using a visual language model, generating text slices using an intelligent segmentation algorithm, extracting metadata to construct knowledge vectors, and configuring a hybrid index, it effectively solves the problems of inaccurate layout parsing, disrupted logical structure, and low retrieval efficiency in traditional technologies, and has the advantages of improving the accuracy and usability of knowledge governance. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic flowchart of an embodiment of the enterprise-level unstructured knowledge governance method provided in this application.

[0052] Figure 2 This is a flowchart illustrating another embodiment of the enterprise-level unstructured knowledge governance method provided in the embodiments of this application.

[0053] Figure 3 This is an example diagram of a method for enterprise-level unstructured knowledge governance provided in the embodiments of this application.

[0054] Figure 4 This is a schematic diagram of an apparatus for enterprise-level unstructured knowledge governance provided in an embodiment of this application.

[0055] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, product, or apparatus that includes a series of steps or modules is not limited to the listed steps or modules but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such processes, methods, products, or apparatus.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply that all embodiments are the same, nor are they independent or alternative embodiments mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] This application provides a method for enterprise-level unstructured knowledge governance, which may include, but is not limited to, the following embodiments and combinations thereof.

[0060] Figure 1 This is a schematic flowchart of an embodiment of the enterprise-level unstructured knowledge governance method provided in this application, as shown in the example. Figure 1 As shown, the method for enterprise-level unstructured knowledge governance provided in this application embodiment may include, but is not limited to, the following steps and combinations thereof. The method includes:

[0061] Step 101: Input the initial document into the visual language model for layout parsing to obtain the document structure information of the target document.

[0062] Step 102: Segment the document structure information to generate text slices to be vectorized.

[0063] Step 103: Extract metadata based on text slices to generate structured knowledge content vectors.

[0064] Step 104: Configure the index based on the knowledge content vector to form an enterprise-level knowledge governance standard.

[0065] The visual language model can be a deep learning model that integrates image and text features. Specifically, it can be implemented using a multimodal model based on the Transformer architecture, extracting the spatial distribution and semantic relationships of page elements through joint training of an image encoder and a text decoder. Document structure information can be structured data including heading levels, paragraph distribution, and image-text positional relationships, stored in XML or JSON format to guide subsequent segmentation. Text slices can be document fragments with complete semantic boundaries, generated using a dynamic window sliding algorithm, with each slice containing continuous semantic units within a fixed character range. Knowledge content vectors can be semantic representations carrying metadata tags, encoded using pre-trained language models such as BERT, with vector dimensions of 768 or 1024. Index configuration can establish a searchable knowledge organization, implemented using a hybrid architecture of inverted index and vector database, supporting keyword matching and semantic similarity retrieval.

[0066] Specifically, the visual language model associates content across columns in two-column documents, identifying the logical order in mixed text and image layouts. Segmentation automatically divides chapter boundaries based on heading levels and adjusts slice lengths based on semantic coherence. In the metadata extraction phase, a summary and tags are generated for each slice, establishing a mapping between content vectors and the original document. The index configuration phase defines two storage strategies: a quality mode to ensure retrieval accuracy and an economy mode to optimize storage costs.

[0067] As an example, a specific method for enterprise-level unstructured knowledge governance can be a human-machine collaborative visual governance method based on RAGFLow, including the following steps: Step 1: Document parsing stage, including layout recognition and manual fine-tuning; for example, using a small visual language model (such as MinerU) to perform deep layout analysis on the input document. This model can accurately identify elements such as images, tables, text, titles, and formulas in the document by fine-tuning the layout recognition small model. The model automatically filters out non-textual interference items such as headers, footers, page numbers, and footnotes, and outputs a continuous text stream that conforms to the reading order. The small model has less than 2 billion parameters (2 Billion, 2B), but it surpasses the traditional 72B level visual language model in parsing accuracy, ensuring high-precision parsing of complex multi-column formatted documents. Step 2: Segmentation design, including intelligent segmentation. For example, the intelligent document segmentation system employs the filler object design pattern to achieve separation of responsibilities: the core algorithm uses stack-based reverse processing and dynamic capacity management, intelligently segmenting document content into fixed-size buckets through a greedy filling strategy. It also integrates a multi-dimensional constraint checker (character limit, semantic integrity, heading hierarchy) and a semantic-aware decision engine to ensure the integrity of structured content such as tables and code blocks. Post-processing algorithms such as bucket merging optimization and forced segmentation of excessively long documents improve performance, ultimately achieving unified intelligent segmentation processing of multimodal content such as text, images, tables, code, and formulas, with a time complexity of O(n). It supports various strategies such as automatic segmentation, item-by-item segmentation, and custom delimiters. Step 3: Metadata Extraction. For example, the metadata extraction module is located in the later stages of the knowledge base pipeline, responsible for semantic compression and feature annotation of the text after slicing. By automatically generating metadata such as summaries, tags, contextual references, and question prompts for each slice, the knowledge slices possess higher semantic density and precise positioning capabilities in subsequent retrieval and recall. This module combines the semantic understanding capabilities of large-scale models with traditional text analysis algorithms, balancing performance and intelligence, and is a key step in improving the accuracy and recall of knowledge base question answering. Step 4: Index Configuration. For example, the index configuration module is the final stage of the knowledge base pipeline, responsible for vectorizing the structured knowledge content and storing it in a high-performance vector database. By embedding vector encoding the text, it achieves rapid semantic matching and recall, enabling large-scale models to efficiently retrieve the most relevant knowledge content in question answering and retrieval scenarios. This step is a crucial bridge connecting knowledge governance and the performance of large-scale model question answering, determining the system's performance in retrieval accuracy, response speed, and scalability.

[0068] This application reconstructs the original document layout structure using a visual language model, solving the problem of disordered parsing order in traditional OCR; it employs a dynamic segmentation algorithm instead of fixed character slices to maintain semantic integrity; it establishes a structured metadata system to improve knowledge vector retrieval efficiency; and it provides a configurable indexing strategy to adapt to different business scenarios. Thus, this application achieves accurate structural parsing of complex formatted documents, avoiding ambiguity in question-answering caused by content fragmentation; the dynamic segmentation algorithm makes knowledge slices conform to human reading logic, improving the relevance of retrieval results; the structured metadata system enhances knowledge correlation and supports cross-document content tracing; and the configurable indexing strategy reduces storage resource consumption and improves system operating efficiency.

[0069] In some embodiments, inputting an initial document into a visual language model for layout parsing to obtain document structure information of a target document includes: inputting the initial document into a visual language model for initialization processing to obtain a target document; extracting the initial structured layout information of the target document; formatting the initial structured layout information to obtain target structured layout information; and performing layout management processing on the target structured layout information to obtain document structure information.

[0070] Among them, the visual language model can be a deep learning model that can process image and text information simultaneously. Specifically, it can be implemented using a multimodal model based on the Transformer architecture. Its role is to analyze the relationship between the visual layout of a document and its text content.

[0071] Initialization processing can involve format conversion and data cleaning of the original document. This can be achieved using a PDF parser or image preprocessing algorithm, and its purpose is to eliminate the interference of document format differences on subsequent processing.

[0072] The initial structured layout information can be intermediate data containing text positions, font attributes, and layout element coordinates. Specifically, it can be extracted through layout analysis algorithms. Its purpose is to establish a mapping relationship between document content and physical layout.

[0073] Formatting can be an operation that converts heterogeneous layout information into a unified data structure. Specifically, it can be implemented using a JSON or XML format converter. Its purpose is to ensure that the structured information of documents from different sources is processable.

[0074] Page layout management involves adjusting the logical relationships between page elements. This can be achieved through rule engines or machine learning models, and its purpose is to restore the reading order and logical structure of a document.

[0075] Specifically, a visual language model is used to extract multimodal features from the initial document, identifying text regions, image locations, and layout features to generate initial structured layout data containing coordinate information. Further, layout information in different formats is converted into a unified structured data format to eliminate structural differences between document sources. Subsequently, based on the document's hierarchical relationships and visual features, the logical order of layout elements is reconstructed and the nested structure is adjusted, ultimately outputting document structure information containing complete semantic relationships.

[0076] As an example, the system employs a multi-model fusion layout recognition framework, including: LayoutLMv3 and DocLayout_YOLO models: recognizing the document's logical structure, paragraph boundaries, and visual layout; ch_PP and OCRv4: responsible for OCR extraction of text from images; Rapid_Table: used for complex table recognition and cell relationship modeling; and Unimernet_Small and YOLO_v8_MFD: performing mathematical formula detection and recognition. After multi-model collaborative processing, the system generates a structured content list, including text blocks, tables, images, and their location information, forming a semantically and visually aligned multimodal content structure. Specific steps include: Model loading: MinerU loads layout recognition and OCR models during initialization to detect text blocks, images, tables, and other areas in the document. Layout parsing: The system performs page-level analysis of the document, identifying the type, location, and reading order of each block, and extracting structured layout information. Structure output: The recognition results are converted into JSON or Markdown format, retaining block type, coordinates, and content information for subsequent segmentation and indexing. Page Layout Management: Noise from headers, footers, page numbers, etc., is filtered out using rules, and text blocks are logically reorganized to form a semantically clear document structure. Structured Information Preservation: MinerU accurately identifies structural units such as titles, paragraphs, tables, and images in a document, preserving their hierarchical relationships and positional information, providing clear semantic boundaries for subsequent metadata extraction and knowledge indexing. Slicing Processing Optimization: By recognizing the logical block boundaries in the layout, intelligent segmentation based on semantics and layout can be achieved during the chunking stage, avoiding sentence truncation or cross-table splitting issues, significantly improving the accuracy of vectorization and retrieval.

[0077] This application captures the global layout features of documents through a visual language model, combines formatting to normalize multi-source data, and reconstructs the logical structure based on layout governance, effectively solving the problem of content misalignment in complex typesetting scenarios. Thus, this application can accurately extract document structure information containing multi-column layouts, mixed text and images, and nested tables, avoiding reading order confusion caused by traditional methods. This provides a correct logical boundary foundation for subsequent knowledge slicing, significantly improving the parsing accuracy and processing efficiency of unstructured documents.

[0078] In some embodiments, segmenting document structure information to generate text slices to be vectorized includes: configuring segmentation constraint parameters based on document structure information and constructing an initial segmentation algorithm; preprocessing the content of the target document to generate iterative content flow information; filling the content flow information into the initial segmentation algorithm using a preset greedy strategy to determine the segmentation boundary information of the document structure information; optimizing the initial segmentation algorithm based on the segmentation boundary information to obtain a target segmentation algorithm; and generating text slices based on the target segmentation algorithm.

[0079] The segmentation constraint parameters can be configuration parameters used to control the text segmentation generation rules. Specifically, they can be implemented using visual layout feature thresholds, semantic coherence thresholds, or character length thresholds to ensure that the segmentation results conform to the visual structure and semantic logic of the document.

[0080] Content stream information can be preprocessed serialized document content data, specifically implemented as paragraph stream, sentence stream, or word stream. It forms an iterative data structure by removing redundant characters and unifying the encoding format.

[0081] Greedy strategies can be used as decision-making mechanisms for dynamically adjusting segmentation boundaries. Specifically, they can be implemented by combining local optimal selection algorithms with semantic similarity calculations. The optimal segmentation position is determined by gradually filling in content stream information and evaluating boundary conditions.

[0082] Segmentation boundary information can be the key markers for dividing text into segments. Specifically, it can be implemented using heading level transition points, paragraph end marks, or semantic turning points, to guide the segmentation algorithm to divide content units while maintaining semantic integrity.

[0083] Specifically, firstly, based on the layout and semantic features extracted from the document structure information, segmentation constraint parameters are set to construct an initial segmentation algorithm. The target document, after preprocessing, forms content flow information, which is input into the initial segmentation algorithm. A greedy strategy dynamically evaluates the attribution of each element in the content flow; for example, segmentation is triggered when a change in heading level or semantic incoherence is detected, thus determining the segmentation boundaries. Subsequently, excessively small text slices generated in the initial algorithm are merged and optimized to form the final target segmentation algorithm. Based on the optimized algorithm, the document content is segmented to generate text slices that conform to visual structure and semantic logic.

[0084] As an example, the algorithm initializes as follows: The system loads segmentation configuration and constraint parameters, constructs filler objects, and initializes a stacked content cache and dynamic capacity controller. Content parsing and preprocessing: The system parses the structural information of the input document (headings, paragraphs, tables, code blocks, etc.), generating an iterable content stream to provide structural identifiers for subsequent segmentation. Greedy filling and dynamic segmentation: A greedy strategy is used to sequentially fill content into buckets of fixed size; segmentation boundaries are dynamically determined based on constraints such as character length, semantic boundaries, and heading levels. Structural integrity protection: A multi-dimensional constraint checker is integrated to ensure that structured content such as tables, code blocks, and formulas are not truncated. Post-processing optimization: Redundant slices are reduced through a bucket merging strategy; forced splitting is performed on excessively long paragraphs to ensure overall balance. Output and strategy selection: The final segmentation result is output, supporting three strategies: automatic segmentation, segmentation by item, and custom delimiter, all uniformly used in the subsequent embedding and index building stages.

[0085] This application integrates layout analysis results with semantic analysis technology, simultaneously considering visual layout features and contextual semantic relationships during the segmentation process. This ensures that the generated text slices retain both the original document's layout structure and the logical coherence of the content. Thus, this application effectively solves the problem of cross-column content misalignment caused by neglecting visual layout during automated segmentation, and avoids semantic fragmentation caused by simply relying on character count. In contract documents containing nested tables, it can accurately identify table boundaries and maintain the integrity of the table structure; in two-column layouts of scientific papers, it can correctly reconstruct the reading order based on layout analysis results, preventing paragraph misalignment. This improves the accuracy of subsequent vectorization processing and knowledge retrieval, providing question-answering systems with knowledge units that have complete semantic support.

[0086] In some embodiments, a preset greedy strategy is used to fill the content flow information into the initial segmentation algorithm to determine the segmentation boundary information of the document structure information, including: filling the content flow information into the initial segmentation algorithm using a greedy strategy to obtain the title level information, content type information and semantic boundary information of the target document; and determining the segmentation boundary information based on the title level information, content type information and semantic boundary information.

[0087] Among them, the greedy strategy can be an algorithm that achieves global optimization by gradually selecting the current optimal solution. Specifically, it can be implemented using a dynamic window matching algorithm to quickly locate the optimal segment node during the content flow filling process.

[0088] Heading hierarchy information can be the hierarchical structure data of headings in a document. Specifically, it can be achieved by recognizing font size, bolding style, and position coordinates through a visual language model, and is used to divide the logical boundaries of chapters.

[0089] Content type information can be a classification identifier for content such as text, tables, or images. Specifically, it can be implemented using a classification model based on convolutional neural networks to avoid incorrect segmentation of different types of content.

[0090] Semantic boundary information can be the start and end markers of natural paragraphs or logical units. Specifically, it can be achieved by calculating the semantic similarity threshold between adjacent sentences to ensure that the segmented text slices maintain semantic coherence.

[0091] Specifically, during the initial segmentation algorithm execution, the content flow information is broken down into a continuous sequence of text units. A dynamic window matching algorithm sequentially fills the current segmentation window with these text units. When a window simultaneously meets the conditions for a title level transition, a content type switch, or a semantic similarity below a preset threshold, a segmentation boundary marker is triggered. For example, when a second-level title is detected and the subsequent text unit is a table, the system automatically inserts a segmentation boundary after the title; when the semantic similarity of consecutive text units is below a critical value, it is determined to be the end point of a natural paragraph and a boundary marker is generated.

[0092] As an example, the application employs a greedy filling and dynamic segmentation strategy: A greedy strategy is used to sequentially fill content into buckets of fixed size; segmentation boundaries are dynamically determined based on constraints such as character length, semantic boundaries, and heading levels. Structural integrity protection is achieved through an integrated multi-dimensional constraint checker, ensuring that structured content such as tables, code blocks, and formulas are not truncated. Post-processing optimization reduces redundant slices through a bucket merging strategy; forced segmentation is performed on excessively long paragraphs to maintain overall balance. Output and strategy selection: The final segmentation result is output, supporting three strategies: automatic segmentation, segmentation by item, and custom delimiters, all uniformly used in subsequent embedding and indexing stages. Structured information preservation in this application maximizes the preservation of the original text's hierarchy and logical relationships by protecting the structure of heading levels, tables, code blocks, formulas, etc., ensuring traceability and contextual coherence of knowledge slices. Improved slice efficiency and accuracy: After intelligent segmentation, each slice achieves a balance between semantic density and content independence, reducing knowledge fragmentation, improving matching accuracy in the vectorization and recall stages, and providing higher-quality contextual input for large models. Multi-strategy adaptation: Strategies such as automatic segmentation and custom delimiters meet the segmentation needs of different document types (technical documents, contracts, graphic reports, etc.), improving the versatility and automation level of the pipeline.

[0093] This application, by integrating visual layout features and semantic association analysis, can accurately identify hierarchical changes in chapter titles, content type switching nodes, and logical endpoints of natural paragraphs, thereby maintaining the semantic integrity of knowledge slices in complex layout scenarios. In this way, this application effectively solves the problems of paragraph misalignment and cross-page content breaks caused by ignoring visual layout and semantic associations during automated slicing, ensuring that the boundaries of knowledge slices remain consistent with the actual logical structure of the document, providing an accurate data foundation for subsequent vectorization processing and index construction. For example, when parsing a two-column layout research paper, the system can automatically skip the blank areas between columns based on the title hierarchy, avoiding cutting the same logical paragraph into multiple fragments; when processing contracts containing cross-page tables, it can completely preserve the table structure information.

[0094] In some embodiments, optimizing the initial segmentation algorithm based on segmentation boundary information to obtain a target segmentation algorithm includes: merging slices with fewer than a preset threshold of characters in the initial segmentation algorithm with adjacent slices based on segmentation boundary information to obtain the target segmentation algorithm.

[0095] The segmentation boundary information can be a logical division identifier determined by analyzing the document title level, content type and semantic relationship. Specifically, it can be achieved by using a natural language processing model to identify chapter titles, paragraph transition words or contextual relevance, in order to accurately capture the semantic integrity boundary of the document content.

[0096] Slices with fewer than a preset threshold of characters can be short text fragments filtered by statistical text length. Specifically, this can be achieved using dynamic threshold setting rules, such as setting the lower limit of the number of characters to a range of 200-500 characters based on the document type, to identify fragmented content caused by mechanical segmentation.

[0097] The merging operation can reorganize overly short slices and adjacent content into a single text unit. Specifically, it can be achieved by using a doubly linked list data structure to traverse adjacent nodes and perform a concatenation operation, thereby eliminating non-semantic cuts caused by algorithm misjudgment.

[0098] Specifically, during the segmentation algorithm optimization phase, excessively short segments resulting from the initial segmentation are identified using segmentation boundary information. For example, supplementary explanations in a contract clause might be segmented separately due to insufficient character count. At this point, a preset threshold is used to determine if the segment length is below the standard. If it meets the condition, it is merged with the adjacent main clause segment. This process iterates through the semantic relationships in the segmentation boundary information, prioritizing the merging of adjacent segments with the strongest logical coherence. For example, merging the footnote paragraph below a table with the description paragraph above the table avoids splitting content across pages.

[0099] This application effectively solves the context fragmentation problem caused by mechanical segmentation by dynamically adjusting the merging strategy in conjunction with semantic boundary information, while preserving algorithm efficiency. Thus, this application can eliminate semantic fragmentation caused by excessively short slices during knowledge governance, such as merging scattered supplementary clauses into the main clause unit, ensuring the integrity of contextual information during subsequent vectorization processing, thereby improving the accuracy of knowledge retrieval and question answering.

[0100] In some embodiments, metadata extraction based on text slices is performed to generate structured knowledge content vectors, including: semantic compression and feature annotation of text slices to generate meta-hint information of text slices; and generating knowledge content vectors based on meta-hint information.

[0101] Semantic compression can be achieved by extracting the core semantic information of text slices through natural language processing technology and reducing redundant content. Specifically, it can be implemented using a text summarization model based on an attention mechanism, which preserves the core semantics of the text and reduces the computational complexity of subsequent processing.

[0102] Feature annotation can be used to structurally label text slices to enhance their retrieval. Specifically, it can be implemented using named entity recognition algorithms and context association analysis algorithms to extract key information from the text and establish semantic relationships.

[0103] Meta-hints can be structured descriptive information that includes summaries, tags, and contextual references. Specifically, they can be generated through a multimodal feature fusion framework to provide multi-dimensional semantic support for knowledge content vectors.

[0104] Specifically, when generating knowledge content vectors, the text slices are first semantically compressed, and a text summarization model is used to extract core semantics to form summary information. Then, a context window analysis algorithm is used to obtain the key content of the preceding and subsequent slices, generating contextual reference information to maintain semantic coherence. Simultaneously, a named entity recognition algorithm is used to extract topic words and keywords from the text, forming tag information. Finally, the summary information, tag information, and contextual reference information are integrated into meta-hint information, and a vector encoder is used to generate a multi-dimensional knowledge content vector.

[0105] This application preserves core semantics through semantic compression, ensures semantic coherence across paragraphs by combining contextual references, and enhances vector interpretability through tag information, effectively solving the semantic fragmentation problem caused by traditional methods. Thus, this application can generate knowledge content vectors with complete semantic representations, avoiding semantic fragmentation caused by text slicing, and improving the accuracy of knowledge retrieval and the response quality of question-answering systems. Simultaneously, the generation of structured meta-hints supports refined management of unstructured knowledge and multi-dimensional retrieval needs.

[0106] In some embodiments, the meta-hint information includes summary information, tag information, and contextual reference information; semantic compression and feature annotation are performed on text slices to generate meta-hint information for the text slices, including: semantic compression of the text slices to extract summary information that retains the core semantics of the text slices; encapsulation of the text slices using a context window to generate contextual reference information; the context window includes key information of the preceding and subsequent slices of the text slice; calling a feature tag generation algorithm on the text slices to extract topic words and keywords from the text slices; and determining tag information based on topic words and keywords.

[0107] Semantic compression involves removing redundant information while retaining core semantics using natural language processing techniques. Specifically, it can be implemented using attention-based text summarization models to address the low storage efficiency caused by retaining excessive redundant information in traditional methods. A context window is a logical container containing information about the relationships between the current slice and its preceding and following segments. Specifically, a sliding window algorithm can be used to dynamically capture key information from adjacent slices, addressing fragmentation caused by semantic breaks across pages or paragraphs. Feature label generation algorithms are operational rules for extracting structured labels from text. Specifically, they can be implemented using keyword extraction algorithms based on word frequency-inverse document frequency combined with named entity recognition technology to improve the retrieval and classification efficiency of knowledge content.

[0108] Specifically, when generating meta-hints, the original text is first transformed into a summary containing core semantics using a semantic compression algorithm, for example, compressing a 500-word paragraph into a 50-word summary. Next, a context window mechanism is employed, encapsulating key information from the two preceding and following slices to form logically related contextual references. Simultaneously, a feature label generation algorithm extracts technical terms and high-frequency words from the text as labels, such as extracting keywords like "neural network" and "training parameters" from technical documents. The resulting meta-hints support accurate matching of knowledge vectors in multi-dimensional retrieval.

[0109] As an example, context window encapsulation: Before slice vectorization, the context information of each slice is embedded in the form of meta_data. When a match is found during the retrieval phase, the context can be traced based on the meta_data to achieve paragraph-level semantic expansion and enhanced understanding. Summary generation: High-density summaries are generated using large models (such as Qwen, GPT, and DeepSeek), retaining core semantics and removing redundant descriptions. The summary results are stored in the meta_data field of the vector library for semantic enhancement and ranking optimization during retrieval. Tag extraction: Feature tag generation algorithms are called to extract topic words and keywords from the slice content. Multi-tag parallel annotation is supported to support keyword retrieval, filtering, aggregation statistics, and other scenarios. Semantic density improvement in this application: Summarization compression makes the information of each slice more concentrated, significantly improving the context utilization of the large model. Enhanced retrieval accuracy: Context window and tag information improve the interpretability and matching accuracy of vector recall. Question answering effect optimization: The automatic question generation mechanism enables the knowledge base to have QA training closed-loop capability, providing a data foundation for subsequent fine-tuning or self-optimization. Clear and traceable structure: All metadata is stored in a structured form in the vector library, which facilitates log analysis, recall optimization, and knowledge visualization.

[0110] This application employs a context window encapsulation mechanism to establish logical connections between slices while preserving core semantics, effectively resolving the semantic fragmentation problem that occurs during cross-page content retrieval. Furthermore, semantic compression-based summary generation, compared to simple first-sentence extraction, more accurately reflects the core content of the text. Thus, this application addresses the semantic fragmentation problem caused by missing context in unstructured document governance, improving the semantic matching accuracy of knowledge vectors during retrieval. Simultaneously, by automatically generating structured metadata, it significantly reduces the workload of manual annotation and ensures the traceability and reusability of knowledge content across different business scenarios.

[0111] In some embodiments, indexing is configured based on knowledge content vectors to form enterprise-level knowledge governance standards, including: transforming knowledge content vectors to obtain multi-dimensional vectors; constructing an index on the multi-dimensional vectors to generate hybrid retrieval information that supports both structured query language and vector query language; the index construction includes at least one of quality model index construction and economic model index construction; and configuring a retrieval strategy based on the hybrid retrieval information to generate knowledge governance standards.

[0112] Among these, multi-dimensional vectors can be vector representations with multiple independent semantic dimensions generated by decoupling knowledge content vectors through features and recombining attributes. Specifically, tensor decomposition algorithms or orthogonal projection algorithms can be used to separate the semantic features of different attributes in the knowledge content for constructing composite indexes. Hybrid retrieval information can be a joint index structure that simultaneously supports relational retrieval in structured query languages ​​and semantic similarity retrieval in vector query languages. Specifically, it can be implemented using a joint storage architecture of inverted indexes and vector indexes, achieving cross-modal retrieval by establishing a mapping relationship between the two query languages. Quality-mode index construction can be an indexing strategy with retrieval accuracy as the core optimization objective. Specifically, it can be implemented using hierarchical clustering algorithms and redundant storage mechanisms, improving recall by increasing index levels and the number of replicas. Economy-mode index construction can be an indexing strategy with storage efficiency and query speed as the core optimization objectives. Specifically, it can be implemented using quantization compression algorithms and approximate nearest neighbor algorithms, improving throughput by reducing vector dimensions and computational complexity.

[0113] Specifically, the knowledge content vector is decomposed into three independent dimensions—title vector, body vector, and metadata vector—using a tensor decomposition algorithm. Each dimension is then indexed with both an inverted index and a vector index to form a composite storage structure. In quality mode, the title vector uses a three-layer clustering structure to build its index while retaining a copy of the original vector. The body vector employs a redundant sharding storage strategy, and the metadata vector has a full inverted index. In economy mode, the title vector is compressed using 8-bit quantization to build a single-layer index, the body vector uses a product quantization algorithm to reduce storage space, and the metadata vector retains only the inverted indexes for key fields. Structured Query Language (SCL) accesses the inverted index by parsing SQL statements to obtain exact matching results, while Vector Query Language (VCL) retrieves similar vectors using an approximate nearest neighbor algorithm. The two retrieval results are then weighted and fused to form the final output. The retrieval strategy configuration module dynamically adjusts the index ratio between quality mode and economy mode based on enterprise storage resource limitations and response time requirements. For example, it enables quality mode indexes when storage server resources are sufficient and switches to economy mode indexes when concurrent query pressure is high.

[0114] As an example, text vectorization (Embedding encoding): Transforms text into a multi-dimensional vector representation using a specified embedding model. Two indexing modes are provided: High-quality mode: Calls models such as Azure GPT-3.5 Embedding to ensure high-precision semantic expression. Economy mode: Employs local vector engines (such as bge-base, text2vec) or keyword indexing, reducing costs but slightly decreasing accuracy. Index building: Builds similarity search indexes (such as HNSW, Faiss, Annoy) to optimize retrieval performance. The generated index files are bound to slice metadata, enabling knowledge traceability and tracking. Vector database access: Milvus mode: Provides distributed high-concurrency retrieval capabilities, supporting K8s elastic deployment and multi-replica expansion. Supports various index structures (HNSW, IVF_FLAT, DISKANN, etc.) to adapt to different latency and accuracy requirements. Oracle mode: Utilizes Oracle AI Vector Search to embed semantic retrieval into the enterprise's existing database system without introducing new components. Supports mixed SQL + vector query retrieval, enabling integrated access to structured data and unstructured knowledge. Search strategy configuration: Set parameters such as similarity threshold and top_k recall. Supports multi-dimensional filtering by knowledge base, tag, or source. This application significantly improves search accuracy: semantic-level vector search is more accurate than traditional keyword matching, significantly reducing false positives and hallucinatory answers. Stable and scalable performance: Achieves elastic scaling through solutions such as Milvus / Oracle, supporting rapid retrieval of millions of documents. Private domain knowledge security and controllability: The indexing process and database deployment are both within the enterprise's private environment, ensuring data security and traceability. Supports unified access from multiple sources: Provides a unified vectorization channel for knowledge sources such as Lark, DingTalk, Confluence, and Notion, supporting cross-source retrieval.

[0115] This application achieves cross-modal joint queries through a hybrid index architecture, while the configurable combination of quality and economic modes can dynamically optimize resource allocation based on actual business scenarios. Thus, this application resolves the contradiction between complex query needs and system resource constraints in enterprise knowledge bases, enabling the knowledge retrieval system to support complex condition queries such as "finding documents containing specific technical clauses in 2023 contracts with a semantic similarity higher than 85%", and to automatically switch between high-precision and high-performance modes based on server load. Furthermore, by separating index construction strategies for different dimensions, the storage redundancy of vector indexes is effectively reduced. For example, after indexing metadata fields independently, the storage space for tabular documents can be reduced by approximately 40% without affecting the query efficiency of key fields.

[0116] In some embodiments, the index configuration includes drag-and-drop metadata configuration; the method further includes: constructing a database of drag-and-drop metadata configuration; the database includes text components, table components, image components, and formula components; combining at least one of the text components, table components, image components, and formula components to construct a reusable structured information extraction template for knowledge content vectors; and determining knowledge governance targets based on the structured information extraction template.

[0117] The drag-and-drop metadata configuration allows users to combine predefined components on demand to generate metadata templates through a visual interface. Specifically, a graphical editor can be used to implement component drag-and-drop and parameter binding, reducing the complexity of manual configuration. The text component is a functional module for extracting plain text paragraphs and their attributes, implemented using regular expression matching or semantic segmentation algorithms to capture narrative content in documents. The table component is a processing unit for recognizing and parsing table structures, implemented using an attention-based table detection model to preserve row and column relationships. The image component is a processing module for extracting image features and annotation information, implemented using convolutional neural networks to recognize logical relationships in diagrams. The formula component is a functional unit for parsing mathematical symbols and formula semantics, implemented using a LaTeX parser or symbol recognition model to maintain the structural integrity of formulas. The structured information extraction template is a standardized processing flow formed by combining multiple components, defined using JSON or XML formats to define component connections, enabling batch processing of similar documents.

[0118] Specifically, a component library containing four basic components is established within the knowledge governance system. Users can select the required components through a visual interface and arrange them by dragging and dropping. For example, when processing research papers, text components can be combined with formula components to form an academic document template, automatically extracting text paragraphs and mathematical formulas; when processing financial statements, table components can be combined with image components to form a financial document template, simultaneously capturing data tables and trend charts. Each component has built-in standardized processing logic, automatically activating the corresponding feature extraction algorithm when a component is dragged into the workspace. By setting the data flow and dependencies between components, an end-to-end structured information extraction pipeline is formed. This template can be saved as a system-level configuration and directly called when processing similar documents in subsequent calls, avoiding the need to repeatedly define processing rules.

[0119] In some specific implementations, the table component can be configured to retain header associations and automatically detect merged cells; the image component can set resolution thresholds and feature extraction depths; and the text component can define segmentation strategies and keyword filtering rules. When processing technical documents containing nested tables, users can drag and drop two linked table components. The first component extracts the outer table, and the second component processes the embedded tables, passing contextual information through a data pipeline.

[0120] As an example, this application uses a drag-and-drop metadata configuration control to enable business users to visually define, edit, and preview metadata extraction logic on the front end, building flexible and reusable structured information extraction templates. It offers a WYSIWYG configuration experience: metadata fields are defined and extraction rules, field types, and output formats are configured via drag-and-drop components. An intelligent mapping engine binds each field to corresponding text features or model output logic, automatically matching and identifying key content. A real-time preview and verification mechanism allows users to view extraction results in real-time during configuration and quickly correct any abnormal content. Templated and reusable capabilities enable extraction rules to be saved as templates, supporting cross-project calls and forming an enterprise-level knowledge governance standard.

[0121] This application utilizes a component-based design, enabling non-technical personnel to intuitively build customized templates, overcoming the limitations of traditional methods that require coding and template modification. Simultaneously, the built-in professional processing algorithms overcome the technical shortcomings of general-purpose parsers in areas such as table reconstruction and formula recognition, ensuring the structural accuracy of extracted information. Thus, this application achieves rapid construction and reuse of unstructured document governance templates, significantly reducing manual configuration workload. For document types across different business scenarios within an enterprise, suitable extraction templates can be quickly assembled, avoiding resource waste caused by repetitive processing of similar documents. The component-based design ensures the integrity and consistency of knowledge content extraction, providing reliable structured data support for subsequent knowledge retrieval and question answering.

[0122] As an example, a specific method for enterprise-level unstructured knowledge governance could be a method based on multi-source data from enterprise-level AI4BI, using RAGFLow for human-machine collaborative visual governance, such as... Figure 2 As shown, Figure 2 This is a flowchart illustrating another embodiment of the enterprise-level unstructured knowledge governance method provided in the embodiments of this application, which includes, for example:

[0123] First, the document parsing stage, including layout recognition and manual fine-tuning, employs a small visual language model (such as MinerU) to perform deep layout analysis on the input document. This model, through fine-tuning the layout recognition mini-model, can accurately identify elements such as images, tables, text, titles, and formulas within the document. The model automatically filters out non-textual distractions such as headers, footers, page numbers, and footnotes, outputting a continuous text stream that conforms to the reading order. Despite having fewer than 2 bytes of parameters, the mini-model surpasses traditional 72-byte visual language models in parsing accuracy, ensuring high-precision parsing of complex multi-column layout documents. The system uses a multi-model fusion layout recognition framework, including: LayoutLMv3 and DocLayout_YOLO models: recognizing the document's logical structure, paragraph boundaries, and visual layout; ch_PP and OCRv4: responsible for OCR extraction of text from images; Rapid_Table: used for complex table recognition and cell relationship modeling; and Unimernet_Small and YOLO_v8_MFD: performing mathematical formula detection and recognition. After multi-model collaborative processing, the system generates a structured content list, including text blocks, tables, images, and their location information, forming a multimodal content structure aligned semantically and visually. The specific steps include: Model Loading: MinerU loads layout recognition and OCR models during initialization to detect text blocks, images, tables, and other areas in the document. Layout Parsing: The system performs page-level analysis of the document, identifying the type, location, and reading order of each block, and extracting structured layout information. Structure Output: The recognition results are converted into JSON or Markdown format, retaining block type, coordinates, and content information for subsequent segmentation and indexing. Layout Governance: Noise such as headers, footers, and page numbers is filtered through rules, and text blocks are reorganized in logical order to form a semantically clear document structure. Among these, structured information retention: MinerU can accurately identify structural units such as titles, paragraphs, tables, and images in the document, and retain their hierarchical relationships and location information, providing clear semantic boundaries for subsequent metadata extraction and knowledge indexing. Slicing optimization: By recognizing the logical block boundaries of the output through layout, intelligent segmentation based on semantics and layout can be achieved in the chunking stage, avoiding sentence truncation or cross-table splitting issues, and significantly improving the accuracy of vectorization and retrieval. For example, Figure 3 This is an example diagram from an embodiment of the method for enterprise-level unstructured knowledge governance provided in this application. For example... Figure 3 As shown, the top image is the original image, and the bottom image is the content blocks such as titles, paragraphs, and images recognized by the machine. Users can compare the top and bottom images and manually correct any deviations in the machine's recognition below.

[0124] Second, the segmentation design includes intelligent segmentation: The intelligent document segmentation system adopts the filler object design pattern to achieve separation of responsibilities. The core algorithm uses stack-based reverse processing and dynamic capacity management. Through a greedy filling strategy, it intelligently segments document content into fixed-size buckets. It also integrates a multi-dimensional constraint checker (character limit, semantic integrity, heading hierarchy) and a semantic-aware decision engine to ensure the integrity protection of structured content such as tables and code blocks. Performance is improved through post-processing algorithms such as bucket merging optimization and forced splitting of extremely long documents. Ultimately, it achieves unified intelligent segmentation processing of multimodal content such as text, images, tables, code, and formulas, with a time complexity of O(n). It supports various strategies such as automatic segmentation, item-by-item segmentation, and custom delimiters. Specific steps include: Algorithm initialization: The system loads segmentation configuration and constraint parameters, constructs filler objects, and initializes the stack-based content cache and dynamic capacity controller. Content parsing and preprocessing: Parses the structural information of the input document (headings, paragraphs, tables, code blocks, etc.), generating an iterable content stream to provide structural identifiers for subsequent segmentation. Greedy Filling and Dynamic Segmentation: A greedy strategy is used to sequentially fill content into buckets of fixed size; segmentation boundaries are dynamically determined based on constraints such as character length, semantic boundaries, and heading levels. Structural Integrity Protection: A multi-dimensional constraint checker is integrated to ensure that structured content such as tables, code blocks, and formulas are not truncated. Post-Processing Optimization: Redundant slices are reduced through a bucket merging strategy; forced segmentation is performed on excessively long paragraphs to ensure overall balance. Output and Strategy Selection: The final segmentation result is output, supporting three strategies: automatic segmentation, segmentation by item, and custom delimiters, all uniformly used in subsequent embedding and index building stages. Structured Information Preservation: By protecting the structure of heading levels, tables, code blocks, formulas, etc., the original text's hierarchy and logical relationships are preserved to the maximum extent, ensuring traceability and contextual coherence of knowledge slices. Segmentation Efficiency and Accuracy Improvement: After intelligent segmentation, each slice achieves a balance between semantic density and content independence, reducing knowledge fragmentation, improving matching accuracy in vectorization and recall stages, and providing higher-quality contextual input for large models. Multi-strategy adaptation: Strategies such as automatic segmentation and custom delimiters meet the segmentation needs of different document types (technical documents, contracts, graphic reports, etc.), improving the versatility and automation level of the pipeline.

[0125] Third, metadata extraction. Located in the later stages of the knowledge base pipeline, the metadata extraction module is responsible for semantic compression and feature annotation of the text after slicing. By automatically generating meta-information such as summaries, tags, contextual references, and question hints for each slice, the knowledge slices possess higher semantic density and precise positioning capabilities in subsequent retrieval and recall. This module combines the semantic understanding capabilities of large models with traditional text analysis algorithms, balancing performance and intelligence, and is a key step in improving the accuracy and recall rate of knowledge base question answering. Specific steps include: Context window encapsulation: Before slice vectorization, the context information of each slice is embedded in the form of meta_data. When a match is found during the retrieval stage, the context can be traced based on the meta_data to achieve paragraph-level semantic expansion and enhanced understanding. Summary generation: High-density summaries are generated using large models (such as Qwen, GPT, and DeepSeek), retaining core semantics and removing redundant descriptions. The summary results are stored in the meta_data field of the vector library for semantic enhancement and ranking optimization during retrieval. Tag extraction: Feature tag generation algorithms are called to extract topic words and keywords from the slice content. It supports multi-label parallel annotation for scenarios such as keyword retrieval, filtering, and aggregation statistics. Key features include: improved semantic density: summary compression makes each slice's information more concentrated, significantly improving the context utilization of large models. Enhanced retrieval accuracy: context windows and label information improve the interpretability and matching accuracy of vector recall. Optimized question-answering performance: the automatic question generation mechanism enables the knowledge base to have QA training closed-loop capabilities, providing a data foundation for subsequent fine-tuning or self-optimization. Clear and traceable structure: all metadata is stored in a structured form in the vector library, facilitating log analysis, recall optimization, and knowledge visualization.

[0126] Fourth, index configuration. The index configuration module is the final stage of the knowledge base pipeline, responsible for vectorizing the structured knowledge content and storing it in a high-performance vector database. By embedding vector encoding of the text, it achieves rapid semantic matching and retrieval, enabling large models to efficiently retrieve the most relevant knowledge content in question-answering and retrieval scenarios. This stage is a key bridge connecting knowledge governance and the question-answering performance of large models, determining the system's performance in retrieval accuracy, response speed, and scalability. Specific steps include: Text vectorization (Embedding encoding): Using a specified embedding model to convert text into a multi-dimensional vector representation. Two indexing modes are provided: High-quality mode: Calling models such as AzureGPT-3.5 Embedding to ensure high-precision semantic expression. Economy mode: Using a local vector engine (such as bge-base, text2vec) or keyword indexing to reduce costs but with a slight decrease in accuracy. Index construction: Building a similarity search index (such as HNSW, Faiss, Annoy) to optimize retrieval performance. The generated index file is bound to slice metadata to achieve knowledge traceability and tracking. Vector Database Integration: Milvus Mode: Provides distributed high-concurrency retrieval capabilities, supporting elastic deployment and multi-replica expansion in Kubernetes. Supports various index structures (HNSW, IVF_FLAT, DISKANN, etc.) to adapt to different latency and accuracy requirements. Oracle Mode: Embeds semantic retrieval into the enterprise's existing database system using Oracle AI Vector Search, without introducing additional new components. Supports hybrid retrieval of SQL + vector queries, enabling integrated access to structured data and unstructured knowledge. Retrieval Strategy Configuration: Sets parameters such as similarity threshold and top_k recall. Supports multi-dimensional filtering by knowledge base, tag, or source. Significantly Improved Retrieval Accuracy: Semantic vector search is more accurate than traditional keyword matching, significantly reducing false positives and phantom responses. Stable and Scalable Performance: Achieves elastic scaling through Milvus / Oracle solutions, supporting rapid retrieval of millions of documents. Secure and Controllable Private Knowledge: The indexing process and database deployment are both within the enterprise's private environment, ensuring data security and traceability. Supports unified access from multiple sources: Provides a unified vectorized channel for knowledge sources such as Lark, DingTalk, Confluence, and Notion, and supports cross-source retrieval.

[0127] Fifth, drag-and-drop functionality. For example, a drag-and-drop metadata configuration control allows business users to visually define, edit, and preview metadata extraction logic on the front end, building flexible and reusable structured information extraction templates. WYSIWYG configuration experience: Define metadata fields, configure extraction rules, field types, and output formats by dragging and dropping components. Intelligent mapping engine: Each field is bound to a corresponding text feature or model output logic, automatically matching and identifying key content. Real-time preview and verification mechanism: Users can view extraction results in real time during configuration and quickly correct any abnormal content. Templated and reusable capabilities: Extraction rules can be saved as templates, supporting cross-project calls and forming enterprise-level knowledge governance standards. This application, by introducing a drag-and-drop visual configuration interface, achieves low-threshold operation and dynamic arrangement of various stages in knowledge base construction (such as layout recognition, intelligent segmentation, metadata extraction, index configuration, etc.). Users can freely combine data flow nodes by dragging and dropping components, flexibly define processing order and parameters, and complete complex pipeline configurations without writing code. The system supports real-time preview and parameter linkage, significantly reducing communication costs for technical and business personnel. Meanwhile, the drag-and-drop design makes dependencies between modules clearer, facilitating maintenance and reuse. This solution not only improves configuration efficiency and operability but also enhances the interpretability and standardization of the process, making the knowledge base construction process more efficient, transparent, and intelligent.

[0128] This embodiment utilizes small visual models such as MinerU to accurately recover the layout structure and reading order of complex documents, ensuring the complete extraction of the main text content. Combined with table of contents information to guide slicing, knowledge paragraphs are strictly aligned with the original text structure. RAGFlow's adaptive chunking templates and human-machine collaborative review mechanism complement each other; automatic chunking combined with expert review effectively avoids over-segmentation or omissions. The result is that each knowledge fragment contains independent and coherent semantic units, improving the effectiveness of vector retrieval and similarity calculation. RAGFlow evaluation is introduced to perform multi-dimensional quality measurement of the generated answers. By automatically detecting the consistency and relevance of the answer to the retrieval context, the reasonableness of the generated results can be effectively controlled, reducing the occurrence of illusory information and improving the credibility of the question-answering system. It is not dependent on specific companies or products and is suitable for various complex document layout scenarios such as medical literature, technical manuals, and financial reports. The complete automated and manual review process can improve efficiency and reduce the cost of human error in the construction of large-scale enterprise knowledge bases.

[0129] Secondly, based on the enterprise-level unstructured knowledge governance method described in the above embodiments, embodiments of this application provide an apparatus for enterprise-level unstructured knowledge governance. The apparatus for enterprise-level unstructured knowledge governance is used to execute the steps in any of the embodiments of the enterprise-level unstructured knowledge governance method described above. For example... Figure 4 As shown, Figure 4 A schematic diagram of an apparatus for enterprise-level unstructured knowledge governance provided in this application embodiment. The apparatus 400 for enterprise-level unstructured knowledge governance may include:

[0130] The parsing module 401 is used to input the initial document into the visual language model for layout parsing to obtain the document structure information of the target document;

[0131] Processing module 402 is used to segment the document structure information to generate text slices to be vectorized;

[0132] Extraction module 403 is used to extract metadata based on the text slices and generate a structured knowledge content vector;

[0133] Configuration module 404 is used to configure the index based on the knowledge content vector to form an enterprise-level knowledge governance standard.

[0134] The specific process in this embodiment can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0135] Thirdly, in order to realize the enterprise-level unstructured knowledge governance method of the embodiments of this application, Figure 5 A schematic diagram of an electronic device provided in an embodiment of this application; as shown Figure 5 As shown, the electronic device 500 may include: a memory 501 for storing computer programs; and a processor 502 for implementing any of the methods described above when executing the computer program. For example, the processor 502 may be used to: input an initial document into a visual language model for layout parsing to obtain document structure information of the target document; segment the document structure information to generate text slices to be vectorized; extract metadata based on the text slices to generate structured knowledge content vectors; and configure indexes according to the knowledge content vectors to form enterprise-level knowledge governance standards. The processor 502 may also implement any of the steps in the methods described above, which will not be elaborated further here.

[0136] It should be noted that the electronic devices provided in the above embodiments and the method embodiments for enterprise-level unstructured knowledge governance belong to the same concept. For details of their specific implementation process, please refer to the method embodiments, which will not be repeated here.

[0137] Of course, in practical applications, such as Figure 5As shown, the electronic device 500 may further include at least one network interface 503. Various components in the electronic device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 Various buses are labeled as bus systems 504. The number of processors 502 can be at least one. Network interface 503 is used for wired or wireless communication between electronic devices and other devices. Memory 501 in this embodiment is used to store various types of data to support the operation of the electronic device. The methods disclosed in the above embodiments can be applied to processor 502, or implemented by processor 502. Processor 502 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 502 or by instructions in software form. The processor 502 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 502 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected in the combined execution of hardware and software modules in a microcontroller. The software module may reside in a storage medium located in memory 501. Processor 502 reads information from memory 501 and, in conjunction with its hardware, completes the steps of the aforementioned method. In an exemplary embodiment, electronic device 500 may be implemented using one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0138] Specifically, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, such as a memory 501 storing the computer program, which can be executed by a processor 502 to complete the aforementioned method steps. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0139] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0140] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0143] The above provides a detailed description of the method and medium for enterprise-level unstructured knowledge governance provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for enterprise-level unstructured knowledge governance, characterized in that, The method includes: The initial document is input into a visual language model for layout parsing to obtain the document structure information of the target document; the visual language model is a multimodal model that integrates image features and text features and is based on a converter model architecture; the document structure information is structured data that includes heading levels, paragraph distribution, and the positional relationship between text and images; The document structure information is segmented to generate text slices to be vectorized; Metadata is extracted from the text slices to generate a structured knowledge content vector; the knowledge content vector includes a semantic representation vector carrying metadata tags. Based on the knowledge content vector, an index configuration is performed to form an enterprise-level knowledge governance standard; the index configuration includes drag-and-drop metadata configuration. The step of extracting metadata based on the text slices to generate a structured knowledge content vector includes extracting metadata based on the text slices through semantic compression, feature annotation, and context window encapsulation, generating a structured knowledge content vector containing summary information, tag information, and context reference information; the context reference information is generated through context window encapsulation; the context window includes key information of the preceding and subsequent slices of the text slices; The step of segmenting the document structure information to generate text slices to be vectorized includes: Based on the document structure information, configure segmentation constraint parameters and construct an initial segmentation algorithm; The content of the target document is preprocessed to generate iterable content stream information; A preset greedy strategy is used to fill the content stream information into the initial segmentation algorithm to determine the segmentation boundary information of the document structure information; The initial segmentation algorithm is optimized based on the segmentation boundary information to obtain the target segmentation algorithm; The text slices are generated based on the target segmentation algorithm; The process of configuring an index based on the knowledge content vector to form an enterprise-level knowledge governance standard includes: The knowledge content vector is transformed to obtain a multi-dimensional vector; The multi-dimensional vector is indexed to generate hybrid retrieval information that supports both structured query language and vector query language; the index construction includes at least one of quality pattern index construction and economic pattern index construction. Based on the hybrid retrieval information, a retrieval strategy is configured to generate the knowledge governance standard. The knowledge content vector is decomposed into three independent dimensions—title vector, text vector, and metadata vector—using a tensor decomposition algorithm. Each dimension is then used to construct an inverted index and a vector index, forming a composite storage structure. In quality mode, the title vector uses a three-layer clustering structure to build an index while retaining a copy of the original vector. The text vector employs a redundant sharding storage strategy, and the metadata vector establishes a full inverted index. In economy mode, the title vector is compressed using 8-bit quantization to build a single-layer index, the text vector uses a product quantization algorithm to reduce storage space, and the metadata vector retains only the inverted index of key fields. Structured query language accesses the inverted index by parsing SQL statements to obtain precise matching results, and vector query language retrieves similar vectors using an approximate nearest neighbor algorithm. The two retrieval results are weighted and fused to form the final output. The index ratio between quality mode and economy mode is dynamically adjusted according to enterprise storage resource limitations and response time requirements.

2. The method for enterprise-level unstructured knowledge governance according to claim 1, characterized in that, The step of inputting the initial document into a visual language model for layout parsing to obtain the document structure information of the target document includes: The initial document is input into the visual language model for initialization processing to obtain the target document; Extract the initial structured layout information of the target document; The initial structured layout information is formatted to obtain the target structured layout information; The target structured layout information is processed to obtain the document structure information.

3. The method for enterprise-level unstructured knowledge governance according to claim 1, characterized in that, The step of using a preset greedy strategy to fill the content stream information into the initial segmentation algorithm and determining the segmentation boundary information of the document structure information includes: The greedy strategy is used to fill the content stream information into the initial segmentation algorithm to obtain the title level information, content type information and semantic boundary information of the target document; The segmentation boundary information is determined based on the title level information, the content type information, and the semantic boundary information.

4. The method for enterprise-level unstructured knowledge governance according to claim 1, characterized in that, The step of optimizing the initial segmentation algorithm based on the segmentation boundary information to obtain the target segmentation algorithm includes: Based on the segmentation boundary information, the slices with fewer than a preset threshold in the initial segmentation algorithm are merged with adjacent slices to obtain the target segmentation algorithm.

5. The method for enterprise-level unstructured knowledge governance according to claim 1, characterized in that, The step of extracting metadata based on the text slices to generate a structured knowledge content vector includes: The text slices are semantically compressed and feature-annotated to generate meta-hint information for the text slices; The knowledge content vector is generated based on the meta-hint information.

6. The method for enterprise-level unstructured knowledge governance according to claim 5, characterized in that, The meta-hint information includes summary information, tag information, and contextual reference information; the step of semantically compressing and characterizing the text slice to generate the meta-hint information for the text slice includes: Semantic compression is performed on the text slices to extract and retain the summary information that preserves the core semantics of the text slices; The text slice is encapsulated using a context window to generate the context reference information; The feature label generation algorithm is applied to the text slice to extract the topic words and keywords from the text slice; The tag information is determined based on the topic words and keywords.

7. The method for enterprise-level unstructured knowledge governance according to any one of claims 1-6, characterized in that, The method further includes: Construct the database for the drag-and-drop metadata configuration; the database includes text components, table components, image components, and formula components; At least one of the text component, table component, image component, and formula component is combined to construct a reusable structured information extraction template for the knowledge content vector; The knowledge governance standard is determined based on the structured information extraction template.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to be executed by a processor to implement the enterprise-level unstructured knowledge governance method according to any one of claims 1 to 7.

Images