Method for constructing enterprise large model knowledge base
By classifying and dynamically segmenting the enterprise's large-scale knowledge base, the problems of time-consuming and inaccurate traditional retrieval methods are solved, achieving efficient and accurate retrieval results and secure data access control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU INT SCI PARK DATA CENT CO LTD
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-17
AI Technical Summary
Traditional keyword retrieval is time-consuming and inaccurate. Fixed-length blocks disrupt the semantic structure of documents, resulting in incomplete retrieval recall and inability to obtain accurate search results.
Data sources are categorized by file type, and different processing methods are used to extract text and coordinate information. Semantic unit integrity is preserved through dynamic segmentation and length control. An embedding model is used to convert the data into high-dimensional vectors and create an index. A re-ranking model is then used for retrieval and answer generation.
It enables efficient and accurate analysis of multiple document types, ensuring the comprehensiveness and accuracy of search results. It supports multiple input formats, is compatible with mixed languages, and provides secure data access control.
Smart Images
Figure CN121880486A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method for constructing a large enterprise model knowledge base. Background Technology
[0002] Building a large model knowledge base is a core component of AI implementation for enterprises, aiming to transform static data resources into dynamic intelligence for models. However, this process still faces a series of significant shortcomings at the data, technology, and architecture levels: traditional keyword retrieval is time-consuming and inaccurate. For example, keyword retrieval on a Windows system can only identify content that matches the entered keywords, and it cannot effectively retrieve relevant content when there are missing words or typos; in addition, traditional document data processing usually involves fixed-length blocks (such as cutting every 512 tokens), which can easily damage the semantic structure of documents (such as breaking up a complete process, table description, or policy clause), resulting in incomplete retrieval recall and failing to obtain relatively accurate search results. Summary of the Invention
[0003] The purpose of this invention is to provide a method for constructing a large enterprise knowledge base to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, one technical solution adopted by the present invention is: a method for constructing a large enterprise model knowledge base, comprising the following steps: S1: Data access and extraction: The data from the data source is classified according to file type, and different types of files are processed by a different processing method. The file types include PDF, Word, PPT, Excel, internal enterprise database, enterprise email and OA work order; The PDF was extracted using unstructured [loca0.l-inference] and PaddleOCR methods to extract plain text and coordinate information; Word, PPT, and Excel files are output as structured paragraphs using python-docx, python-pptx, and openpyxl, respectively. The enterprise's internal database uses a JDBC / ODBC connector and SQL templates to output JSON row records; The enterprise email and OA work order system extracts and outputs conversational text blocks through IMAP / API and regular expressions; S2: Data chunking processing: including file structure parsing, semantic unit extraction, dynamic chunking and length control, and metadata injection. The file structure parsing uses corresponding parsing tools to construct document object trees for different types of files. The dynamic chunking and length control uses a sliding window and semantic boundary alignment algorithm to segment semantic units exceeding max_tokens. The metadata injection provides structured metadata for each chunk. S3: Vectorization and Storage: Use an embedding model to convert the processed text blocks into high-dimensional vectors, store the vectors in a pre-made vector database and create an index; S4: Retrieval and Generation: The user query text is converted into a vector through the embedding model. First, the keyword screening is performed to output candidate results. Then, the vector semantic retrieval is performed. The re-ranking model is used to re-rank the retrieval results. The fusion strategy is used to calculate the final score and select no less than 5 results as the context of the large model. The answer is generated in combination with the user query. S5: Security Control: During the retrieval process, filtering is performed based on permission and timeliness information in the metadata to ensure that users can only access authorized data that is within the time limit.
[0005] Preferably, in step S2, when extracting semantic units, the smallest semantically complete unit is extracted as the basis for candidate blocks, including complete chapters with titles, independent paragraphs, structured lists, tables, figure captions, and formula descriptions.
[0006] Preferably, in step S2, in the dynamic block segmentation and length control, max_tokens is set to 512, the number of overlapping characters is set to 64, and the semantic boundaries include periods, newlines, and the beginning of list items.
[0007] Preferably, in step S2, the data includes document ID, document title, chapter path, heading level, element type, source format, page number, effective time, permissions, and source link.
[0008] Preferably, in step S3, the vectorized data includes document ID, document title, chapter path, heading level, element type, source format, page number, effective time, permissions, and source link.
[0009] Preferably, in step S4, when a user submits a query, the query text is first converted into a vector using the same embedding model. In the vector database, initial keyword screening outputs at least 20 candidate results, followed by vector semantic retrieval. The search results are reordered using a re-ranking model, and a final fusion strategy is adopted. The calculation rules are as follows: Final score = 0.3 * BM25_norm + 0.7 * Rerank_score; The top 5 results are taken as the LLM context, and the context is combined with the user query to form a suggestion, which is then sent to the large model to generate an answer.
[0010] Preferably, the large model uses Qwen3, which concatenates the first three paragraphs into a context when generating the answer, adds an identifier before each paragraph, and marks the source number after each sentence in the answer. If there is no relevant information in the data, the answer is "According to the existing knowledge base, this question cannot be answered".
[0011] Preferably, in step S2, when processing an Excel document, the Excel document is converted into a dual representation of descriptive text and structured data, which preserves semantics and facilitates understanding by the embedding model.
[0012] Preferably, in step S2, for cross-page content, the chars coordinate information of pdfplumber is used to determine whether it is a cross-page content of the same paragraph. If so, it is merged into a logical paragraph and then divided into blocks.
[0013] Preferably, in step S2, for cases involving multiple languages, the text is not segmented by language during the block division process, and the original mixed text state is preserved.
[0014] The beneficial effects of this invention are as follows: This solution classifies common file types and extracts data from them separately, achieving efficient analysis of multiple file types. At the same time, by using dynamic block segmentation and length control, and employing sliding window and semantic boundary alignment algorithms to segment semantic units exceeding max_tokens, the semantic coherence of document data is ensured, resulting in more comprehensive and accurate retrieval results. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the construction and use of the knowledge base of this invention; Figure 2 This is a code example diagram illustrating the method of constructing a document object tree and predicting the heading level in this invention. Figure 3 This is a code example diagram illustrating dynamic segmentation and length control in the method of this invention; Figure 4 This is a code example diagram illustrating the method of this invention for attaching structured metadata to each chunk; Figure 5 This is a code example diagram illustrating how the embedding model is used in the method of this invention to convert text blocks into vectors; Figure 6 This is a code example diagram illustrating the method of this invention, which involves writing vectors into a database and creating an index. Figure 7 This is a code example diagram illustrating the vector semantic retrieval process in the method of this invention; Figure 8 This is a code example diagram illustrating the reordering of search results in the method of this invention. Detailed Implementation
[0016] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.
[0017] Example: See Figure 1 A method for constructing a large-scale enterprise knowledge base includes the following steps: S1: Data access and extraction: The data from the data source is classified according to file type, and different types of files are processed by a different processing method. The file types include PDF, Word, PPT, Excel, internal enterprise database, enterprise email and OA work order; The PDF was extracted using unstructured [loca0.l-inference] and PaddleOCR methods to extract plain text and coordinate information; Word, PPT, and Excel files are output as structured paragraphs using python-docx, python-pptx, and openpyxl, respectively. The enterprise's internal database uses a JDBC / ODBC connector and SQL templates to output JSON row records; The enterprise email and OA work order system extracts and outputs conversational text blocks through IMAP / API and regular expressions; This step enables comprehensive access to multi-source heterogeneous data, providing a rich data foundation for subsequent knowledge base construction. Specialized parsing methods are used for different data sources to ensure the accuracy and completeness of data extraction and avoid data loss or errors caused by differences in data formats.
[0018] S2: Data chunking processing: including file structure parsing, semantic unit extraction, dynamic chunking and length control, and metadata injection. The file structure parsing uses corresponding parsing tools to construct document object trees for different types of files. The dynamic chunking and length control uses a sliding window and semantic boundary alignment algorithm to segment semantic units exceeding max_tokens. The metadata injection provides structured metadata for each chunk. Structured-aware chunking preserves the logical structure of the original document, ensuring that each chunk has independent semantic integrity. This avoids the problem of traditional fixed-length chunking destroying the semantic structure of the document, which is beneficial to improving the accuracy of subsequent retrieval and the accuracy of answers generated by large models. The injection of metadata provides a key basis for subsequent retrieval filtering, information tracing, and security control.
[0019] When extracting semantic units, the smallest semantically complete unit is extracted as the basis for candidate blocks, including complete chapters with titles, independent paragraphs, structured lists, tables, figure captions, and formula descriptions.
[0020] The core objective of structure-aware chunking is to preserve the logical structure of the original document (heading levels, paragraph categories, list items, etc.); ensure that each chunk has independent semantic integrity; support efficient vectorization and accurate retrieval; and be compatible with multiple input formats (PDF, Word, HTML, Markdown, etc.).
[0021] Long texts are divided into smaller blocks (e.g., 1000 characters per block, with 200 characters overlapping) using text chunking algorithms (such as recursive text segmentation, fixed-size overlapping chunking, etc.). The purpose of chunking is to adapt to the context window of a large model and improve retrieval accuracy. First, the document structure is parsed, and the heading levels (H1~H3) are extracted using langchain.document_loaders to construct the document tree, and then dynamic chunking is performed.
[0022] The core steps are as follows: 1. Document Structure Parsing: The following processing methods are used for different document formats (PDF / Word / HTML / Markdown) (see Table 1): Table 1 Taking PDF documents as an example, a document object tree is constructed, and the heading level is predicted using the LayoutLMv3 model. (See [link / reference]). Figure 2 It demonstrates a simple document output result.
[0023] 2. Semantic Unit Extraction: Perform a depth-first traversal of the document tree to extract the smallest semantically complete unit as the basis for candidate chunking: complete chapters with titles, independent paragraphs, structured lists, tables, and figure / formula descriptions.
[0024] 3. Dynamic Segmentation and Length Control: When a single semantic unit exceeds `max_tokens` (e.g., a long policy document), further segmentation is required, while maintaining local contextual coherence. This can be achieved using sliding windows and semantic boundary alignment algorithms. Figure 3The code demonstrates its implementation steps. The workflow is as follows: flatten the node into plain text (preserving newline / list symbols), convert the text into a token sequence, if the text length is less than or equal to the maximum number of tokens, return the complete block directly, store the block results, calculate the step size, cut out candidate windows, search forward for the nearest semantic boundary (period, newline, beginning of list item), while avoiding excessively short chunks, decode the token into text, construct the block and add metadata, and calculate the starting point of the next window: the actual end position minus the overlap.
[0025] 4. Metadata Injection: Structured metadata is appended to each chunk for subsequent retrieval, filtering, and tracing. See [link / reference]. Figure 4 It assumes a document and parses it to obtain relevant results.
[0026] In some special scenarios, special processing is required, such as table processing: converting tables into a dual representation of "descriptive text + structured data" to preserve semantics and facilitate understanding by the embedding model; for cross-page content: using the chars coordinate information of pdfplumber to determine whether it is a cross-page paragraph of the same paragraph, and if so, merging it into a logical paragraph and then dividing it into blocks; for multilingual content: not cutting according to language when dividing into blocks, preserving the original mixed state, and using a multilingual support model (such as bge-m3) for the embedding model.
[0027] S3: Vectorization and Storage: The processed text blocks are converted into high-dimensional vectors using an embedding model. These vectors are then stored in a vector database that supports payload filtering and indexed. The embedding model is BAAI / bge-large-zh-v1.5, and the vector database is Qdrant. Vectorization transforms text information into a computer-efficient vector form, supporting semantic similarity retrieval and achieving high recall and relevance in Chinese scenarios. The use of the Qdrant vector database, which supports payload filtering, facilitates subsequent filtering operations based on metadata. Furthermore, its support for HNSW and quantization compression technologies saves memory, and it provides gRPC / REST dual protocols to meet enterprise-level application requirements. In this step, an embedding model (BAAI / bge-large-zh-v1.5) is used to convert text blocks into 1024-dimensional vectors (high-dimensional floating-point arrays). The input format for each chunk is: {section_title}{content}. Finally, embedding code is generated; see the specific code example below. Figure 5 .
[0028] After generating the embedding code, the vectors need to be stored in the vector database Qdrant. This step involves batch inserting vector data into the Qdrant database, generating unique IDs using UUIDs, supporting metadata filtering (permissions field), and automatically converting NumPy arrays to list format. Finally, it is integrated into the Enterprise Knowledge Base v1 collection. See the detailed operation code for more information. Figure 6 .
[0029] S4: Retrieval and Generation: The user query text is converted into a vector using the embedding model. Initial keyword screening outputs candidate results, followed by vector semantic retrieval. A re-ranking model is used to re-rank the retrieval results. A fusion strategy is employed to calculate the final score, and at least five results are selected as the context of the larger model. This is then combined with the user query to generate an answer. The re-ranking model is BAAI / bge-reranker-v2-m3, and the fusion strategy is as follows: The final score = 0.3BM25_norm + 0.7Rerank_score. BM25_norm refers to the normalized BM25 score in hybrid retrieval (such as a fusion of BM25 and dense retrieval), used for weighted fusion with other retrieval methods (such as dense retrieval scores) after unifying their dimensions. Rerank_score is the relevance score output by the re-ranking model after refining the scoring of candidate documents, used to optimize the ranking of retrieval results. It is usually compared with the initial retrieval score (such as retrieval_score) to evaluate the actual gain of the re-ranking module. For example, in the Dify logs, document d1's initial score is 0.72, and after re-ranking, its score becomes 0.79, while document d2's score increases from 0.68 to 0.85. The final ranking result is adjusted based on the rerank_score.
[0030] The multi-stage retrieval process first narrows down the scope quickly through keyword screening, then improves relevance through vector semantic retrieval, and finally optimizes the results further through re-ranking models and fusion strategies to ensure that the most relevant and accurate information is obtained. Using the filtered results as the context of the large model can effectively avoid the large model from producing illusions, ensuring the accuracy and reliability of the generated answers, while clear source labeling facilitates information traceability.
[0031] The specific operation steps are as follows: When a user submits a query, the query text is first converted into a vector using the same embedding model. In the vector database, initial keyword screening (BM25) is performed to output the Top 50 candidates. Then, vector semantic retrieval (ANN) is performed using Qdrant queries. The specific operation code can be found in [reference needed]. Figure 7Then, the search results are re-ranked using a re-ranking model (BAAI / bge-reranker-v2-m3). The operation code can be found in [link to relevant documentation]. Figure 8 Finally, the top 5 scores are used as LLM contexts based on the final score calculation formula. These contexts, along with the user query, form a prompt and are sent to the large model to generate an answer. When generating the answer, the first three paragraphs are concatenated into the context, each paragraph is marked with an identifier, and each sentence in the answer is followed by a source number. If there is no relevant information in the data, the answer is "Based on the existing knowledge base, this question cannot be answered".
[0032] S5: Security Control: During the retrieval process, filtering is performed based on permission and time-sensitive information in the metadata to ensure that users can only access authorized data within its validity period. This step ensures that users can only access authorized data within its validity period through strict permission management and time-sensitive filtering, effectively preventing the leakage of sensitive corporate data, meeting the compliance requirements of heavily regulated industries such as finance and healthcare, and protecting the security of corporate intellectual property assets.
[0033] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A method for constructing a large-scale enterprise knowledge base, characterized in that: Includes the following steps: S1: Data access and extraction: The data from the data source is classified according to file type, and different types of files are processed by a different processing method. The file types include PDF, Word, PPT, Excel, internal enterprise database, enterprise email and OA work order; The PDF was extracted using unstructured [loca0.l-inference] and PaddleOCR methods to extract plain text and coordinate information; Word, PPT, and Excel files are output as structured paragraphs using python-docx, python-pptx, and openpyxl, respectively. The enterprise's internal database uses a JDBC / ODBC connector and SQL templates to output JSON row records; The enterprise email and OA work order system extracts and outputs conversational text blocks through IMAP / API and regular expressions; S2: Data chunking processing: including file structure parsing, semantic unit extraction, dynamic chunking and length control, and metadata injection. The file structure parsing uses corresponding parsing tools to construct document object trees for different types of files. The dynamic chunking and length control uses a sliding window and semantic boundary alignment algorithm to segment semantic units exceeding max_tokens. The metadata injection provides structured metadata for each chunk. S3: Vectorization and Storage: Use an embedding model to convert the processed text blocks into high-dimensional vectors, store the vectors in a pre-made vector database and create an index; S4: Retrieval and Generation: The user query text is converted into a vector through the embedding model. First, the keyword screening is performed to output candidate results. Then, the vector semantic retrieval is performed. The re-ranking model is used to re-rank the retrieval results. The fusion strategy is used to calculate the final score and select no less than 5 results as the context of the large model. The answer is generated in combination with the user query. S5: Security Control: During the retrieval process, filtering is performed based on permission and timeliness information in the metadata to ensure that users can only access authorized data that is within the time limit.
2. The method for constructing a large-scale enterprise knowledge base according to claim 1, characterized in that: In step S2, when extracting semantic units, the smallest semantically complete unit is extracted as the basis for candidate blocks, including complete chapters with titles, independent paragraphs, structured lists, tables, figure captions, and formula descriptions.
3. The method for constructing a large-scale enterprise knowledge base according to claim 1, characterized in that: In step S2, the dynamic block segmentation and length control set max_tokens to 512, the number of overlapping characters to 64, and the semantic boundaries to include periods, newlines, and the beginning of list items.
4. The method for constructing a large enterprise knowledge base according to claim 2, characterized in that: In step S2, the data includes document ID, document title, chapter path, heading level, element type, source format, page number, effective date, permissions, and source link.
5. The method for constructing a large enterprise knowledge base according to claim 4, characterized in that: In step S3, the vectorized data includes document ID, document title, chapter path, heading level, element type, source format, page number, effective date, permissions, and source link.
6. The method for constructing a large enterprise model knowledge base according to claim 1, characterized in that: In step S4, when a user submits a query, the query text is first converted into a vector using the same embedding model. In the vector database, initial keyword screening outputs at least 20 candidate results, followed by vector semantic retrieval. The search results are reordered using a re-ranking model, and a final fusion strategy is adopted. The calculation rules are as follows: Final score = 0.3 * BM25_norm + 0.7 * Rerank_score; The top 5 results are taken as the LLM context, and the context is combined with the user query to form a suggestion, which is then sent to the large model to generate an answer.
7. The method for constructing an enterprise large-scale model knowledge base according to claim 6, characterized in that: The large model uses Qwen3. When generating an answer, the first three paragraphs are concatenated into a context, each paragraph is marked with an identifier, and each sentence in the answer is marked with a source number. If there is no relevant information in the data, the answer is "Based on the existing knowledge base, this question cannot be answered".
8. The method for constructing a large enterprise model knowledge base according to claim 1, characterized in that: In step S2, when processing Excel documents, the documents are converted into a dual representation of descriptive text and structured data, which preserves semantics and facilitates understanding by the embedding model.
9. The method for constructing a large-scale enterprise knowledge base according to claim 1, characterized in that: In step S2, for content spanning multiple pages, the chars coordinate information of pdfplumber is used to determine whether it is the same paragraph spanning multiple pages. If so, it is merged into a logical paragraph and then divided into blocks.
10. The method for constructing an enterprise large-scale model knowledge base according to claim 1, characterized in that: In step S2, for cases involving multiple languages, the chunks are not segmented by language, and the original mixed text state is preserved.