Method for semantic indexing and retrieval of enterprise knowledge base based on stable recall evidence

Abstract

The application discloses an enterprise knowledge base semantic index and retrieval method based on stable recall evidence, parses user files to construct context, extracts and post-processes stable recall evidence text according to file types; variable user organization information is excluded during vectorization input, only stable recall evidence is used, and document or chapter level vectorization is selected according to document structure; mixed candidate recall is executed within the effective scope of calculation; after sorting and rearranging the candidate set, the associated relationship is expanded to generate a to-be-verified candidate set; windowed reading is performed on the source file or the extracted text, and a large language model is returned to generate traceable results. The application separates stable evidence from variable information, cooperates with multi-path recall and effective scope constraint, improves retrieval stability, long document hit rate and answer traceability, and reduces repeated vectorization cost.

Description

Technical Field

[0001] This invention relates to the field of enterprise knowledge base data processing technology, and in particular to a semantic indexing and retrieval method for enterprise knowledge bases based on stable recall evidence. Background Technology

[0002] General-purpose RAG knowledge base systems (Retrieval-Augmented Generation) typically vectorize the full-text or segmented text directly after file upload, and then perform similarity matching between the user query vector and document vectors during retrieval. The retrieved text is then input into a large language model to generate an answer. The aforementioned existing technologies suffer from at least the following problems: If user-variable information such as filenames, directories, tags, and titles are included in vectorized input, frequent changes in vector indexes can easily occur when users rename, move files, or adjust tags, leading to unstable recall results and increasing the cost of repeated vectorization (embedding).

[0003] When vectorizing full text or plain text blocks directly, stable information such as key chapters, pages, worksheets, log levels, and time ranges in structured or semi-structured materials such as long documents, tables, logs, PPTs, PDFs, compressed files, and code repositories is easily diluted, resulting in a low hit rate for long document retrieval.

[0004] In multi-tenant and cross-organizational sharing scenarios, if permission filtering is only performed after retrieval, it may result in access to unauthorized data during candidate set generation, rearrangement, or large model context construction, increasing the risk of data isolation.

[0005] General retrieval systems typically only return text fragments, lacking a windowed reading and verification mechanism for the original source file. When large models generate answers, problems such as inaccurate citations and unverifiable sources are prone to occur.

[0006] Chinese corporate documents often contain short words, abbreviations, text without spaces, and mixed Chinese and English content. Relying solely on ordinary full-text search or vector search can easily lead to insufficient recall or unstable ranking.

[0007] Therefore, there is an urgent need for a semantic indexing and retrieval method for enterprise knowledge bases that separates stable recall evidence from variable user organizational information, while improving recall stability, long document hit capability, and answer verifiability, while ensuring the isolation of tenant, organization, knowledge base, and view scopes. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a semantic indexing and retrieval method for enterprise knowledge bases based on stable recall evidence. This method is applicable to multi-tenant enterprise knowledge bases, enabling the system to stably generate searchable semantic indexes when facing scenarios such as multi-format files, long documents, Chinese content, cross-organizational knowledge base sharing, dynamic tags, and directory changes. After retrieval, the method returns candidate results that can be verified by the original source file, thereby improving retrieval accuracy, reducing the cost of repetitive vectorization, and reducing the risk of untraceability in the large model question answering process.

[0009] To achieve the objectives of this invention, the technical solution adopted is as follows: A semantic indexing and retrieval method for enterprise knowledge bases based on stable recall evidence includes the following steps: Step S1: Receive the request source file uploaded by the user, parse the tenant identifier, organization identifier, user identifier, and knowledge base identifier to form a request context; Step S2: Select the corresponding extraction process according to the file type of the requested source file, perform post-processing on the extraction results, and generate stable recall evidence text. Step S3: When constructing the vectorized input, variable user organization information is excluded, and only stable recall evidence text is used as the vectorized input; and document-level or chapter-level vectorized input is selected according to the document structure. Step S4: Receive the retrieval request and calculate the effective scope of the current request; perform hybrid candidate recall within the effective scope to obtain the candidate set. Step S5: Sort and optionally rearrange the candidate set; Step S6: Expand the candidate set according to the item association relationships; Step S7: Generate a candidate set to be verified from the recall results and relationship expansion results; Step S8: Perform windowed reading on the original source file or extracted text and return it to the large language model; the large language model generates traceable answers or search results.

[0010] Further, in step S2, the stable recall evidence text is retrieval evidence derived from the file content version, including one or more combinations of file summary, document description, chapter information, keywords, OCR text, page visual description, table worksheet name and header, log level statistics, log time range, compressed package member preview, Git repository branch and commit summary.

[0011] Further, in step S3, it is determined whether the number of stable chapters has reached a preset threshold. When the number of stable chapters reaches the configured threshold, the system generates multiple vectorized inputs at the chapter level, with each input associated with a corresponding chapter identifier. When the number of chapters has not reached the threshold, the system generates a document-level vectorized input.

[0012] Further, step S4, performing hybrid candidate recall within the effective scope includes: Database full-text search recall based on abstracts, titles, and structured metadata; Downgraded recall of ternary grammar fuzzy matching algorithm for Chinese content; Metadata recall based on document digests, stable descriptions, and metadata; Semantic recall based on query vectors and stable recall evidence vectors; Assisted recall based on historical logs, parsed tags, and entry metadata; The candidate results retrieved from each path are merged according to document identifier or entry identifier, and the candidate source, matching field, full-text search score, semantic search score, and update time are recorded.

[0013] Further, in step S5, after the candidate set is initially sorted, if secondary sorting is enabled, a candidate window is selected within the effective scope, and a secondary sorted text is constructed using stable recall evidence or document digests. The secondary sorting model is then called to calculate the rearrangement score, and the secondary sorting process is continuously constrained by the effective scope.

[0014] Further, in step S8, windowed reading supports partial window reading of the original source file by page number, slide, line number, paragraph, chapter identifier, title, regular expression matching, or compressed package member path; if the content of the window being read is in a compressed state, a compression status mark is made, and a prompt to reopen the original window is given before the large language model references it.

[0015] A semantic indexing and retrieval system for an enterprise knowledge base based on stable recall evidence includes: The request parsing and context building module is used to receive the request source file uploaded by the user, parse the tenant identifier, organization identifier, user identifier and knowledge base identifier to form the request context; The extraction and post-processing module is used to select the corresponding extraction process according to the file type of the request source file, perform post-processing on the extraction results, and generate stable recall evidence text. The semantic indexing and vectorization building module is used to exclude variable user organization information when constructing vectorized input, using only stable recall evidence text as vectorized input, and selecting document-level or chapter-level vectorized input according to the document structure. The scope calculation and hybrid recall module is used to receive retrieval requests and calculate the effective scope of the current request, and perform hybrid candidate recall within the effective scope to obtain a candidate set; The sorting and rearranging module is used to sort and optionally rearrange the candidate set; The item relationship extension module is used to expand the candidate set based on item relationships; The candidate set generation module is used to generate a candidate set to be verified from the recall results and the relationship expansion results; The windowed reading and source generation module is used to perform windowed reading of the original source file or extracted text and return it to the large language model, which then generates traceable answers or search results.

[0016] A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the enterprise knowledge base semantic indexing and retrieval method based on stable recall evidence.

[0017] A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the enterprise knowledge base semantic indexing and retrieval method based on stable recall evidence.

[0018] The beneficial effects of the present invention are as follows: Compared with the prior art, the present invention has the following advantages: This invention uses only stable recall evidence for vectorized input, excluding variable information such as filenames, tags, and directories. Therefore, when users perform operations such as renaming, moving, or tagging files, the semantic vectors do not change frequently, improving index stability. This invention eliminates the need for re-vectorization due to changes in user-displayed attributes, reducing the number of repeated vectorizations and lowering the cost of calling vectorization services for large models and the cost of backend computation.

[0019] This invention employs chapter identifier-level vectorization for long documents, improving the hit rate of local topics in long PDFs, PPTs, DOCXs, logs, tables, and other materials, thus mitigating the problem of long documents being diluted by a single vector. The stable recall evidence of this invention includes summaries, chapters, OCR, visions, tables, logs, compressed files, and Git metadata. The system can handle various enterprise file types, improving the searchability of non-plain text materials.

[0020] In this invention, intelligent views, knowledge base ACLs, tenant and organizational scopes are applied during the full-text search, semantic search, secondary ranking, and reading stages, rather than just filtering in the final results, thus reducing the risk of cross-tenant or cross-organizational data leakage. This invention simultaneously uses full-text search, metadata retrieval, semantic search, and secondary ranking to improve recall stability for Chinese enterprise information, short queries, and mixed Chinese and English queries.

[0021] This invention generates a set of identifiers for items to be verified and supports windowed reading of source files. Large model responses can be verified by returning to the original file's page, line, paragraph, chapter, or member file, reducing the number of untraceable responses. This invention allows the original window to be reopened before citation, preventing compressed or abstract text from being directly used as exact citations, thus improving citation accuracy.

[0022] This invention utilizes session co-occurrence, tag overlap, and assistant reference co-occurrence to recall information that has business relevance to the initially hit document but whose keywords do not completely match, thereby improving the coverage of enterprise knowledge retrieval. Attached Figure Description

[0023] Figure 1 This is a flowchart of the semantic indexing and retrieval method for enterprise knowledge base based on stable recall evidence as described in this invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of this application.

[0025] Stable recall evidence in this invention refers to retrieval evidence derived from file content versions and relatively unaffected by user operations, including but not limited to: file summaries, document descriptions extracted by large language models or rules, chapter information, key terms, OCR text, image or page visual descriptions, table worksheet names and headers, log level statistics, log time ranges, compressed package member previews, Git repository branch and commit summaries, etc. Stable recall evidence does not include volatile information such as filenames, display names, user-defined titles, variable tags, user directories, AI-maintained semantic classification directories (AI catalogs), and entry metadata.

[0026] like Figure 1 As shown, the semantic indexing and retrieval method for enterprise knowledge bases based on stable recall evidence described in this invention includes the following steps: S1 receives the request source file uploaded by the user, parses the tenant identifier, organization identifier, user identifier and knowledge base identifier to form a request context; The system receives the request source file uploaded by the user, parses the tenant identifier, organization identifier, user identifier, and knowledge base identifier to form a request context. The system verifies the user's tenant, organization, and knowledge base access permissions. The raw bytes of the request source file are saved to object storage, and the database stores file asset records, file entry records, knowledge base ownership, organization ownership, storage address, hash value, file type, and audit records.

[0027] S2, perform the corresponding extraction process based on the file type, and after the file extraction is completed, perform post-processing on the extraction results and generate stable recall evidence text; The system selects the appropriate extraction process based on the file type. For different types of files, such as text, Markdown, PDF, DOCX, PPTX, email, CSV, TSV, JSON, NDJSON, Parquet, Excel, images, compressed files, log files, and code repository compressed files, it extracts the main text, structured metadata, page numbers, slides, paragraphs, chapters, worksheets, table headers, log time ranges, log levels, image sizes, OCR text, visual descriptions, compressed file member paths, Git repository branches, and commit information.

[0028] After document extraction is complete, the extraction results are post-processed to generate document summaries, structured descriptions, chapter lists, additional fields, OCR / vision evidence, and stable extraction metadata. The system combines these elements into stable recall evidence text. During the combination process, duplicate evidence is removed, and the length of individual evidence is limited to avoid excessively long inputs impacting vectorization costs and stability.

[0029] S3, when constructing vectorized input, excludes variable user organization information and uses only stable recall evidence derived from the document content version; selects document-level or chapter-level vectorized input based on the document structure; When constructing vectorized input, the system excludes variable fields such as full text, file name, display name, user title, tags, table of contents, AI category directory, and entry metadata, using only stable recall evidence derived from the file content version. This way, when a user only modifies the file name, moves the directory, adjusts tags, or changes display attributes, there is no need to regenerate semantic vectors.

[0030] The system determines whether the number of stable chapters has reached a preset threshold. For example, when the number of stable chapters reaches 20 or other configurable thresholds, the system generates multiple vectorized inputs at the chapter level, with each input associated with a corresponding chapter identifier (section_id). When the number of chapters has not reached the threshold, the system generates a single document-level vectorized input. Each vector record stores the organization identifier, document identifier, chapter identifier, model name, vector dimension, vector value, and stable evidence text hash value.

[0031] When a file is uploaded for the first time, reprocessed, or when stable recall evidence changes, the system deletes old vector records under the same document and model, and batch calls the vectorization service to generate new vectors. The system records the vector status, model, dimension, input source, granularity type, and number of chapters.

[0032] S4, Receive the retrieval request and calculate the effective scope, perform hybrid candidate recall within the effective scope, and obtain the candidate set; The system calculates the valid scope that the current request is allowed to access based on the tenant, organization, user role, knowledge base ACL, and view filtering rules (view.filter_spec). This scope is used for subsequent full-text search (lexical search), semantic search, secondary ranking, and source file reading, rather than just filtering at the final results stage.

[0033] The system verifies tenant, organization, and user roles, and establishes an access control context for the current request based on the verification results, thereby determining the scope of accessible knowledge bases, entries, and documents. Full-text search, metadata retrieval, semantic search, relation expansion, secondary sorting, and source file reading are all limited to the above access scope, and access permissions are verified again before source file reading; the database row-level security policy (RLS) also applies a second layer of isolation constraints to related queries.

[0034] The system performs multi-path recall within its effective scope, including: Lexical recall for full-text search of PostgreSQL databases based on abstracts, titles, and structured metadata. Fallback recall for CJK n-gram (Chinese-Japanese-Korean character set n-gram) or trigram / ILIKE (trigram fuzzy matching algorithm) for Chinese content. Metadata recall based on file digest, stable description and metadata. Semantic recall based on query vectors and stable recall evidence vectors. Assisted recall based on historical logs, parsed tags, and entry metadata.

[0035] The system merges candidates from different sources according to document identifiers or entry identifiers, and records the candidate source, matching field, full-text search score, semantic search score, and update time.

[0036] S5, sorting and optional rearrangement of the candidate set; The system first performs an initial ranking based on factors such as full-text search score, semantic search score, and update time. If a secondary ranking model is enabled, candidate windows are selected within the effective scope, and secondary-ranked texts are constructed using stable recall evidence or document summaries. The secondary ranking model is then invoked to obtain the re-ranked score. The re-ranked candidates still maintain the scope constraints of tenant, organization, knowledge base, and smart view.

[0037] S6, Expand the candidate set based on the relationship between the items; The system can expand the candidate set based on entry relationships. Relationship sources include conversation co-occurrence, tag overlap, and helper reference co-occurrence. The system records the number of observations for relationships and performs optional large-scale model review, using only relationships that pass the review for candidate expansion. When no large-scale language model is configured, relationships reaching a preset threshold can be marked as passed according to rules, thereby discovering supplementary materials with stable associations to the initially hit documents.

[0038] Methods for calculating the number of observations: Conversation co-occurrence: the number of times two items appear together in the conversation record within a preset period; Assistant citation co-occurrence: the number of times two items are cited by the same answer; Tag overlap: converted into observations based on the number of common tags and similarity.

[0039] S7, Generate a candidate set to be verified from the recall results and relationship expansion results; The system generates a set of candidate entry identifiers (candidate_entry_ids), a set of expanded entry identifiers (expansion_entry_ids), and a set of entry identifiers to be verified (verify_entry_ids) from the recall results and relationship expansion results. Verify_entry_ids is used for subsequent detailed verification and prioritizes high-confidence candidates and relationship expansion candidates, with its number limited by a preset upper limit.

[0040] S8 performs windowed reading of the original source file or extracted text and returns it to the large language model; the large language model generates traceable answers or search results. When the large language model or the caller needs to validate candidate content, the system reads the original source file or extracts text using the entry identifier (entry_id), and supports reading partial windows by page number, slide, line number, paragraph, chapter identifier, title, regular expression matching, and compressed file member path. For PDF, PPTX, table, log, compressed file, and image files, the system selects the corresponding reading method based on the file type.

[0041] The system returns candidate results, the source file window, the reference location, the entry identifier, the chapter identifier, and the page or line number to the caller or the large language model. When generating the answer, the large language model is based on the read source file window and may include reference information. If the read content is compressed, the system marks it as compressed and prompts that the original window must be reopened when citing it to ensure the accuracy of the cited text.

[0042] In one specific embodiment, the user inputs the question, "What click interaction events are there on the story management page in the 'Children's Fun Mode'?" Based on the access control context of the current request, the system performs candidate retrieval within the allowed knowledge base, identifying the user experience design document and product requirement document as relevant candidate documents. Then, according to the structure of the candidate documents, it reads the page windows related to story recommendations, story rankings, story details, and audio picture books. The system provides the user question, candidate document information, the read partial text, and the corresponding entry identifiers and page numbers to the large language model.

[0043] After summarizing the source file windows, the large language model determined that there was no "Story Management Page" with the exact same name in the source file. Related functions were distributed across multiple pages, and the following responses were generated: "On the Story Recommendation page, clicking the play button on an album card will directly play the corresponding album; clicking other areas of the card will take you to the album details page. On the Story Details page, clicking the 'Play All' button will continue playing the album; clicking a single program entry will play the corresponding program. On the Audio Picture Book page, clicking the 'Recommended Story' card will play the story; clicking the 'My Picture Books' entry will take you to the Historical Picture Book Records page." The system appended the corresponding source file identifier and page number to each response for the caller to return to the original file for verification.

[0044] This invention also provides a semantic indexing and retrieval system for an enterprise knowledge base based on stable recall evidence, comprising: The request parsing and context building module is used to receive the request source file uploaded by the user, parse the tenant identifier, organization identifier, user identifier and knowledge base identifier to form the request context; The extraction and post-processing module is used to select the corresponding extraction process according to the file type of the request source file, perform post-processing on the extraction results, and generate stable recall evidence text. The semantic indexing and vectorization building module is used to exclude variable user organization information when constructing vectorized input, using only stable recall evidence text as vectorized input, and selecting document-level or chapter-level vectorized input according to the document structure. The scope calculation and hybrid recall module is used to receive retrieval requests and calculate the effective scope of the current request, and perform hybrid candidate recall within the effective scope to obtain a candidate set; The sorting and rearranging module is used to sort and optionally rearrange the candidate set; The item relationship extension module is used to expand the candidate set based on item relationships; The candidate set generation module is used to generate a candidate set to be verified from the recall results and the relationship expansion results; The windowed reading and source generation module is used to perform windowed reading of the original source file or extracted text and return it to the large language model, which then generates traceable answers or search results.

[0045] A computing device includes: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement steps of semantic indexing and retrieval methods for an enterprise knowledge base based on stable recall evidence.

[0046] A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement steps for a semantic indexing and retrieval method for an enterprise knowledge base based on stable recall evidence.

[0047] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

Claims

1. A semantic indexing and retrieval method for an enterprise knowledge base based on stable recall evidence, characterized in that, Includes the following steps: Step S1: Receive the request source file uploaded by the user, parse the tenant identifier, organization identifier, user identifier, and knowledge base identifier to form a request context; Step S2: Select the corresponding extraction process according to the file type of the requested source file, perform post-processing on the extraction results, and generate stable recall evidence text. Step S3: When constructing the vectorized input, variable user organization information is excluded, and only stable recall evidence text is used as the vectorized input; and document-level or chapter-level vectorized input is selected according to the document structure. Step S4: Receive the retrieval request and calculate the effective scope of the current request; perform hybrid candidate recall within the effective scope to obtain the candidate set. Step S5: Sort and optionally rearrange the candidate set; Step S6: Expand the candidate set according to the item association relationships; Step S7: Generate a candidate set to be verified from the recall results and relationship expansion results; Step S8: Perform windowed reading on the original source file or extracted text and return it to the large language model; the large language model generates traceable answers or search results.

2. The semantic indexing and retrieval method for enterprise knowledge base based on stable recall evidence according to claim 1, characterized in that, Step S2, the stable recall evidence text is retrieval evidence derived from the file content version, including one or more combinations of file summary, document description, chapter information, keywords, OCR text, page visual description, table worksheet name and header, log level statistics, log time range, compressed package member preview, Git repository branch and commit summary.

3. The semantic indexing and retrieval method for enterprise knowledge base based on stable recall evidence according to claim 1, characterized in that, Step S3: Determine whether the number of stable chapters has reached a preset threshold. When the number of stable chapters reaches the configured threshold, the system generates multiple vectorized inputs at the chapter level, with each input associated with a corresponding chapter identifier. When the number of chapters has not reached the threshold, the system generates a document-level vectorized input.

4. The semantic indexing and retrieval method for enterprise knowledge base based on stable recall evidence according to claim 1, characterized in that, Step S4, performing hybrid candidate recall within the effective scope includes: Database full-text search recall based on abstracts, titles, and structured metadata; Downgraded recall of ternary grammar fuzzy matching algorithm for Chinese content; Metadata recall based on document digests, stable descriptions, and metadata; Semantic recall based on query vectors and stable recall evidence vectors; Assisted recall based on historical logs, parsed tags, and entry metadata; The candidate results retrieved from each path are merged according to document identifier or entry identifier, and the candidate source, matching field, full-text search score, semantic search score, and update time are recorded.

5. The semantic indexing and retrieval method for enterprise knowledge base based on stable recall evidence according to claim 1, characterized in that, Step S5: After the candidate set is initially sorted, if secondary sorting is enabled, a candidate window is selected within the effective scope, and a secondary sorted text is constructed using stable recall evidence or document digests. The secondary sorting model is called to calculate the rearrangement score, and the secondary sorting process is continuously constrained by the effective scope.

6. The semantic indexing and retrieval method for enterprise knowledge bases based on stable recall evidence according to claim 1, characterized in that, Step S8: Windowed reading supports partial window reading of the original source file by page number, slide, line number, paragraph, chapter identifier, title, regular expression matching, or compressed package member path; if the content of the window being read is compressed, the compression status is marked, and the original window is prompted to be reopened before the large language model references it.

7. A semantic indexing and retrieval system for an enterprise knowledge base based on stable recall evidence, used to implement the semantic indexing and retrieval method for an enterprise knowledge base based on stable recall evidence as described in any one of claims 1 to 6, characterized in that, include: The request parsing and context building module is used to receive the request source file uploaded by the user, parse the tenant identifier, organization identifier, user identifier and knowledge base identifier to form the request context; The extraction and post-processing module is used to select the corresponding extraction process according to the file type of the request source file, perform post-processing on the extraction results, and generate stable recall evidence text. The semantic indexing and vectorization building module is used to exclude variable user organization information when constructing vectorized input, using only stable recall evidence text as vectorized input, and selecting document-level or chapter-level vectorized input according to the document structure. The scope calculation and hybrid recall module is used to receive retrieval requests and calculate the effective scope of the current request, and perform hybrid candidate recall within the effective scope to obtain a candidate set; The sorting and rearranging module is used to sort and optionally rearrange the candidate set; The item relationship extension module is used to expand the candidate set based on item relationships; The candidate set generation module is used to generate a candidate set to be verified from the recall results and the relationship expansion results; The windowed reading and source generation module is used to perform windowed reading of the original source file or extracted text and return it to the large language model, which then generates traceable answers or search results.

8. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the enterprise knowledge base semantic indexing and retrieval method based on stable recall evidence as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the enterprise knowledge base semantic indexing and retrieval method based on stable recall evidence as described in any one of claims 1 to 6.

Images