Knowledge base version management method and system based on unified distributed index

Abstract

The invention discloses a knowledge base version management method and system based on a unified distributed index, and the method comprises the steps: constructing a unified distributed index and all version data used for a knowledge base, each document in the distributed index comprises knowledge slice content, a vector corresponding to the content, and a metadata field used for version distinguishing; and maintaining a mapping relationship between a semantic version number and a monotone increasing integer version ID on an application layer, creating new version preparation data, classifying new version modification contents, performing ID modification operation according to classification, releasing a new version, and releasing according to the modification and deletion intention recorded in the previous step. And meanwhile, the expired or invalid documents are cleaned, and through distributed indexing, the storage requirement is reduced, and the retrieval efficiency is remarkably improved.

Description

Technical Field

[0001] This invention relates to computer knowledge bases, and more particularly to a knowledge base version management method and system based on a unified distributed index. Background Technology

[0002] With the widespread application of artificial intelligence and Large Language Models (LLM), the importance of knowledge bases, as their core external memory units, is becoming increasingly prominent. Enterprises and developers need to continuously iterate, update, and revise the content within these knowledge bases. In this process, version management has become a critical and complex issue.

[0003] Existing knowledge base version management solutions suffer from the following pain points: Low query efficiency: Many solutions rely on traditional relational databases to store version metadata and even document content. As the number of versions increases, cross-version queries or historical version rollbacks become necessary. Complex table joins or data redundancy lead to a sharp decline in query performance, failing to meet the low-latency requirements of real-time question-and-answer systems. Furthermore, version rollbacks require row-by-row UPDATE / DELETE operations, resulting in heavy transactions and potential table locking. High storage costs: Some rudimentary version management methods, such as creating independent index snapshots or full database backups for each version, result in significant storage waste. Simultaneously, the linear growth of the total number of cluster shards puts pressure on the Master node, making it a bottleneck. Complex version release and rollback: Traditional release processes may involve heavy operations such as data migration and index rebuilding. These processes are complex and error-prone, sometimes requiring service downtime and impacting business continuity. During version rollbacks, improper operation can easily lead to data confusion or loss of related data, compromising the integrity of the knowledge base. Weak isolation between development and production environments: Draft content during the development phase, such as newly added or modified slices, is not strictly isolated from the stable online version. This can easily lead to the draft content being called by the online API due to accidental operations, resulting in business data errors.

[0004] Therefore, there is an urgent need for a knowledge base version management solution that can overcome the above-mentioned shortcomings, provide efficient querying, low storage consumption, convenient operation, and support secure release. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a knowledge base version management method based on a unified distributed index, which solves the problems existing in the prior art, reduces storage requirements, and improves retrieval efficiency; the second purpose of this invention is to provide a knowledge base version management system based on a unified distributed index.

[0006] Technical solution: The knowledge base version management method based on a unified distributed index described in this invention includes:

[0007] A unified distributed index is constructed for all version data of the knowledge base. Each document in the distributed index contains knowledge slice content, the corresponding vector, and metadata fields for version differentiation. A mapping relationship between semantic version numbers and monotonically increasing integer version IDs is maintained at the application layer. New version preparation data is created, and the modified content of the new version is categorized, with ID modification operations performed based on the categorization. The new version is published. Based on the modification and deletion intentions recorded in the previous step, the invalid version ID field of the affected historical valid documents in the unified index is updated to the integer ID of the new version, and the status of all documents associated with the new version and in draft state is updated to published. Expired or invalid documents are also cleaned up.

[0008] Preferably, the metadata fields include: a knowledge slice logical unique identifier field, an effective version ID field, an invalid version ID field, and a status field.

[0009] Preferably, the distributed index responds to a query request for a specific historical version by initiating a query and attaching filtering conditions to obtain a set of valid documents for the corresponding version. The filtering conditions are: the status is published, the effective version ID is less than or equal to the historical version, and the invalid version ID is greater than the historical version or is empty.

[0010] Preferably, in response to a preview request for a future release version, the distributed index obtains a set of preview documents by initiating a combined query to the unified index. The combined query includes: querying all documents whose effective version ID is the release version and whose status is draft; and querying documents in the current release version that are valid and whose logically unique identifiers do not appear in the unreleased version draft change set.

[0011] Preferably, when the distributed index performs a version rollback operation, it switches the version ID parameter of the external query service to the ID of the target historical version.

[0012] Preferably, the cleanup operation includes permanently deleting documents in the index whose expired version IDs are earlier than a specified threshold, based on a preset threshold.

[0013] Preferably, the index field design supports mixed retrieval, which combines similarity retrieval for the vector field, keyword retrieval for the knowledge slice content field, and version filtering conditions based on the metadata field during querying.

[0014] Preferably, the ID modification operation based on classification includes: for new content in the new version, inserting a new document in the unified index with a draft status, and setting its effective version ID to the integer ID of the new version; for modified content, inserting a new document in the unified index with the same logically unique identifier as the old slice but different content, and with an effective version ID of the integer ID of the new version and a draft status; for deleted content, recording the logically unique identifier of the knowledge slice to be deleted at the application layer.

[0015] Preferably, for the modified content, the old document remains unchanged in the index.

[0016] The knowledge base version management system based on a unified distributed index as described in this invention includes:

[0017] The index management module is used to configure and maintain the mapping and field design of the unified index;

[0018] The version mapping module is used to maintain and manage the mapping relationship between semantic version numbers and integer version IDs at the application layer;

[0019] The version editing module is used to handle the addition, modification and deletion intentions during the version creation process, and to insert draft documents into the unified index or record changes at the application layer;

[0020] The atomic release module is used to perform atomic operations for version releases, including batch logically invalidating old documents and activating draft documents for new versions.

[0021] The version query module is used to parse the version information in the query request and construct corresponding filtering conditions to retrieve a set of documents of a specified version from the unified index.

[0022] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. By incrementally recording changes and reusing unchanged data, data redundancy is effectively avoided and storage costs are reduced; 2. System data updates do not require downtime maintenance, improving business continuity; 3. Support for version consistency of vector retrieval, keyword retrieval, and mixed retrieval reduces the complexity of index calls and improves query efficiency. Attached Figure Description

[0023] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described below.

[0025] This invention provides a knowledge base version management method based on distributed storage. Its core idea is that all historical versions and current version data records of a single knowledge base are stored in a unified, logically isolated distributed data structure. By attaching a set of metadata to each data record, this invention enables accurate data snapshot retrieval of any historical version without the need for physical data copying.

[0026] The specific technical solutions include the following aspects:

[0027] Firstly, unified index design.

[0028] This invention employs a "one knowledge base, one index" model to establish multiple mappings, allowing all historical versions and the current version of a knowledge base to coexist in a unified index. Version differentiation is achieved through custom fields. The index mapping design includes the following core data fields:

[0029] `chunk_id`: A logically unique identifier for a knowledge slice. This ID remains unchanged throughout the slice's lifecycle and is the core of cross-version association.

[0030] content: The text content of the slice.

[0031] Embedding: The vector representation of text content.

[0032] `start_version_id`: Effective version identifier. A monotonically increasing integer used to mark the starting point of the lifecycle of this data record. It precisely defines from which version this knowledge slice becomes a valid part of the knowledge base.

[0033] `end_version_id`: Expired version identifier. A monotonically increasing integer that is set to the ID of the new version when the content of a record is replaced (updated) or obsolete (deleted). If this field is null, it means that the document remains valid in the current and future versions. This design constitutes the core mechanism of the "soft deletion" of this invention, ensuring that no historical data is physically erased, providing integrity guarantees for data auditing and version backtracking.

[0034] `status`: A document status flag (e.g., `draft` or `published`) used to distinguish between draft data awaiting publication and stable, published data. Documents in the `published` status indicate that the record is a stable, verified version, and is official data provided for retrieval services to end users. Documents in the `draft` status contain additions or modifications prepared for future versions, and may be under review or development, not visible to external users. This status design achieves secure isolation between the development and production environments within a single data structure, ensuring a smooth and atomic version release process.

[0035] Secondly, the core of this invention is an innovative version management mechanism based on metadata of the data record lifecycle.

[0036] The version management of this invention maintains version mapping relationships at the application layer, associating semantic version numbers (such as v1.0) with integer IDs, and supporting efficient range comparisons.

[0037] The procedural process for version creation and content modification is as follows:

[0038] Assuming the current online version is vN, and its corresponding numerical ID is N, when the system begins preparing to create the next version vN+1 (corresponding ID N+1), all changes are initially in a "draft" state. The specific process is as follows:

[0039] New features: When one or more new documents are inserted into the knowledge base, the system will create one or more new data records. The effective version identifier start_version_id of these records will be set to N+1, and the document status will be marked as draft.

[0040] Modifications: This invention adheres to the core principle that published data cannot be altered. Modifying an existing knowledge slice does not directly overwrite the old record, but rather creates a completely new data record with the same chunk_id as the old slice. The effective version start_version_id is set to N+1, and the slice status is set to "draft". During this process, the original version N record remains unchanged and is unaffected.

[0041] Content deletion: Instead of directly deleting documents from distributed storage, the application layer records a chunk_id identifier that will be deleted in version vN+1.

[0042] Version release procedure:

[0043] When version vN+1 is ready, the system performs a two-phase release operation that ensures transactional consistency:

[0044] (1) "Retiring" old documents: Based on the change list recorded at the application layer (including modified and deleted chunk_ids), the system initiates a conditional batch update operation to the distributed storage system. The query condition for this operation is: to filter out all records whose chunk_id is in the change list and is still valid (i.e., the expired version identifier end_version_id is empty) and published (status is published). The operation action is: to uniformly set the value of the expired version identifier end_version_id field of these records to N+1.

[0045] (2) "Activate" new document: The system initiates a second conditional batch update operation. The query condition for this operation is: to filter out all records with the effective version identifier start_version_id of N+1 and the status of "draft". The operation action is: to update the status field of these records to published.

[0046] Once these two phases of the process are completed, version vN+1 will be officially released, and the entire process will not interrupt external query services in any way.

[0047] Thirdly, through the above-mentioned mechanism, the present invention achieves the following key functions:

[0048] 1. Query any historical version: Retrieving data from any historical version M (M is the version numerical ID) is achieved by constructing a precise set of filtering conditions. A data record is considered to belong to version M if and only if:

[0049] (1) The status must be published.

[0050] (2) start_version_id must be less than or equal to M.

[0051] (3) end_version_id must be greater than M or the end_version_id field does not exist.

[0052] This query logic can accurately filter out the collection of valid, published documents at the vM point in time.

[0053] 2. Future Version Preview: This invention provides the ability to preview future version content without affecting online services. This function is implemented through a composite query logic consisting of two parallel branches:

[0054] (1) Query all documents with start_version_id of N+1 and slice status of "draft";

[0055] (2) Retrieve all data records that are valid in the current version vN, but whose chunk_id does not appear in the application layer change set of version vN+1;

[0056] By combining the results of these two branches, you can obtain a complete view of future versions that includes all changes.

[0057] 3. Version Rollback: The version rollback of this invention is extremely efficient. Since all historical versions of documents are retained in the index through "soft deletion," rolling back to any old version vM only requires switching the version ID parameter in the query logic from the current version to the vM during API calls. This is a completely lossless, instantaneous "logical rollback" that requires no data writing or migration operations.

[0058] 4. Version Cleanup: To control index size, strategies can be set to automatically clean up outdated versions. For example, data records whose invalid version identifier end_version_id is less than a preset threshold (e.g., the current latest version ID minus a constant K) can be physically removed, thereby safely reclaiming storage resources while preserving the integrity of recent versions.

[0059] Fourthly, the effectiveness of this invention benefits from its deep coupling with the core characteristics of modern distributed storage systems:

[0060] 1. In this invention, version control metadata (such as effective / invalid version identifiers and status fields) is designed as indexable data types. The distributed storage system automatically constructs efficient index structures (such as inverted indexes) for it. Therefore, complex version filtering logic can be quickly completed through underlying set operations (such as bitmap operations), avoiding inefficient scanning of the entire dataset.

[0061] 2. Implement distributed storage using a sharding mechanism: As the number of knowledge base versions increases and the amount of data grows, the index shards can be expanded by adding computing nodes, thereby achieving a linear improvement in storage capacity and query performance.

[0062] 3. Support for hybrid retrieval using the dense_vector type: The embedding field uses the dense_vector type, facilitating vector similarity calculation; simultaneously, the content field supports keyword matching, allowing queries to simultaneously input vector retrieval, keyword retrieval, and version filtering conditions, achieving integrated processing of "hybrid retrieval + version isolation." Support for integrated processing of hybrid retrieval and version isolation: Query requests can simultaneously include: semantic similarity retrieval conditions based on vector representation, keyword matching conditions based on sliced ​​text, and version filtering conditions defined in this invention. The distributed storage system can integrate these three conditions in a single query, efficiently returning accurate results within the specified version that simultaneously meet semantic and keyword requirements.

[0063] The implementation process of the present invention will be further described below with reference to another specific embodiment.

[0064] 1. The system initialization workflow is as follows:

[0065] The version mapping table maintained by the application layer is shown below, used to record the version status of the knowledge base:

[0066] Serial Number Domain English name Domain Chinese name Data types illustrate 1 id Unique Identifier VARCHAR(64) 2 version_id Internal numerical version number INT Monotonically increasing (1, 2, 3...) 3 version_name Semantic version number VARCHAR(20) Semantic version numbers (e.g., v1.0, v2.0) 4 dataset_id Knowledge Base ID VARCHAR(64) 5 published_at Version release time DATETIME 6 is_published Has it been published? TINYINT(1) 0: Not published, 1: Published 7 api_token API call token corresponding to the version VARCHAR(64)

[0067] The application layer also maintains a "change log" to track modifications and deletions that occur in the draft version:

[0068] Serial Number Domain English name Domain Chinese name Data types illustrate 1 dataset_id Knowledge Base ID VARCHAR(64) 2 version_id Internal version number INT 3 chunk_id Slice ID VARCHAR(64) 4 action Execute action VARCHAR(64) Only delete and modify actions are recorded. 5 action_time Action execution time DATETIME

[0069] 2. Creation and release of the initial version (version ID=1):

[0070] During system initialization, insert a record of an unreleased version belonging to the knowledge base dataset with an internal version ID of 1 into the application layer version mapping table;

[0071] Create an index corresponding to the knowledge base in the distributed storage and define the mapping as described in the technical solution.

[0072] The initial data record is injected into the storage instance, its "effective version identifier" is set to 1, and its "slice status" is set to "draft";

[0073] Upon release, the record status of the corresponding knowledge base in the application layer version mapping table is updated to "released", and an unreleased record with ID 2 is pre-created. Subsequently, a conditional batch update operation is performed to update the status of all records in the storage instance with "effective version identifier" of 1 and status of "draft" to "released".

[0074] 3. Preparing the draft stage for the next version (version ID=2):

[0075] (1) Modification operation (e.g., modify record B)

[0076] Insert a new record into the storage instance, with version 2 and status "draft". Also, record this change in the change log.

[0077] (2) Deletion operation (e.g., delete record A)

[0078] Records that do not operate on the storage system but only record "Logical Identifier" as "A" in the change log will be discarded in version 2.

[0079] (3) Adding new operations (e.g., adding record D)

[0080] Insert a new record into the storage instance with a "Logical Identifier" of "D", an effective version of 2, and a status of "Draft".

[0081] 4. Data retrieval operation example:

[0082] (1) Query the released version 1.0:

[0083] The version filtering criteria included in the data retrieval request are: slice status is published, effective version identifier is less than or equal to 1, and invalid version identifier is either nonexistent or greater than 1. The system will return record A, record B before modification, and record C precisely. All records with a status of "draft" (such as new version B and new version D) will be automatically filtered.

[0084] (2) Preview of future version 2.0:

[0085] The data retrieval request triggers a composite query logic, whose filtering conditions will combine "all records that are effective in version 2 and are in draft status" with "all records that are valid in version 1 and whose logical identifiers do not appear in the version 2 change log".

[0086] 5. Deployment, Verification, and Rollback:

[0087] Step 5.1 Release version v2.0: Perform the two-stage release procedure described in step 2 above.

[0088] Step 5.2 Verify v2.0 related content: Use the same query logic as in step 4, but version_id is 2.

[0089] Step 5.3 Rollback to version v1.0: If a problem is found in v2.0, the service caller only needs to change the version number in the request parameters back to v1.0, and the system will immediately start returning the content of v1.0 without any server-side data operation.

[0090] 6. Example of version cleanup operation:

[0091] Cleanup strategy: Assume the system is configured to retain the 5 most recent released versions;

[0092] Execution condition: When the latest released version is 11.0, it means that versions 1.0 to 6.0 are out of the retention range.

[0093] Operation performed: The system initiates a conditional batch deletion procedure to physically delete all data records with an "invalid version identifier" less than or equal to 6. This operation will safely remove obsolete records that only existed in older versions.

Claims

1. A knowledge base version management method based on a unified distributed index, characterized in that, Specifically, the following steps are included: (1) Construct a unified distributed index for all version data of the knowledge base. Each document in the distributed index contains knowledge slice content, the vector corresponding to the content, and metadata fields for version differentiation. (2) Maintain the version mapping of the index, and maintain the mapping relationship between semantic version number and monotonically increasing integer version ID at the application layer; (3) Prepare data for creating a new version, classify the changes in the new version, and perform ID modification operations according to the classification; (4) Publish a new version. Based on the modification and deletion intentions recorded in the previous step, in the unified index, update the invalid version ID field of the affected historical valid documents to the integer ID of the new version, and update the status of all documents associated with the new version and in draft status to published. At the same time, clean up expired or invalid documents.

2. The knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, The metadata fields include: a knowledge slice logical unique identifier field, an effective version ID field, an invalid version ID field, and a status field.

3. The knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, The distributed index responds to a query request for a specific historical version by initiating a query and attaching filtering conditions to obtain a set of valid documents for the corresponding version. The filtering conditions are: the status is published, the effective version ID is less than or equal to the historical version, and the invalid version ID is greater than the historical version or is empty.

4. The knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, In response to a preview request for a future release version, the distributed index retrieves a set of preview documents by initiating a combined query to the unified index. The combined query includes: querying all documents whose effective version ID is the release version and whose status is draft; and querying documents in the currently released version that are valid and whose logically unique identifiers do not appear in the unreleased version draft change set.

5. A knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, When the distributed index performs a version rollback operation, it switches the version ID parameter of the external query service to the ID of the target historical version.

6. A knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, The cleanup operation includes permanently deleting documents in the index whose expired version IDs are earlier than a specified threshold, based on a preset threshold.

7. A knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, The index's field design supports hybrid retrieval, allowing queries to simultaneously combine similarity retrieval for the vector field, keyword retrieval for the knowledge slice content field, and version filtering conditions based on the metadata field.

8. A knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, The ID modification operation based on classification includes: for new content in the new version, inserting a new document in the unified index with a draft status, and setting its effective version ID to the integer ID of the new version; for modified content, inserting a new document in the unified index with the same logically unique identifier as the old slice but different content, and with an effective version ID of the integer ID of the new version and a draft status; for deleted content, recording the logically unique identifier of the knowledge slice to be deleted at the application layer.

9. A knowledge base version management method based on a unified distributed index according to claim 1, characterized in that, For the aforementioned modifications, the old documents remain unchanged in the index.

10. A knowledge base version management system based on a unified distributed index, characterized in that, The system includes: The index management module is used to configure and maintain the mapping and field design of the unified index; The version mapping module is used to maintain and manage the mapping relationship between semantic version numbers and integer version IDs at the application layer; The version editing module is used to handle the addition, modification and deletion intentions during the version creation process, and to insert draft documents into the unified index or record changes at the application layer; The atomic release module is used to perform atomic operations for version releases, including batch logically invalidating old documents and activating draft documents for new versions. The version query module is used to parse the version information in the query request and construct corresponding filtering conditions to retrieve a set of documents of a specified version from the unified index.

Images