Method and system for automatic updating and retrieval of knowledge base

Abstract

The application relates to the technical field of knowledge base retrieval, and discloses a knowledge base automatic updating and retrieval method and system, which comprises the following steps: dividing a document set into semantic atoms, storing a benchmark vector into a database and initializing a differential log queue; generating a text differential patch based on a modification request and writing the patch into a queue, extracting change characteristic words to update an inverted index; performing two-way retrieval of the benchmark vector and the inverted index based on a query request to generate a candidate set; applying the patch to the candidate set to build a real-time text view; calculating the text matching degree of the query and the view, combining a benchmark vector similarity to calculate a drift penalty term, and reordering and outputting the atoms according to a comprehensive ranking score containing the drift penalty term. Through the differential log and two-way retrieval mechanism, the application realizes the decoupling of the static stability of the vector index and the dynamic timeliness of the text, avoids the high-frequency recalculation of the vector, and ensures the real-time accuracy of the retrieval result through semantic drift calibration.

Description

Technical Field

[0001] This invention relates to the field of knowledge base retrieval technology, specifically to a method and system for automatic knowledge base updating and retrieval. Background Technology

[0002] With the increasing prevalence of large language models and retrieval-enhanced generative applications, vector databases have become a core component for storing and retrieving massive amounts of unstructured text data. Conventional knowledge base maintenance processes typically involve segmenting the original documents and transforming them into high-dimensional dense vectors using deep neural network models, then constructing index structures such as hierarchical navigation small-world graphs or inverted indexes to accelerate approximate nearest neighbor searches.

[0003] However, this static vector index-based architecture reveals its limitations in production environments where document content changes frequently. Since both the inference computation of deep learning models and the construction of vector index structures are computationally intensive tasks, when semantic fragments in the knowledge base are modified, inserted, or deleted, the system typically needs to recalculate the associated vectors, triggering partial or full reconstruction of the index file. In scenarios with frequent data updates, this synchronous update mechanism consumes significant computing resources and triggers read-write lock contention in the database, severely impacting the system's concurrent processing capabilities and response speed.

[0004] To alleviate computational pressure, existing technologies often employ a strategy of periodic batch updates, accumulating a certain amount of changes before uniformly recalculating vectors and refreshing the index. While this delayed update method reduces instantaneous load, it leads to inconsistencies between the retrieved view and the actual data state, preventing users from retrieving the latest business information in a timely manner and resulting in a lack of timeliness in knowledge. Furthermore, within the time window where data content has changed but vectors have not yet been updated, the system still performs semantic matching and ranking based on the baseline vectors corresponding to the old version of the text. When changes in text content cause semantic shifts, the old baseline vectors can no longer accurately represent the current text meaning, leading the retrieval system to incorrectly recall low-relevance or even irrelevant document fragments, or to rank highly relevant documents that have become more relevant due to content updates at the bottom, severely reducing the accuracy and reliability of search results. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for automatic updating and retrieval of knowledge bases. It solves the problems of high computational overhead for vector recalculation and index reconstruction in high-frequency update scenarios, time lag in data updates, and reduced retrieval accuracy due to inconsistencies between the baseline vector and real-time semantics caused by changes in text content.

[0006] To achieve the above objectives, the first aspect of the present invention provides a method for automatic updating and retrieval of a knowledge base, comprising the following steps:

[0007] The original document set is divided into several semantic atoms, the baseline vector of each semantic atom is calculated and stored in the vector database, and a differential log queue is initialized for each semantic atom.

[0008] Receive a content modification request for the semantic atom, generate a text difference patch based on the content modification request and write it to the difference log queue, and simultaneously extract the changed feature words in the text difference patch to update the inverted index;

[0009] Receive a query request input by the user, perform similarity retrieval based on the baseline vector and inverted index retrieval based on the changed feature words based on the query request, and merge the retrieval results to form a candidate set;

[0010] For the semantic atoms in the candidate set, read the baseline text, apply the text difference patch in the differential log queue, and construct a real-time text view;

[0011] Calculate the text matching degree between the query request and the real-time text view, calculate the drift penalty term based on the similarity retrieval result of the baseline vector, and reorder the candidate semantic atoms according to the comprehensive ranking score including the drift penalty term and output them.

[0012] Preferably, the step of generating a text differential patch based on the content modification request and writing it to the differential log queue specifically includes:

[0013] Read the reference text of the semantic atom, compare the differences between the modified text content and the reference text, and generate the text difference patch that records the position and length of the deleted characters and the content of the added characters;

[0014] Construct an operation log entry that includes the operation type, the text difference patch, and a timestamp;

[0015] The operation log entries are appended to a key-value storage system independent of the vector database storage, as new records in the differential log queue, without triggering an update operation on the base vector.

[0016] Preferably, when the text content of the semantic atom changes, the value and physical address of the base vector stored in the vector database remain unchanged;

[0017] Monitor the length of the differential log queue. When the number of operation log entries reaches a preset threshold, perform a snapshot fusion operation in the background to generate a text snapshot and update the snapshot pointer.

[0018] The step of reading the reference text of the semantic atom specifically involves: reading the text snapshot pointed to by the snapshot pointer as the reference text.

[0019] Preferably, the step of merging the search results to form a candidate set specifically includes:

[0020] Based on the query request, a query vector is generated, and semantic atoms that meet the preset conditions for similarity with the benchmark vector are retrieved from the vector database to generate a benchmark candidate set.

[0021] Based on the query keywords extracted from the query request, the query keywords are matched with the change feature words stored in the inverted index, the associated semantic atoms are retrieved, a transient candidate set is generated, and the anchor recall attribute is marked.

[0022] Perform a union operation on the baseline candidate set and the transient candidate set. For each semantic atom contained in the union operation result, check the life cycle state of each semantic atom according to the global state bitmap, remove semantic atom identifiers whose state bits show as deleted or obsolete, and obtain the candidate set.

[0023] Preferably, the step of constructing the real-time text view further includes reference resolution and recursive processing, specifically including:

[0024] The text difference patch is applied to the base text to generate intermediate text, and the intermediate text is parsed to identify reference markers in the intermediate text that point to other semantic atoms;

[0025] In response to the identification of the reference identifier, a recursive call request for the referenced semantic atom is generated, and a recursive synthesis context object containing the current set of recursive paths and a recursion depth counter is created;

[0026] Before executing the recursive call request, it is determined whether the current recursion depth exceeds a preset threshold and whether the referenced semantic atom already exists in the current recursion path set based on the recursive synthesis context object;

[0027] If the threshold is not exceeded and there is no circular reference, the recursive call request is executed to obtain the real-time view content of the referenced semantic atom, and the reference identifier in the intermediate text is replaced with the real-time view content.

[0028] Preferably, the step of calculating the drift penalty term specifically includes:

[0029] The similarity scores obtained from the similarity retrieval results of the baseline vector and the text matching degree between the query request and the real-time text view are normalized respectively.

[0030] The absolute value of the difference between the normalized similarity score and the text matching degree is calculated as the drift degree.

[0031] The dynamic confidence weight is calculated based on the drift degree and used as the drift penalty term. When the drift degree exceeds the preset drift tolerance threshold, the dynamic confidence weight decays non-linearly.

[0032] Preferably, the step of reordering the candidate semantic atoms based on the comprehensive ranking score including the drift penalty term specifically includes:

[0033] Determine whether the semantic atom has the anchor recall attribute. If it has the anchor recall attribute, set the indicator function value to a valid value; otherwise, set it to zero.

[0034] The normalized similarity score is multiplied by the dynamic confidence weight, and then weighted and summed with the normalized text matching degree and the product of the indicator function value and the preset reward bias value to obtain the comprehensive ranking score.

[0035] The semantic atoms in the candidate set are sorted in descending order based on the comprehensive ranking score, and the top-ranked real-time text views are extracted as the search results.

[0036] A second aspect of the present invention provides a knowledge base automatic update and retrieval system, comprising:

[0037] The semantic storage module (100) is used to divide the original document set into several semantic atoms, calculate the base vector of each semantic atom and store it in the vector database, and initialize a differential log queue for each semantic atom.

[0038] The state management module (200) is used to receive content modification requests for the semantic atom, generate text difference patches based on the content modification requests and write them to the difference log queue, and at the same time extract the changed feature words in the text difference patches to update the inverted index.

[0039] The dual-path retrieval module (300) is used to receive a query request input by the user, perform similarity retrieval based on the reference vector and inverted index retrieval based on the changed feature words based on the query request, and merge the retrieval results to form a candidate set;

[0040] The view synthesis module (400) is used to read the baseline text for the semantic atoms in the candidate set, apply the text difference patch in the differential log queue to construct a real-time text view, calculate the text matching degree between the query request and the real-time text view, calculate the drift penalty term in combination with the similarity retrieval result of the baseline vector, and output the retrieval result based on the comprehensive ranking score.

[0041] Preferably, the semantic storage module (100) uses a regular expression-based paragraph recognition, a sliding window-based truncation, or a DOM tree-based node parsing strategy to segment the original document set into semantic atoms, and uses a hash algorithm or a distributed unique ID generator to generate a unique identifier for each semantic atom.

[0042] Preferably, the view synthesis module (400) uses an algorithm based on a probability retrieval model to calculate the text matching degree. The calculation parameters of the algorithm include the word frequency, inverse document frequency and average text length of all semantic atoms in the real-time text view of the query keyword.

[0043] This invention provides a method and system for automatic updating and retrieval of a knowledge base. It has the following beneficial effects:

[0044] 1. This invention constructs a differential log queue independent of the vector database and locks the value and storage address of the baseline vector when content changes, avoiding the computational overhead of frequent deep learning model inference and vector index reconstruction in high-frequency update scenarios. By decoupling the static index storage from the dynamic change log architecture, the system can handle high-concurrency text modification requests with extremely low resource consumption, reducing the latency of knowledge base content updates.

[0045] 2. This invention combines semantic retrieval based on baseline vectors with inverted index retrieval based on changed feature words, and employs runtime dynamic view synthesis technology to solve the problem that static vector indexes cannot reflect the latest changes in text in real time. This ensures that user queries can retrieve basic content through semantic similarity and newly added content whose vectors have not yet been updated through keyword matching, effectively guaranteeing the recall and timeliness of search results.

[0046] 3. This invention introduces a dynamic scoring and re-ranking mechanism based on semantic drift. By calculating the text matching degree between the query request and the real-time synthesized view, the similarity score of the baseline vector is corrected. It can automatically identify and reduce the ranking weight of semantic atoms that, although similar to the baseline vector, have deviated in actual content. Thus, without updating the vector, it eliminates the retrieval bias caused by inconsistent data versions and improves the accuracy of the final output results. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall architecture of the knowledge base dynamic view synthesis system of the present invention;

[0048] Figure 2 This is a schematic diagram of the overall process of the knowledge base dynamic view synthesis method of the present invention;

[0049] Figure 3 This is a schematic diagram of the knowledge base initialization and atomization locking process of the present invention;

[0050] Figure 4 This is a schematic diagram of the non-intrusive update process based on differential logs of the present invention;

[0051] Figure 5 This is a schematic diagram of the transient dual-path collaborative retrieval process of the present invention;

[0052] Figure 6 This is a schematic diagram of the dynamic view synthesis process with loop detection according to the present invention;

[0053] Figure 7 This is a schematic diagram of the semantic consistency calibration and output process of the present invention.

[0054] Among them, 100 is the semantic storage module; 200 is the state management module; 300 is the dual-path retrieval module; and 400 is the view synthesis module. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] See attached document Figure 1 This invention provides an automatic knowledge base update and retrieval system, which runs on an electronic computing device. The electronic computing device includes at least one processor, a memory, a communication interface, and a bus connecting the various components. The memory stores computer-executable instructions, which, when executed by the processor, implement the method provided by this invention.

[0057] like Figure 1 As shown, the system mainly includes a semantic storage module 100, a state management module 200, a dual-path retrieval module 300, and a view synthesis module 400. The modules interact with each other via a system bus or internal communication protocol.

[0058] The semantic storage module 100 is configured to handle the access and initialization of raw document data. This module is responsible for segmenting the unstructured document collection into the smallest management unit with independent semantics, namely semantic atoms. The semantic storage module 100 calls a pre-trained model to perform vectorization calculation on the base text of the semantic atoms, generating base vectors. The base vectors are associated with the unique identifiers of the semantic atoms and stored in the vector database. During the system's runtime, the semantic storage module 100 implements a write-locking strategy for the stored base vectors, that is, when the text content of the semantic atom changes, the value of the base vector remains unchanged, and no recalculation or overwriting operation is performed.

[0059] The state management module 200 is configured to handle update requests for semantic atoms and maintain the dynamic state of the atoms. This module uses a log-structured storage method, receives update instructions containing new text content, calculates the difference between the new text and the current base text or snapshot text, and generates a text difference patch. The text difference patch, along with the operation type and timestamp, constitutes an operation log entry, which is appended sequentially to the difference log queue corresponding to the semantic atom identifier. The state management module 200 monitors the length of the difference log queue. When the number of log entries reaches a preset threshold, it triggers a background merge task, generates a new text snapshot, and clears the merged log entries.

[0060] The dual-path retrieval module 300 is configured to respond to user query requests by performing parallel dual-path retrieval operations to generate a candidate semantic atom set. This module includes a main-path retrieval unit and an auxiliary-path retrieval unit. The main-path retrieval unit, based on the query vector generated from the query request, retrieves semantic atoms in the semantic storage module 100 that meet preset similarity conditions with the baseline vector. The auxiliary-path retrieval unit, based on the keyword features extracted from the query request, retrieves semantic atoms containing corresponding semantic anchors in the inverted index. The dual-path retrieval module 300 performs a union operation on the two retrieval results and, based on the state bitmap maintained by the state management module 200, removes semantic atoms marked for deletion, outputting an initial candidate set.

[0061] The view synthesis module 400 is configured to perform runtime content reconstruction and sorting optimization on semantic atoms in the initial candidate set. This module reads the baseline text or the latest text snapshot of the semantic atoms and obtains the associated differential log queue from the state management module 200. The view synthesis module 400 sequentially applies text differential patches in the memory buffer to synthesize the current valid text view. During the synthesis process, the module parses reference pointers in the text, recursively calls the synthesis logic to obtain the content of the referenced atoms, and performs recursive depth detection to prevent circular references. The module further calculates the real-time relevance score between the query request and the synthesized valid text view, combines the similarity score of the baseline vector with a preset drift penalty factor, and calculates the final ranking score. The system reorders the candidate semantic atoms according to the final ranking score and outputs the sorted synthesized text view as the retrieval result.

[0062] See attached document Figure 2 This invention provides a method for automatic updating and retrieval of a knowledge base, comprising the following steps:

[0063] S1, Perform knowledge base initialization. The system receives the original document set and divides it into several semantic atoms. For each semantic atom, the system calculates the base vector of its text content and writes the base vector into the vector database. At this time, the system initializes an empty differential log queue and a snapshot pointer to the base text for each semantic atom;

[0064] S2, performs non-intrusive updates. When a content modification request for a specific semantic atom is received, the system reads the current text state of that atom, compares it with the modified content, and generates a differential patch. The system encapsulates the differential patch as a log entry and writes it to the differential log queue. Simultaneously, the system extracts feature words from the differential patch, updates the inverted index, and establishes a mapping relationship between feature words and semantic atom identifiers. If the length of the differential log queue exceeds a preset threshold, the system performs a snapshot fusion operation in the background, generating a new version of the snapshot text.

[0065] S3, Perform dual-path collaborative retrieval. In response to the received query request, the system performs parallel similarity retrieval based on the baseline vector and inverted index retrieval based on feature words. The system merges the semantic atom identifiers returned by the two retrievals to form a candidate set. For each semantic atom in the candidate set, the system checks its state bitmap and removes atoms that have been marked as deleted.

[0066] S4, Perform dynamic view composition. For each semantic atom in the candidate set, the system loads its latest snapshot text and sequentially replays the differential log entries after the snapshot generation time, constructing its real-time text view in memory. If the real-time text view contains references to other semantic atoms, the system recursively executes the loading and replay steps, embedding the referenced content into the current view;

[0067] S5, Perform consistency calibration and output. The system calculates the text matching degree between the query request and the real-time text view. If a semantic atom is recalled only through benchmark vector retrieval, and the benchmark vector similarity is high but the text matching degree of the real-time text view is lower than a preset threshold, the system generates a drift penalty term to reduce the ranking weight of that semantic atom. The system sorts all candidate semantic atoms according to the weighted total score and outputs the top-ranked real-time text views.

[0068] See attached document Figure 3 Step S1 illustrates the specific execution flow of knowledge base initialization and atomic locking. As the foundational stage of the dynamic view synthesis method for the knowledge base provided by this invention, this step aims to construct a static baseline layer based on immutable vector indexes, thereby providing data reference for subsequent low-cost dynamic updates. Step S1 specifically includes the following sub-steps:

[0069] In sub-step S101, the segmentation and structured definition of semantic atoms are performed. The system receives the original unstructured document collection and uses a preset granularity control strategy to segment the document stream into discrete sets of semantic atoms. Specific implementations of the granularity control strategy include, but are not limited to: paragraph recognition based on regular expressions, fixed-length truncation based on sliding windows, or node parsing based on a DOM tree structure. For each semantic atom obtained from the segmentation, the system uses a hash algorithm (such as MD5, SHA-256) or a distributed unique ID generator (such as the Snowflake algorithm) to generate a globally unique identifier for it. The system defines the initial text content of the semantic atom as the base text and constructs the data structure as tuples stored in a persistent database, which contains a metadata set including creation time, source path, and initial state markers.

[0070] In sub-step S102, the baseline vector is calculated. The system calls a pre-defined deep semantic coding model to perform forward reasoning on the baseline text for each semantic atom, generating a high-dimensional dense vector, i.e., the baseline vector. This calculation process satisfies the formula: ;

[0071] Here, the transformation function represents the mapping of discrete text sequences to a continuous vector space, and the dimension of the output vector is consistent with the dimension configured in the vector database index.

[0072] For the specific architecture and training methods of deep semantic coding models, those skilled in the art can use existing mature technical solutions, which will not be elaborated here.

[0073] In sub-step S103, vector storage and write locking are performed. The system establishes an index association between the generated base vector and the identifier, and stores it in the vector retrieval engine (such as an HNSW index or an IVF index). During this process, the system implements a logical preservation mechanism. Specifically, when the system subsequently receives a content update request for a semantic atom, the system is configured to skip the update or deletion operation for that base vector, keeping the physical address and value of the vector in the index constant. This mechanism avoids the overhead of inverted chain merging or HNSW graph structure reconstruction caused by high-frequency updates by maintaining the stability of the index structure.

[0074] In sub-step S104, the dynamic state container is initialized. For each semantic atom, the system initializes its corresponding dynamic state structure in the key-value storage system. This operation specifically includes: creating and initializing an empty differential log queue for subsequent storage of time-series update patches, with the initial state satisfied; creating a log length counter and setting its value to zero; creating a snapshot pointer, with its key-value reference pointing to the storage address of the base text in the persistent database. After completing the above initialization, the semantic atom has the structural basis to accept differential data writes.

[0075] See attached document Figure 4 Step S2 illustrates the specific execution flow of the non-intrusive update based on differential logs. This step aims to achieve dynamic maintenance of the knowledge base content by decoupling text update operations from vector maintenance operations at the physical storage level. Step S2 specifically includes the following sub-steps:

[0076] In substep S201, text difference calculation and patch generation are performed. When the system receives an update request for a specified semantic atom, and the request contains new text content, the system reads the current reference text for that semantic atom. The reference text is the latest text snapshot pointed to by the snapshot pointer; if no snapshot exists, the baseline text is loaded. The system uses a difference comparison algorithm based on the longest common subsequence (such as the implementation algorithm of the Google-Diff-Match-Patch library) to compare and generate text difference patches. This process can be represented as:

[0077] Here, is the difference calculation function, whose output is encoded as a string or JSON object containing the difference operation sequence, explicitly recording the position and length of the deleted characters and the content of the newly added characters.

[0078] In sub-step S202, a structured operation log is constructed and appended. The system uses text differential patches to construct operation log entries. These log entries are defined as data structures containing the following fields: an operation type field (Action, with enumerated values ​​of UPDATE, APPEND, or DELETE), a patch content field to store the text differential patches, and a timestamp field indicating the operation was generated. The system writes to the differential log queue associated with the semantic atom. The write operation uses a key-value list append command (such as Redis's RPUSH command), which is independent of the vector database storing the base vector. Therefore, this write process does not trigger index rebuilding or file locks in the vector database, achieving zero interference with the vector retrieval service.

[0079] In sub-step S203, change feature words are extracted and the inverted index is updated. For text segments marked as new in the text difference patch, the system performs word segmentation and stop word filtering to extract the set of change feature words. For each change feature word in the set, the system updates the inverted index residing in memory. The update operation specifically involves: searching for the record with the key in the hash table or skip list structure, and appending the semantic atom identifier to the inverted list corresponding to that key. This mapping update operation is represented as: ;

[0080] Through this step, the system establishes an index path from new vocabulary to atomic identifiers, enabling semantic atoms containing new vocabulary to be retrieved through keyword matching.

[0081] In sub-step S204, an adaptive snapshot fusion determination is performed. After each log append operation, the system reads the current length counter of the differential log queue. The system determines whether a preset snapshot trigger threshold has been exceeded (for example). If the determination is yes, the system adds the semantic atomic identifier to the background asynchronous processing queue, triggering the snapshot fusion task.

[0082] In sub-step S205, asynchronous snapshot merging and state reset are performed. The background task process reads the reference text and all log entry sequences from the differential log queue. The system applies patches to the reference text sequentially according to the timestamp field order, synthesizing a new text snapshot. The synthesis function is defined as:

[0083] Here, represents an ordered sequence. After generation, the system persists it and updates the snapshot pointer to point to the new storage address. To ensure data consistency, the system performs an atomic truncation operation before clearing old logs:

[0084] The system records the offset of the last merged log entry, deletes only entries in the differential log queue with offsets less than or equal to the current offset, retains log entries newly written during the fusion process, and updates the length counter accordingly. This step ensures that the snapshot fusion process does not result in the loss of new data in high-concurrency write scenarios.

[0085] See attached document Figure 5 Step S3 illustrates the specific execution flow of transient dual-path collaborative retrieval. This step is configured to respond to a user query request by performing semantic retrieval based on dense vectors and precise retrieval based on sparse keywords in parallel, generating a candidate semantic atom set containing initial relevance information. Step S3 specifically includes the following sub-steps:

[0086] In sub-step S301, preprocessing and parallel parsing of the query request are performed. The system receives the natural language query text input by the user. The system concurrently performs the corresponding feature extraction operations in a multi-threaded execution environment or asynchronous task pool. On one hand, the system calls a pre-built deep semantic encoding model to perform inference and generate a query vector that satisfies the query. On the other hand, the system calls a word segmenter to perform word segmentation and stop word removal, generating a set containing query keywords. The generated query vector and keyword set are transmitted to the main retrieval unit and the auxiliary retrieval unit, respectively.

[0087] In substep S302, the main path baseline semantic retrieval is performed. The system inputs the query vector into the vector database. The vector database performs an approximate nearest neighbor search based on a pre-built index structure (such as an HNSW graph index or an IVF inverted file index). The system calculates the similarity score between the reference vector and the stored semantic atoms. The similarity score is calculated using cosine similarity:

[0088] The system selects the top semantic atom identifiers with the highest similarity scores based on a preset recall parameter, forming a baseline candidate set. For each semantic atom identifier in the set, the system retains its original similarity score for subsequent ranking calculations.

[0089] In sub-step S303, the transient anchor point retrieval for the auxiliary path is performed. The system traverses the query keyword set and uses the inverted index residing in memory. Search within. For Each query keyword in The system is based on The system retrieves the corresponding inverted list in time. It then performs a set union operation on all retrieved inverted lists to generate a transient candidate set. The retrieval logic is expressed as follows: ;

[0090] To facilitate subsequent processing, the system will The semantic atom identifier in the text is marked as an anchor recall attribute, and this attribute is bound to the atom identifier. This step ensures that semantic atoms that have been added to the query due to text updates can be captured by the system, regardless of the similarity performance of their baseline vectors.

[0091] In sub-step S304, candidate set fusion and state bitmap filtering are performed. The system first processes the baseline candidate set... With transient candidate set Perform a union operation to generate the initial full set. Subsequently, the system accesses the global state bitmap in the differential state management module 200. This state bitmap uses semantic atomic identifiers. This index offset stores the lifetime status of each atom (e.g., 00 indicates normal, 01 indicates obsolete, and 10 indicates deleted). System traversal. Each identifier in Perform bitwise operation detection. If the detection result shows... Corresponding status bit If an identifier is deleted or obsolete, the system removes it and its associated similarity score from the set. This filtering mechanism ensures that the baseline vector is removed even if it has been logically deleted. Physically retained atoms will not proceed to subsequent processes. The final candidate set after filtering is denoted as... This includes atomic identifiers, source markers (main road / secondary road), and baseline similarity scores (if present).

[0092] See attached document Figure 6 Step S4 illustrates the specific execution flow of dynamic view synthesis with loop detection. This step is configured to reconstruct the static baseline data in the storage layer and the dynamic differential log in memory, and resolve the reference relationships between semantic atoms. Step S4 specifically includes the following sub-steps:

[0093] In substep S401, the synthesis context is initialized. This is for the candidate set... Each target semantic atom identifier selected in The system creates a recursive composition context object in the memory stack. The context object Used to maintain state consistency during recursive calls, it internally contains: the current set of recursive paths. (Used to record nodes on the current recursive chain to detect loops), and the current recursion depth counter. In the initial state, .

[0094] In substep S402, benchmark loading and log replay are performed. The system reads the semantic atom identifier. Snapshot pointer If the pointer points to a valid text snapshot The system loads it into a variable string buffer; if no snapshot exists, the base text is loaded. Subsequently, the system retrieves data from the differential log queue. Retrieve all generated timestamp fields Later than the snapshot generation time The system iterates through the log entries in ascending order of timestamps, performing memory operations based on the Action field in the entry: if Action is APPEND, the patch content is appended to the end of the buffer; if Action is DELETE or UPDATE, a specified range of characters in the buffer is deleted or overwritten based on the offset parameter in the patch. The intermediate text generated in this process is denoted as... Formal description: ;in, This is a serialization replay function based on operation logs.

[0095] In sub-step S403, reference identifier parsing is performed. The system uses a preset regular expression to process the intermediate text. Perform a scan and build a set containing all referenced target identifiers. .like If empty, the system will directly... Returned as the final view; if If not empty, the system pauses text output at the current level and continues output for each referenced identifier in the set. Prepare for recursive calls, where, .

[0096] In sub-step S404, loop detection and recursion depth control are performed. This is initiated when targeting the identifier... Before making a recursive request, the system utilizes the context object. Perform verification. First, the system determines the current depth. Does it exceed the preset maximum recursion depth threshold? (For example Secondly, system query. Does it already exist in the current set of recursive paths? The checking logic is defined by the following discriminant: ;in, This is a Boolean logic discrimination flag.

[0097] like If true, the system terminates the recursion of that branch and returns the preset placeholder text to replace the reference label. If If false, the system creates a new context copy. ,in and Jump to execution Recursively call steps S402 to S405 to obtain Real-time view content .

[0098] In substep S405, view recursive embedding and final composition are performed. After all recursive calls to child references return, the system obtains the resolved content map. The system iterates through the intermediate text. Replace all matched reference tags with the corresponding live view content. After all replacement operations are completed, the text in the buffer is the semantic atomic identifier. Full dynamic view at the current moment This view aggregates the latest changes to the current atom and the latest states of all external atoms in its dependency chain.

[0099] See attached document Figure 7 Step S5 illustrates the specific execution flow of semantic consistency calibration and output. This step is configured to re-evaluate the relevance of the dynamically synthesized semantic atomic view and correct the ranking result by calculating a semantic drift penalty term. Step S5 specifically includes the following sub-steps:

[0100] In sub-step S501, real-time text relevance scoring is performed. The system iterates through the candidate semantic atom set generated after step S4. For each semantic atomic identifier in the set. and its corresponding real-time dynamic view The system calculates its relationship with the user's query text. The degree of text matching between them is denoted as the real-time relevance score. The calculation employs the BM25 algorithm based on a probabilistic retrieval model. Its calculation logic satisfies the formula: ;in, Indicates the search keywords In real-time dynamic view Word frequency in; The word count length of the current view; The average text length of all semantic atoms within the knowledge base; and Adjustment factor (typical value) ); Represents the inverse document frequency.

[0101] In sub-step S502, score normalization is performed. This is based on the similarity score of the main road baseline vector generated in step S3. Real-time correlation score Given different numerical units, the system performs a normalization operation based on the statistical distribution of the current candidate set. The system first iterates through the current set. Obtain the maximum value of the real-time relevance score. and minimum value Subsequently, the system uses the Min-Max scaling formula to calculate the normalized real-time score. : ;in, To prevent division by zero of small constants.

[0102] Similarly, the system calculates the similarity score of the baseline vector. Performing the same normalization process yields Specifically, if semantic atomic identifiers The source is marked as secondary road-only recall, and the system uses its baseline vector similarity score. Forced to be set to 0.

[0103] In sub-step S503, the semantic drift penalty coefficient is calculated. The system calculates the absolute value of the difference between the normalized baseline vector similarity score and the real-time text score, which is defined as the drift degree. The system calculates dynamic confidence weights based on drift. This is used to measure the reliability of the baseline vector at the current moment. The calculation formula is as follows: ;

[0104] in, Hyperparameters used to control the steepness of the penalty curve; This is a preset drift tolerance threshold. The formula exhibits an S-shaped curve characteristic; when the drift degree... Less than hour, Approaching 1; when the drift exceeds the threshold, It rapidly decays to 0.

[0105] In sub-step S504, the comprehensive ranking score is calculated. The system reads the semantic atom identifier. The anchor recall attributes bound in step S3 are used to calculate the final score according to the following weighted fusion formula. ; ;

[0106] in, These are the preset base weights for the baseline vector score and the real-time text score, respectively, and satisfy the following conditions: ; This is an indicator function whose value depends on the anchor recall attribute generated in step S3: if If this property is present, the function value is 1; otherwise, it is 0. The preset reward bias value is configured as a positive number (e.g., 0.2) and is used as a reward for secondary path recall to improve the ranking of atoms containing the new keywords.

[0107] In sub-step S505, sorting truncation and result output are performed. The system then... For candidate set All semantic atomic identifiers are sorted in descending order. The system selects the first element after sorting. bits (e.g.) Extract the corresponding real-time dynamic view from the semantic atomic identifier of ). The system construction includes The response message for each dynamic view is returned to the user through the application programming interface. Each view in the output is a composite text containing the latest differential content and recursive references.

[0108] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for automatic updating and retrieving of a knowledge base, characterized in that, Includes the following steps: The original document set is divided into several semantic atoms, the baseline vector of each semantic atom is calculated and stored in the vector database, and a differential log queue is initialized for each semantic atom. Receive a content modification request for the semantic atom, generate a text difference patch based on the content modification request and write it to the difference log queue, and simultaneously extract the changed feature words in the text difference patch to update the inverted index; Receive a query request input by the user, perform similarity retrieval based on the baseline vector and inverted index retrieval based on the changed feature words based on the query request, and merge the retrieval results to form a candidate set; For the semantic atoms in the candidate set, read the baseline text, apply the text difference patch in the differential log queue, and construct a real-time text view; Calculate the text matching degree between the query request and the real-time text view, calculate the drift penalty term based on the similarity retrieval result of the baseline vector, and reorder the candidate semantic atoms according to the comprehensive ranking score including the drift penalty term and output them.

2. The method for automatic updating and retrieving a knowledge base according to claim 1, characterized in that, The step of generating a text differential patch based on the content modification request and writing it to the differential log queue specifically includes: Read the reference text of the semantic atom, compare the differences between the modified text content and the reference text, and generate the text difference patch that records the position and length of the deleted characters and the content of the added characters; Construct an operation log entry that includes the operation type, the text difference patch, and a timestamp; The operation log entries are appended to a key-value storage system independent of the vector database storage, as new records in the differential log queue, without triggering an update operation on the base vector.

3. The method for automatic updating and retrieving a knowledge base according to claim 2, characterized in that, Also includes: When the text content of the semantic atom changes, the value and physical address of the base vector stored in the vector database remain unchanged; Monitor the length of the differential log queue. When the number of operation log entries reaches a preset threshold, perform a snapshot fusion operation in the background to generate a text snapshot and update the snapshot pointer. The step of reading the reference text of the semantic atom specifically involves: reading the text snapshot pointed to by the snapshot pointer as the reference text.

4. The method for automatic updating and retrieving a knowledge base according to claim 1, characterized in that, The step of merging the search results to form a candidate set specifically includes: Based on the query request, a query vector is generated, and semantic atoms that meet the preset conditions for similarity with the benchmark vector are retrieved from the vector database to generate a benchmark candidate set. Based on the query keywords extracted from the query request, the query keywords are matched with the change feature words stored in the inverted index, the associated semantic atoms are retrieved, a transient candidate set is generated, and the anchor recall attribute is marked. Perform a union operation on the baseline candidate set and the transient candidate set. For each semantic atom contained in the union operation result, check the life cycle state of each semantic atom according to the global state bitmap, remove semantic atom identifiers whose state bits show as deleted or obsolete, and obtain the candidate set.

5. The method for automatic updating and retrieving a knowledge base according to claim 1, characterized in that, The steps for constructing a real-time text view also include reference resolution and recursive processing, specifically including: The text difference patch is applied to the base text to generate intermediate text, and the intermediate text is parsed to identify reference markers in the intermediate text that point to other semantic atoms; In response to the identification of the reference identifier, a recursive call request for the referenced semantic atom is generated, and a recursive synthesis context object containing the current set of recursive paths and a recursion depth counter is created; Before executing the recursive call request, it is determined whether the current recursion depth exceeds a preset threshold and whether the referenced semantic atom already exists in the current recursion path set based on the recursive synthesis context object; If the threshold is not exceeded and there is no circular reference, the recursive call request is executed to obtain the real-time view content of the referenced semantic atom, and the reference identifier in the intermediate text is replaced with the real-time view content.

6. The method for automatic updating and retrieving a knowledge base according to claim 1, characterized in that, The steps for calculating the drift penalty term specifically include: The similarity scores obtained from the similarity retrieval results of the baseline vector and the text matching degree between the query request and the real-time text view are normalized respectively. The absolute value of the difference between the normalized similarity score and the text matching degree is calculated as the drift degree. The dynamic confidence weight is calculated based on the drift degree and used as the drift penalty term. When the drift degree exceeds the preset drift tolerance threshold, the dynamic confidence weight decays non-linearly.

7. The method for automatic updating and retrieving a knowledge base according to claim 6, characterized in that, The step of reordering the candidate semantic atoms based on the comprehensive ranking score including the drift penalty term specifically includes: Determine whether the semantic atom has the anchor recall attribute. If it has the anchor recall attribute, set the indicator function value to a valid value; otherwise, set it to zero. The normalized similarity score is multiplied by the dynamic confidence weight, and then weighted and summed with the normalized text matching degree and the product of the indicator function value and the preset reward bias value to obtain the comprehensive ranking score. The semantic atoms in the candidate set are sorted in descending order based on the comprehensive ranking score, and the top-ranked real-time text views are extracted as the search results.

8. A knowledge base automatic update and retrieval system, characterized in that, The method for automatic updating and retrieving of a knowledge base, applied to any one of claims 1-7, includes: The semantic storage module (100) is used to divide the original document set into several semantic atoms, calculate the base vector of each semantic atom and store it in the vector database, and initialize a differential log queue for each semantic atom. The state management module (200) is used to receive content modification requests for the semantic atom, generate text difference patches based on the content modification requests and write them to the difference log queue, and at the same time extract the changed feature words in the text difference patches to update the inverted index. The dual-path retrieval module (300) is used to receive a query request input by the user, perform similarity retrieval based on the reference vector and inverted index retrieval based on the changed feature words based on the query request, and merge the retrieval results to form a candidate set; The view synthesis module (400) is used to read the baseline text for the semantic atoms in the candidate set, apply the text difference patch in the differential log queue to construct a real-time text view, calculate the text matching degree between the query request and the real-time text view, calculate the drift penalty term in combination with the similarity retrieval result of the baseline vector, and output the retrieval result based on the comprehensive ranking score.

9. The knowledge base automatic update and retrieval system according to claim 8, characterized in that, The semantic storage module (100) uses regular expression-based paragraph recognition, sliding window-based truncation, or DOM tree structure-based node parsing strategies to segment the original document set into semantic atoms, and uses a hash algorithm or distributed unique ID generator to generate a unique identifier for each semantic atom.

10. The knowledge base automatic update and retrieval system according to claim 8, characterized in that, The view synthesis module (400) uses an algorithm based on a probability retrieval model to calculate the text matching degree. The calculation parameters of the algorithm include the word frequency, inverse document frequency and average text length of all semantic atoms in the real-time text view of the query keyword.

Images