Local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction

Abstract

The application relates to the technical field of big data, and provides a local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction, which comprises the following steps: in response to an index refreshing request of a target knowledge base, reading all enabled sources under the target knowledge base; collecting source data of each enabled source, converting the collection results into document texts in a unified format, generating knowledge segments by fragmentation, aggregating all the knowledge segments into a same segment set, accumulating a unified index abstract, and forbidding direct writing of a single source into a local full-text index during the collection process; after the collection of all the enabled sources is completed, writing all the knowledge segments in the same segment set into the local full-text index at a time in an overall reconstruction mode; and writing back the index state of the target knowledge base according to the unified index abstract, so that the index state represents the consistency between the current index snapshot and all the enabled sources. The application can completely avoid the state inconsistency problems caused by mutual coverage of multiple sources, residual old segments and partial failures.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method for maintaining the consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction. Background Technology

[0002] The local knowledge base question-answering system allows users to incorporate various external resources, such as local folders, webpage URLs, or image recognition results, into a single knowledge base for unified management and question answering. The system scans various sources of information through periodic or user-triggered index refreshes, extracts text content, segments it into fragments, and writes them into a full-text index, enabling users to search and ask questions based on this index. In single-source scenarios (i.e., a knowledge base is associated with only one local directory or one webpage), the system can directly write all fragments from that source into the index, making the logic relatively simple.

[0003] In practical applications, the same knowledge base is often associated with multiple sources (e.g., a mixture of multiple local folders and multiple web page URLs). If the index maintenance method of writing to each source independently or simple incremental overwriting is still used, the following problems will occur:

[0004] 1. Overwriting between multiple sources leads to incomplete knowledge base content. If each source writes independently by creating an index when refreshing, the source that refreshes later will overwrite all the indexed content of the previous source. This results in the final knowledge base only retaining fragments from the last source, with data from other sources being lost. Users will not be able to retrieve content from the overwritten sources when asking questions.

[0005] 2. Old fragments of deleted data persist, reducing the reliability of question and answer responses. If the system only appends indexes to newly added or modified documents without processing documents removed from the source, then when a user deletes a local file, removes a URL, or disables a source, the corresponding old index fragments will still remain in the index. Subsequent question and answer responses will still hit these outdated data, generating incorrect answers.

[0006] 3. Heterogeneous source types lead to inconsistent index status. The extraction processes and parsing tools for local folders, web page URLs, and image recognition text are different. If each type of source is written to the index independently, the system will find it difficult to form a complete index snapshot of the knowledge base within the same refresh cycle. The index status of different sources is inconsistent over time, affecting the reliability of search results.

[0007] 4. Lack of a unified fault handling mechanism when source parts fail. In actual data collection, it is difficult to completely avoid some file parsing errors or URL crawling timeouts. Without a unified fragment aggregation and status write-back mechanism, the system cannot accurately determine whether the knowledge base still has usable search content, nor can it decide whether to allow users to continue asking questions, reducing the system's availability and maintainability.

[0008] 5. Simple incremental strategies are prone to introducing consistency errors in engineering implementation. Incremental indexing requires reliable maintenance of multiple types of metadata, such as file hash values, modification times, deletion lists, and failure retry statuses. In early products or resource-constrained desktop systems, when these states are not yet fully integrated, forcibly implementing incremental refreshes can easily lead to consistency problems such as mixing of old and new fragments, fragment duplication, or loss.

[0009] Chinese patent CN122432172A discloses a multi-index knowledge base and its update method and query self-healing method. Chinese patent CN118585496A discloses a knowledge base update method. Neither of them can solve the problems of mutual overwriting between sources, persistent old fragments corresponding to deleted files or removed URLs, lack of unified status judgment when some sources fail, and inconsistent index status of multiple sources when multiple heterogeneous sources (local folders, URL web pages, image recognition text) under the same knowledge base are refreshed in parallel.

[0010] Therefore, how to provide a method that can uniformly aggregate the collection results from all sources and atomically replace the index as a whole, so as to ensure that the retrieval results of the knowledge base after the refresh are strictly consistent with the currently enabled set of all sources, thereby overcoming the defects of multi-source coverage, old content residue, inconsistent status and insufficient fault handling, has become an urgent technical problem to be solved. Summary of the Invention

[0011] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide a method for maintaining the consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction.

[0012] This invention provides a method for maintaining the consistency of a local knowledge base index based on multi-source aggregation and overall reconstruction. The method includes:

[0013] Step S1: In response to the index refresh request of the target knowledge base, read all enabled sources under the target knowledge base;

[0014] Step S2: Collect source data from each enabled source, convert the collected results into document text in a unified format, generate knowledge fragments by segmentation, aggregate all knowledge fragments into the same fragment set, and accumulate a unified index summary. During the collection process, it is prohibited to directly write a single source into the local full-text index.

[0015] Step S3: After all source collection is completed, write all knowledge fragments in the same fragment set into the local full-text index at once using the overall reconstruction method;

[0016] Step S4: Write back the index status of the target knowledge base based on the unified index digest, so that the index status represents the consistency between the current index snapshot and all enabled sources.

[0017] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S1, the enabled sources include one or more of local folder sources and Uniform Resource Locator (URL) list sources; when reading all enabled sources under the target knowledge base, all enabled sources are queried according to the knowledge base identifier of the target knowledge base, and the queried list of all enabled sources is passed to the index service.

[0018] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S2, source data of each enabled source is collected in the following manner: recursively scanning the local directory for the local folder source, filtering document files and image files corresponding to preset extensions; parsing each Uniform Resource Locator (URL) for the Uniform Resource Locator (URL) list source line by line and capturing web page content; when a single document, image, or URL fails during the collection process, only the failure count is increased and other objects are collected, without interrupting the collection process of the current source and subsequent sources.

[0019] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S2, the collected results are converted into document text in a unified format as follows: for document files, the document parser is used first to extract the main text, and direct text reading is used as a fallback if the parsing fails; for image files, the image content recognition module is called to convert them into searchable text; for Uniform Resource Locator (URL) pages, the webpage main text is extracted and scripts, styles, navigation, and footer noise nodes are removed; after conversion, the collected results of each type are uniformly converted into document text containing the title, source location, and main text.

[0020] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S2, knowledge fragments are generated by segmenting in the following manner: after normalizing the whitespace characters of the main text of the document, it is segmented according to a preset fragment length and a preset overlap length; each knowledge fragment includes at least a fragment identifier, a document identifier, a document title, a source location, and fragment text, wherein the fragment identifier is generated by combining the document identifier and the fragment sequence number to maintain the sequential relationship and traceability relationship of each knowledge fragment within the same document.

[0021] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S2, the same fragment set is maintained in memory or temporary storage during the collection phase. After each source is successfully collected, all the knowledge fragments generated are appended to the same fragment set. The unified index summary includes at least the number of indexed documents, the number of generated fragments, and the number of failed documents, and is updated synchronously with the collection progress. The same fragment set constitutes a complete snapshot of the target knowledge base at the current refresh time.

[0022] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S3, all knowledge fragments in the same fragment set are written to the local full-text index at once in an overall reconstruction manner. Specifically, the local full-text index directory corresponding to the target knowledge base is opened in creation mode, the old index content is cleared, and all knowledge fragments in the same fragment set are written at once, so that the knowledge fragments that exist in the old index but were not collected this time are removed as a whole, and the current knowledge fragments from all successful sources are retained as a whole.

[0023] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S4, the index status of the target knowledge base is written back according to the unified index digest in the following manner: when there is at least one knowledge fragment in the same fragment set, the target knowledge base is marked as a queryable state and the number of failed documents is retained in the index digest; when no knowledge fragments are generated from all enabled sources and there are failed documents, the target knowledge base is marked as a failed state; when the enabled sources are empty, an empty index digest is returned and the target knowledge base is marked as an unready state.

[0024] Optionally, in the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention, in step S4, the index status is displayed in the user interface as the number of indexed documents, the number of generated fragments, and the number of failed documents; during the refresh task execution, the interface displays the status in the index; after the refresh is completed, the interface displays a unified index summary; when the refresh fails, the interface displays a failure prompt message.

[0025] Optionally, the local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention further includes, after step S4: in response to a user's query request, retrieving knowledge fragments only in the local full-text index after the most recent overall reconstruction; when a file in the target knowledge base is deleted or a Uniform Resource Locator (URL) is removed, during the next index refresh of the target knowledge base, the old knowledge fragments corresponding to the deleted file or the removed URL are completely removed from the local full-text index through overall reconstruction, and subsequent searches will no longer hit the old knowledge fragments.

[0026] The local knowledge base index consistency maintenance method based on multi-source aggregation and overall reconstruction of the present invention has the following beneficial technical effects: taking the target knowledge base as the refresh boundary, the collection results of all enabled sources are first aggregated into a unified fragment set, and then written to the local full-text index in one go in the overall reconstruction manner. This fundamentally avoids the mutual overwriting problem caused by independent writing between multiple sources. At the same time, by clearing the old index and then writing, it is ensured that the old fragments corresponding to deleted files or removed URLs are completely removed, so that the index snapshot after the refresh is strictly consistent with the current set of all enabled sources, effectively improving the reliability and consistency of multi-source local knowledge base index maintenance. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to an exemplary embodiment 1 of the present invention.

[0029] Figure 2 This is a flowchart illustrating the multi-source heterogeneous data acquisition and failure tolerance method according to Exemplary Embodiment 1 of the present invention.

[0030] Figure 3 This is a schematic diagram of the process for unifying the format of heterogeneous source acquisition results according to the method of exemplary embodiment 1 of the present invention;

[0031] Figure 4 This is a schematic diagram of the knowledge base index status determination and write-back process according to an exemplary embodiment 1 of the present invention;

[0032] Figure 5 This is a schematic diagram showing the index status user interface display process of the method according to an exemplary embodiment 1 of the present invention;

[0033] Figure 6 This is a schematic diagram of the overall reconstruction index retrieval and expired fragment removal process according to Exemplary Embodiment 1 of the present invention. Detailed Implementation

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0037] Example 1

[0038] Exemplary embodiment 1 of the present invention provides a method for maintaining the consistency of a local knowledge base index based on multi-source aggregation and overall reconstruction. Figure 1 This is a flowchart illustrating the method for maintaining consistency of a local knowledge base index based on multi-source aggregation and overall reconstruction according to an exemplary embodiment 1 of the present invention. Figure 1 As shown, in this embodiment, the method of the present invention is implemented in the following manner:

[0039] Step S1: In response to the index refresh request of the target knowledge base, read all enabled sources under the target knowledge base.

[0040] In this embodiment, the enabled sources include one or more of the local folder source and the Uniform Resource Locator (URL) list source; when reading all enabled sources under the target knowledge base, all enabled sources are queried based on the knowledge base identifier of the target knowledge base, and the list of all enabled sources obtained from the query is passed to the index service.

[0041] This embodiment queries all enabled sources based on the knowledge base identifier and uniformly transmits the source list to the index service. This ensures that the refresh operation always operates within the entire knowledge base, preventing incomplete indexing due to missing sources or inconsistent source status. This gives index refresh in multi-source scenarios clear input boundaries and complete operation objects.

[0042] Step S2: Collect source data from each enabled source, convert the collected results into document text in a unified format, generate knowledge fragments by segmentation, aggregate all knowledge fragments into the same fragment set, and accumulate a unified index summary. During the collection process, it is prohibited to directly write a single source into the local full-text index.

[0043] Figure 2 This is a schematic diagram of the multi-source heterogeneous data acquisition and failure tolerance process according to Exemplary Embodiment 1 of the present invention, as shown below. Figure 2 As shown, in this embodiment, source data from each enabled source is collected in the following manner: For local folder sources, the local directory is recursively scanned, and document and image files with preset extensions are filtered; for Uniform Resource Locators (URLs) from the URL list sources, each URL is parsed line by line, and webpage content is captured, achieving unified collection capabilities for heterogeneous sources. If a single document, image, or URL fails during the collection process, only the failure count is incremented, and collection of other objects continues without interrupting the collection process for the current and subsequent sources. This effectively prevents single-point failures from dragging down the entire refresh process, significantly improving fault tolerance and collection completion rate in complex data environments.

[0044] Figure 3 This is a schematic diagram of the process for unifying the format of heterogeneous source acquisition results according to the method of exemplary embodiment 1 of the present invention, as shown below. Figure 3 As shown, in this embodiment, the collected results are converted into document text of a unified format in the following way: For document files, a document parser is used first to extract the main text; if parsing fails, direct text reading is used as a fallback. For image files, an image content recognition module is called to convert them into searchable text. For Uniform Resource Locator (URL) pages, the webpage text is extracted and scripts, styles, navigation, and footer noise nodes are removed. After conversion, all types of collected results are uniformly converted into document text containing a title, source location, and main text. This embodiment shields the format differences between different sources, enabling subsequent segmentation, indexing, and retrieval stages to handle multiple data types with a unified processing logic, reducing complexity and enhancing code reusability and extensibility.

[0045] In this embodiment, knowledge fragments are generated by segmenting the document text as follows: after normalizing whitespace characters, the text is divided according to a preset fragment length and a preset overlap length. Each knowledge fragment includes at least a fragment identifier, a document identifier, a document title, a source location, and fragment text. The fragment identifier is generated by combining the document identifier and the fragment sequence number to maintain the sequential and traceable relationship between knowledge fragments within the same document. When searching, users can not only obtain relevant fragment content but also accurately trace back to the original document and its specific location, significantly improving the interpretability and verifiability of the question-and-answer results.

[0046] In this embodiment, during the acquisition phase, a set of the same fragments is maintained in memory or temporary storage. After successful acquisition from each source, all knowledge fragments generated are appended to this same fragment set. The unified index summary includes at least the number of indexed documents, the number of generated fragments, and the number of failed documents, and is updated synchronously with the acquisition progress. The same fragment set constitutes a complete snapshot of the target knowledge base at the current refresh time. This ensures the integrity of the unified fragment set and the real-time performance of the unified index summary throughout the entire refresh process, enabling the acquisition of a complete snapshot of the current refresh batch after all acquisitions are completed, providing a reliable data foundation for subsequent overall reconstruction and state write-back.

[0047] Step S3: After all source collection is enabled, write all knowledge fragments in the same fragment set into the local full-text index at once using the overall reconstruction method.

[0048] In this embodiment, all knowledge fragments in the same fragment set are written to the local full-text index at once using the overall reconstruction method. Specifically, the local full-text index directory corresponding to the target knowledge base is opened in creation mode, the old index content is cleared, and all knowledge fragments in the same fragment set are written at once. This removes the knowledge fragments that exist in the old index but were not collected this time, while all current knowledge fragments from successful sources are retained.

[0049] This embodiment achieves atomic index replacement by opening the index directory in creation mode, clearing the old index, and then writing all knowledge fragments at once for overall reconstruction. This ensures that any fragments that exist in the old index but were not collected this time are removed as a whole, while ensuring that all current fragments from successful sources are completely preserved. This completely eliminates the risk of fragment residue or omission caused by incremental updates and ensures strong consistency in index switching.

[0050] Step S4: Write back the index status of the target knowledge base based on the unified index digest, so that the index status represents the consistency between the current index snapshot and all enabled sources.

[0051] Figure 4 This is a schematic diagram of the knowledge base index status determination and write-back process according to an exemplary embodiment 1 of the present invention, as follows: Figure 4 As shown, in this embodiment, the index status of the target knowledge base is written back according to the unified index digest in the following manner: when there is at least one knowledge fragment in the same fragment set, the target knowledge base is marked as a questionable state and the number of failed documents is retained in the index digest; when no knowledge fragments are generated by all enabled sources and there are failed documents, the target knowledge base is marked as a failed state; when the enabled sources are empty, an empty index digest is returned and the target knowledge base is marked as an unready state.

[0052] This embodiment establishes a complete knowledge base health assessment system by marking the knowledge base as available for questioning, failed, or not ready based on the existence of knowledge fragments and failed documents in the same fragment set. It also implements a failure-tolerant state write-back mechanism, which allows successfully collected knowledge fragments to still provide question-and-answer services even when some sources or documents fail. At the same time, the number of failures is retained and displayed, which maximizes the availability of the knowledge base and provides clear guidance for users to troubleshoot problems later.

[0053] Figure 5 This is a schematic diagram illustrating the index state user interface display process of the method according to Exemplary Embodiment 1 of the present invention, as follows: Figure 5 As shown in this embodiment, the index status is displayed in the user interface as the number of indexed documents, the number of generated fragments, and the number of failed documents. During the refresh task, the interface displays the index status. After the refresh is complete, the interface displays a unified index summary. If the refresh fails, the interface displays a failure message. This embodiment provides users with real-time, transparent, and operable index status feedback, enabling users to clearly perceive the refresh progress, results, and any anomalies, facilitating timely intervention or retrying.

[0054] Figure 6 This is a schematic diagram of the overall index reconstruction retrieval and expired fragment removal process according to Exemplary Embodiment 1 of the present invention, as follows: Figure 6 As shown, in this embodiment, after step S4, the method further includes: responding to the user's query request, retrieving knowledge fragments only from the local full-text index after the most recent overall reconstruction; when a file in the target knowledge base is deleted or a Uniform Resource Locator is removed, during the next index refresh of the target knowledge base, the old knowledge fragments corresponding to the deleted file or the removed Uniform Resource Locator are completely removed from the local full-text index through overall reconstruction, and subsequent searches will no longer hit the old knowledge fragments.

[0055] This embodiment ensures that the retrieval scope during the question-and-answer phase always points to the latest valid snapshot of the knowledge base by retrieving knowledge fragments only from the most recently reconstructed index when a user asks a question, and by completely removing the corresponding old fragments from the index through the next overall reconstruction after a file is deleted or a URL is removed. This effectively prevents erroneous output caused by expired information continuing to participate in the answer, and fundamentally guarantees the accuracy and timeliness of the retrieval results of the local knowledge base question-and-answer system.

[0056] Example 2

[0057] Exemplary Example 2 of this invention provides a method for maintaining the consistency of a local knowledge base index based on multi-source aggregation and overall reconstruction. This embodiment further details the method of this invention in conjunction with the DDKnowledge local knowledge base system. In this embodiment, the method of this invention is implemented according to the following steps:

[0058] Step 1: Maintain the knowledge base and source data structure

[0059] The system maintains a knowledge base identifier, name, status, creation time, and update time for each knowledge base. Each knowledge base can be associated with one or more sources. Source records must include at least the source identifier, knowledge base identifier, source type, source location, and enabled status.

[0060] Source types may include: 1. Local folder source, used to represent a local directory; 2. URL list source, used to represent one or more web page addresses.

[0061] In practical applications, when refreshing the index, the knowledge base service first queries all enabled sources by knowledge base identifier, and then passes the list of enabled sources to the index service.

[0062] Step 2: Read the source list uniformly

[0063] After receiving the refresh command, the system first updates the target knowledge base status to reflect the index. Then, it reads all enabled sources for that knowledge base. If no enabled sources are found, the system returns an empty index summary. If enabled sources exist, the system collects them sequentially according to their order, but does not directly replace the index during the collection phase. The refresh operation targets the "knowledge base," not a single source. All sources collectively constitute the snapshot boundary of the currently available question-and-answer materials for that knowledge base.

[0064] Step 3: Collect local folder source

[0065] For local folder sources, the system recursively scans common document and image files within the directory. Common documents may include text files, Markdown files, PDFs, Word documents, and RTF documents; image files may include PNG, JPG, JPEG, WebP, and BMP.

[0066] For regular documents, the system prioritizes using the document parser to extract the main text. If parsing a text file or Markdown file fails, the file content can be read directly as a fallback. If the extraction result is empty, it will attempt to read the text.

[0067] For image files, the system calls the image content extraction module to convert the image into searchable text. Image recognition failures should not interrupt the entire knowledge base refresh; instead, they should be counted as failed documents.

[0068] Each successfully extracted local document will generate a document title, source location, and body text. The source location can use a relative path for subsequent retrieval and evidence presentation.

[0069] Step 4: Collect URL Sources

[0070] For URL sources, the system parses the URLs line by line from the source location. Each URL, after being crawled, forms a page title and body text. During page body extraction, noisy nodes such as scripts, styles, navigation, and footers can be removed, ensuring that only the main text of the page is indexed. When crawling a URL fails, the system only increments the failure count, without terminating the collection of other URLs or other sources. This prevents a single page failure from causing the entire knowledge base to become completely unrefreshable.

[0071] Step 5: Text Segmentation

[0072] The system segments the main text of each document or webpage. During segmentation, whitespace characters can be normalized first, and then the segments can be divided according to preset fragment lengths and overlap lengths.

[0073] Each segment should include at least the following fields: 1. Segment ID; 2. Document ID; 3. Document Title; 4. Source Location; 5. Page Number or Anchor Point; 6. Segment Text. The segment ID can be generated by combining the document ID and the segment sequence number, thereby maintaining the order and traceability of segments within the same document.

[0074] Step Six: Unify the Fragment Set

[0075] The system maintains a unified fragment set during the refresh process. After each source successfully collects data, the system appends the fragments generated by that source to the unified fragment set, instead of immediately writing them to the index.

[0076] Meanwhile, the system maintains a unified index summary, including: 1. the number of documents successfully indexed; 2. the number of fragments generated; and 3. the number of documents that failed to be collected or parsed. This unified fragment set represents the current snapshot of the target knowledge base after a single refresh.

[0077] Step 7: Rebuild the index as a whole

[0078] Once all enabled sources have been collected, the system opens the local index directory corresponding to the target knowledge base and writes the index in creation mode. Creation mode clears the old index content and then writes all fragments from a unified fragment set. This differs from document-by-document appending and from independent replacement for each source. In this embodiment, fragments that existed in the old index but were not collected in this refresh will be removed; current fragments from all successful sources will be retained.

[0079] For example, a knowledge base may have sources A and B. If the system first collects fragments from source A, then collects fragments from source B, and finally rebuilds the index, the final index will contain the current fragments from both sources A and B. If a file in source A has been deleted, the current collection will not generate fragments for that file, and the old fragments will not continue to exist after the overall reconstruction.

[0080] Step 8: State Writeback and Failure Tolerance

[0081] After the overall reconstruction is completed, the system writes back the knowledge base status based on the unified index summary. In practical applications, if the number of failures is zero, the knowledge base is marked as question-asking; if there are failures but usable fragments are still generated, the knowledge base can also be marked as question-asking, and the number of failures will be displayed in the index summary; if no fragments are generated and there are failures, it is marked as failed. Partial source failures should not block the question-answering capabilities of successful sources, but the number of failures must be retained and displayed to facilitate subsequent user troubleshooting.

[0082] Step Nine: Search and Question Answering

[0083] When a user asks a question, the system performs a search within the overall reconstructed index corresponding to the target knowledge base. Search results are only from the current set of fragments generated during the most recent refresh. If a user deletes a file or removes a URL from the source, the old fragment will disappear due to the overall reconstruction once the index is refreshed again. Subsequent questions and answers will not reference that old fragment.

[0084] Step 10: Coordinating with on-screen prompts

[0085] The system can display the current indexing status, number of files, number of fragments, and number of failures of the knowledge base on the interface. When a refresh task is in progress, the interface will indicate that indexing is in progress; after the refresh is completed, the interface will display an index summary; if the refresh fails, the interface will prompt you to check the path, URL, or model configuration.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for maintaining the consistency of a local knowledge base index based on multi-source aggregation and overall reconstruction, characterized in that, The method includes: Step S1: In response to the index refresh request of the target knowledge base, read all enabled sources under the target knowledge base; Step S2: Collect source data from each enabled source, convert the collected results into document text in a unified format, generate knowledge fragments by segmentation, aggregate all knowledge fragments into the same fragment set, and accumulate a unified index summary. During the collection process, it is prohibited to directly write a single source into the local full-text index. Step S3: After all source collection is completed, write all knowledge fragments in the same fragment set into the local full-text index at once using the overall reconstruction method; Step S4: Write back the index status of the target knowledge base based on the unified index digest, so that the index status represents the consistency between the current index snapshot and all enabled sources.

2. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S1, the enabled sources include one or more of the local folder source and the Uniform Resource Locator (URL) list source; when reading all enabled sources under the target knowledge base, all enabled sources are queried based on the knowledge base identifier of the target knowledge base, and the list of all enabled sources obtained from the query is passed to the index service.

3. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S2, source data from each enabled source is collected in the following manner: For local folder sources, the local directory is recursively scanned, and document files and image files corresponding to preset extensions are filtered; for Uniform Resource Locators (URLs) list sources, each URL is parsed line by line and web page content is captured; if a single document, image, or URL fails during the collection process, only the failure count is increased and other objects are collected, without interrupting the collection process of the current source and subsequent sources.

4. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S2, the collected results are converted into document text in a unified format as follows: For document files, the document parser is used first to extract the main text, and direct text reading is used as a fallback if parsing fails; for image files, the image content recognition module is called to convert them into searchable text; for Uniform Resource Locator (URL) pages, the webpage text is extracted and scripts, styles, navigation, and footer noise nodes are removed; after conversion, all types of collected results are uniformly converted into document text containing the title, source location, and main text.

5. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S2, knowledge fragments are generated by segmenting the document text as follows: after normalizing the whitespace characters in the main text, the text is segmented according to the preset fragment length and preset overlap length; each knowledge fragment contains at least a fragment identifier, a document identifier, a document title, a source location, and fragment text. The fragment identifier is generated by combining the document identifier and the fragment number to maintain the sequential and traceable relationship of knowledge fragments within the same document.

6. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S2, during the collection phase, the same set of fragments is maintained in memory or temporary storage. After each source is successfully collected, all the knowledge fragments generated are appended to the same set of fragments. The unified index summary includes at least the number of indexed documents, the number of generated fragments, and the number of failed documents. It is updated synchronously with the collection progress. The same set of fragments constitutes a complete snapshot of the target knowledge base at the current refresh time.

7. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S3, all knowledge fragments in the same fragment set are written to the local full-text index at once using the overall reconstruction method. Specifically, the local full-text index directory corresponding to the target knowledge base is opened in creation mode, the old index content is cleared, and all knowledge fragments in the same fragment set are written at once. This removes the knowledge fragments that exist in the old index but were not collected this time, while all current knowledge fragments from all successful sources are retained.

8. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S4, the index status of the target knowledge base is written back according to the unified index digest in the following manner: when there is at least one knowledge fragment in the same fragment set, the target knowledge base is marked as quest-readable and the number of failed documents is retained in the index digest; when no knowledge fragments are generated from all enabled sources and there are failed documents, the target knowledge base is marked as failed; when the enabled sources are empty, an empty index digest is returned and the target knowledge base is marked as not ready.

9. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, In step S4, the index status is displayed in the user interface as the number of indexed documents, the number of generated fragments, and the number of failed documents; during the refresh task execution, the interface displays the status of the index; after the refresh is completed, the interface displays a unified index summary; if the refresh fails, the interface displays a failure message.

10. The method for maintaining consistency of local knowledge base indexes based on multi-source aggregation and overall reconstruction according to claim 1, characterized in that, Following step S4, the method further includes: responding to a user's query request, retrieving knowledge fragments only from the local full-text index after the most recent overall reconstruction; when a file in the target knowledge base is deleted or a Uniform Resource Locator (URL) is removed, during the next index refresh of the target knowledge base, the old knowledge fragments corresponding to the deleted file or the removed URL are completely removed from the local full-text index through overall reconstruction, and subsequent searches will no longer hit the old knowledge fragments.

Images