A dynamic knowledge retrieval system data management method and system

Abstract

The application relates to the technical field of knowledge retrieval system data management, and discloses a dynamic knowledge retrieval system data management method and system. An asynchronous deep refinement stage and a quick basic index stage are executed asynchronously. In the quick basic index stage, basic information of a knowledge entry is extracted, the basic information is written into a basic index, and after the basic index completes the basic information writing, the processing state of the knowledge entry is updated. In the asynchronous deep refinement stage, deep semantic analysis is performed on the knowledge entry whose processing state is that the basic index completes the basic information writing, and the deep semantic analysis result is written into a high-level semantic index in an incremental manner. Automatic cleaning service is performed according to the residence time of the knowledge entry in a temporary area and the processing state of the knowledge entry, including: when the residence time of the knowledge entry exceeds a preset threshold value and the processing state of the knowledge entry is that the basic index completes the basic information writing, the knowledge entry is exempted from cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge retrieval system data management, in particular to a dynamic knowledge retrieval system data management method and system. BACKGROUND

[0002] In the modern enterprise environment, an efficient dynamic knowledge retrieval system is the key infrastructure to support core business, which usually needs to process data from multiple sources. In order to ensure that the knowledge content is timely published and available for user retrieval, the system internally designs a precise knowledge entry inverted index concurrent write control mechanism, which performs outstanding performance in the conventional business scenario. However, when the business department launches a large-scale knowledge import plan to quickly respond to the market, the system faces challenges beyond the conventional load. This centralized and high-intensity import requirement requires the system to digest and index a large amount of data within a time window far exceeding the daily processing capacity, which puts a huge pressure on the carrying capacity of the system. These newly imported knowledge contents not only have a large quantity, but also have a complexity far beyond the daily processing range of the system. Emerging technical fields are often accompanied by a large number of highly specialized terms, novel abbreviations, and semantic relationships that have not been widely accepted, so that the system needs to perform deeper vocabulary analysis, entity recognition and semantic association analysis when building inverted index, resulting in a more time-consuming and resource-intensive index process for each knowledge entry. In addition, due to the high concentration of knowledge concerned by the emergency import plan in a few specific emerging technical fields, it conflicts with the existing index sharding strategy of the system, resulting in serious write hotspots, that is, a few storage nodes bear far more write pressure than their designed load.

[0003] On these core storage nodes that bear the write hotspots, the internal solid state disk (SSD) write units continue to bear high-frequency and high-intensity write operations, causing the write performance of some storage blocks to gradually decline and begin to exhibit tiny but frequent write delay jitter. These accumulated write delay jitters seriously interfere with the originally efficient knowledge entry inverted index concurrent write control lock mechanism of the system. The task holding the lock may occupy the lock resource for a long time due to waiting for the underlying write to complete, or the task waiting to acquire the lock may be forced to retry or block due to timeout, causing a large number of knowledge update tasks to be blocked in the queue, unable to complete index construction in a timely manner, resulting in some newly imported knowledge failing to enter the searchable state in a timely manner, and a significant and expanding time difference between the information obtained by the user when querying and the actual latest data, seriously affecting the real-time performance and accuracy of knowledge retrieval. In order to effectively manage the data import process and prevent resource waste, a temporary data staging and preprocessing area and an automatic cleaning mechanism are usually provided in the system to delete entries that have stayed in the area for more than a predetermined time. However, due to the above series of chain reactions, especially the severe write blocking, the index construction time of these newly imported knowledge is much longer than expected during system design. These data that are being normally processed stay in the staging area for more than the threshold set by the automatic cleaning mechanism. Therefore, the automatic cleaning mechanism mistakenly judges these still-in-processing valuable new knowledge as failed zombie data imported, and mistakenly triggers the cleaning process. This ultimately leads to the accidental deletion of a large amount of new knowledge being indexed, causing serious and irreparable damage to data integrity. SUMMARY

[0004] In order to solve the deficiencies of the prior art, the present application provides a dynamic knowledge retrieval system data management method and system.

[0005] In a first aspect, the present application provides a dynamic knowledge retrieval system data management method applied in the index of knowledge entries, the index construction process of knowledge entries including a fast basic index stage and an asynchronous deep refinement stage, the asynchronous deep refinement stage being executed asynchronously with the fast basic index stage, the method comprising the following steps:

[0006] In the fast basic index stage, the basic information of the knowledge entries is extracted, the basic information is written into the basic index, and after the basic index completes the basic information writing, the processing state of the knowledge entries is updated;

[0007] In the asynchronous deep refinement stage, the knowledge entries with the processing state of the basic index completing the basic information writing are subjected to deep semantic analysis, and the deep semantic analysis results are written into the advanced semantic index in an incremental manner;

[0008] The automatic cleaning service is performed according to the residence time of the knowledge item in the staging area and the processing state of the knowledge item, and includes: when the residence time of the knowledge item exceeds a preset threshold, and the processing state of the knowledge item is that the basic index completes the basic information writing, the knowledge item is exempted from the cleaning.

[0009] By introducing the two stages of fast basic indexing and asynchronous deep refinement, parallelization and optimization of knowledge item index construction are realized, system pressure during large-scale knowledge import is effectively relieved, intelligent automatic cleaning service is used to avoid the mistaken deletion of valuable knowledge in processing, and data integrity and system stability are significantly improved.

[0010] Further, the step of writing the basic information into the basic index and updating the processing state of the knowledge item after the basic index completes the basic information writing includes:

[0011] The knowledge item is pre-scanned to identify and record the specific identifier existing in the knowledge item;

[0012] The basic information is written into the basic index;

[0013] After the basic information is written into the basic index, the specific identifier contained in the basic index is functionally verified according to the recorded specific identifier, and a functional verification result is obtained;

[0014] According to the functional verification result, the processing state of the knowledge item is updated.

[0015] Further, the step of updating the processing state of the knowledge item according to the functional verification result includes:

[0016] When the functional verification result shows that the specific identifier is completely contained, the processing state of the knowledge item is updated to be that the basic index completes the basic information writing;

[0017] When the functional verification result shows that the specific identifier is not completely contained, the processing state of the knowledge item is updated to be that the basic index is functionally defective.

[0018] Further, the step of pre-scanning the knowledge item to identify and record the specific identifier existing in the knowledge item includes:

[0019] According to the content characteristics of the knowledge item or the source information of the knowledge item, one or more matching rules are selected from a preset identification rule library;

[0020] The selected matching rule is used for text scanning of the knowledge item, and a text segment conforming to the matching rule is identified and extracted as the specific identifier.

[0021] Further, the present application also proposes that the step of pre-scanning the knowledge entry and identifying and recording the specificity identifier existing in the knowledge entry further comprises:

[0022] Contextual semantic verification is performed on the extracted specificity identifier, and the specificity identifier passing the contextual semantic verification is retained to confirm the semantic specificity of the specificity identifier in the knowledge entry.

[0023] Further, the present application also proposes that the step of, in the asynchronous deep refining stage, performing deep semantic analysis on the knowledge entry whose processing state is the basic index completing the basic information writing, and writing the deep semantic analysis result into the advanced semantic index in an incremental manner comprises:

[0024] In the asynchronous deep refining stage, the knowledge entry whose processing state is the basic index completing the basic information writing is subjected to a corresponding specialized semantic analysis module according to the field or theme to which the knowledge entry belongs;

[0025] The specialized semantic analysis module is used to perform entity recognition and relationship extraction on the knowledge entry;

[0026] The entity and relationship are associated with the existing knowledge structure, and the existing knowledge structure is updated;

[0027] In the process of updating the existing knowledge structure, an active semantic association area in the knowledge structure is identified, and local semantic reconstruction is performed on the active semantic association area to obtain the deep semantic analysis result;

[0028] The deep semantic analysis result is written into the advanced semantic index in an incremental manner.

[0029] By introducing the specialized semantic analysis module and the local semantic reconstruction, deeper semantic understanding of the knowledge entry and knowledge structure updating are realized, especially the local reconstruction in the active semantic association area, which effectively solves the writing hotspot problem and improves the accuracy and real-time performance of the advanced semantic index.

[0030] Further, the present application also proposes that the step of associating the entity and relationship with the existing knowledge structure and updating the existing knowledge structure comprises:

[0031] The entity and relationship are associated with the existing knowledge structure;

[0032] In the process of associating the entity and relationship with the existing knowledge structure, semantic consistency verification is performed on the newly extracted entity or relationship and the corresponding entity or relationship in the existing knowledge structure to identify the semantic conflict or inconsistency existing between the newly extracted entity or relationship and the existing knowledge structure;

[0033] According to the semantic conflict or inconsistency, the source information or time information of the newly extracted entity or relationship is combined to evaluate the semantic conflict or inconsistency;

[0034] According to the evaluation result, the corresponding entity or relationship in the existing knowledge structure is updated.

[0035] By introducing the semantic consistency checking and conflict evaluation mechanism, the accuracy and consistency of the fusion of new knowledge and existing knowledge structure are ensured, the data quality decline caused by semantic conflict is effectively avoided, and the robustness of the knowledge structure is improved.

[0036] Further, the application also proposes that, in the process of updating the existing knowledge structure, the active semantic association area in the knowledge structure is identified, and the step of performing local semantic reconstruction for the active semantic association area includes:

[0037] In the process of updating the existing knowledge structure, a plurality of active semantic association areas in the knowledge structure are identified;

[0038] For the plurality of active semantic association areas, a plurality of local semantic reconstruction tasks are started;

[0039] Analyzing the plurality of local semantic reconstruction tasks, identifying whether there are shared entities or relationships and whether there are pre-dependence relationships between the plurality of local semantic reconstruction tasks, to determine the associativity between the plurality of local semantic reconstruction tasks;

[0040] According to the associativity, the plurality of local semantic reconstruction tasks are scheduled;

[0041] According to the scheduling result, the local semantic reconstruction is performed on the active semantic association area.

[0042] Further, the application also proposes that the step of scheduling the plurality of local semantic reconstruction tasks includes:

[0043] The task without dependence or with satisfied dependence is preferentially scheduled;

[0044] For parallel tasks involving shared entities or relationships, a resource locking mechanism is adopted to ensure that only one task modifies the shared entity or relationship at the same time.

[0045] By preferentially scheduling the task without dependence and adopting the resource locking mechanism, the execution efficiency and concurrent control of the local semantic reconstruction task are optimized, data conflict and deadlock are effectively avoided, and the stability of the system in the high concurrency scenario is further improved.

[0046] In a second aspect, the present application further provides a dynamic knowledge retrieval system data management system, which is applied in the index of knowledge items, and the index construction process of the knowledge items includes a fast basic index stage and an asynchronous deep refinement stage, the asynchronous deep refinement stage is executed asynchronously with the fast basic index stage, and the system includes:

[0047] a basic index processing module, configured to extract basic information of the knowledge items in the fast basic index stage, write the basic information into the basic index, and update the processing state of the knowledge items after the basic index completes the basic information writing;

[0048] a deep refinement processing module, configured to perform deep semantic analysis on the knowledge items with the processing state of the basic index completing the basic information writing in the asynchronous deep refinement stage, and write the deep semantic analysis result into the advanced semantic index in an incremental manner;

[0049] an automatic cleaning service module, configured to perform automatic cleaning service according to the residence time of the knowledge items in the temporary area and the processing state of the knowledge items, including exempting the knowledge items from cleaning when the residence time of the knowledge items exceeds a preset threshold and the processing state of the knowledge items is the basic index completing the basic information writing.

[0050] In summary, the dynamic knowledge retrieval system data management method and system provided by the present application effectively solve the problems of insufficient system carrying capacity and excessively long index construction time in the prior art during large-scale knowledge import, by dividing the index construction process of the knowledge items into the fast basic index stage and the asynchronous deep refinement stage, and executing the two stages asynchronously. The fast basic index stage can quickly extract the basic information of the knowledge items and write the basic information into the basic index, and timely update the processing state of the knowledge items, to ensure the rapid online of the knowledge. The asynchronous deep refinement stage performs deep semantic analysis on the knowledge items that have completed the basic index without affecting the efficiency of the basic index, and writes the result into the advanced semantic index in an incremental manner, to realize the fine management and retrieval of the knowledge. In addition, the present application further introduces intelligent automatic cleaning service, which judges according to the residence time of the knowledge items in the temporary area and the processing state of the knowledge items, and exempts the knowledge items from cleaning when the residence time of the knowledge items exceeds a preset threshold and the processing state of the knowledge items is the basic index completing the basic information writing. This mechanism effectively avoids the problem that valuable new knowledge being processed is mistakenly deleted by the automatic cleaning mechanism due to write blocking in the prior art, and significantly improves the data integrity and system stability. The present application overcomes the great pressure faced by the system under the centralized and high-intensity import requirement in the prior art, solves a series of technical problems such as write hotspots, write delay jitter, knowledge update task blocking, and data deletion errors, and significantly improves the real-time performance, accuracy, and data integrity of knowledge retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a dynamic knowledge retrieval system data management method provided by an embodiment of the present application.

[0052] Figure 2 A structural diagram of a dynamic knowledge retrieval system data management system provided by an embodiment of the present application.

[0053] Label explanation: 210, basic index processing module; 220, deep refining processing module; 230, automatic cleaning service module. DETAILED DESCRIPTION

[0054] The technical solutions in the present application will be described in detail below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0055] It should be noted that: similar labels and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0056] The traditional existing dynamic knowledge retrieval system faces severe challenges when processing large-scale and high-concurrency knowledge import. Especially when the business department starts a large-scale knowledge import plan to quickly respond to the market, it needs to digest and index a large amount of data, which causes the system carrying capacity to face great pressure. In addition, the complexity of newly imported knowledge, specialized terms and emerging semantic relationships make the indexing process more time-consuming and resource-intensive. The write hot spot problem causes the performance of the storage node to decline, and then causes the write delay jitter, which seriously interferes with the knowledge entry inverted index concurrent write control lock mechanism, causes a large number of knowledge update tasks to be blocked, and affects the real-time and accuracy of knowledge retrieval. The automatic cleaning mechanism inside the system may misjudge the valuable new knowledge being processed as "import failed zombie data", and incorrectly trigger the cleaning process, causing a large amount of new knowledge being indexed to be accidentally deleted, causing serious and irreparable damage to the data integrity of the system.

[0057] To this end, in a first aspect, see Figure 1The application provides a dynamic knowledge retrieval system data management method, which is applied to an index of knowledge entries, and an index construction process of the knowledge entries includes a fast basic index stage and an asynchronous deep refinement stage. The asynchronous deep refinement stage is executed asynchronously with the fast basic index stage. The method includes the following steps:

[0058] In the fast basic index stage, basic information of the knowledge entries is extracted, the basic information is written into a basic index, and after the basic index completes the basic information writing, a processing state of the knowledge entries is updated.

[0059] In the asynchronous deep refinement stage, deep semantic analysis is performed on the knowledge entries with the processing state of the basic index completing the basic information writing, and the deep semantic analysis result is written into a high-level semantic index in an incremental manner.

[0060] An automatic cleaning service is performed according to a residence time of the knowledge entries in a temporary area and the processing state of the knowledge entries, including: when the residence time of the knowledge entries exceeds a preset threshold value and the processing state of the knowledge entries is the basic index completing the basic information writing, the knowledge entries are exempted from cleaning.

[0061] The application realizes the parallelization and refinement of knowledge entry index construction by introducing a fast basic index stage and an asynchronous deep refinement stage, effectively alleviating the performance bottleneck of traditional systems when processing large-scale knowledge import. At the same time, intelligent automatic cleaning services are combined with the processing state and residence time of knowledge entries to avoid the accidental deletion of valuable data, significantly improving the real-time performance, accuracy and data integrity of the system. Knowledge entries refer to the basic information units that constitute a knowledge retrieval system, which can be documents, web pages, pictures, video metadata, or any data that can be indexed and retrieved. Basic information usually refers to the core metadata of knowledge entries, such as title, author, creation time, abstract, keywords, etc., which are sufficient to support fast preliminary retrieval. The basic index is an index structure that stores the basic information of knowledge entries, aiming to provide fast preliminary retrieval capabilities. The advanced semantic index is an index structure that stores the results of deep semantic analysis of knowledge entries, supporting more complex and accurate semantic retrieval. The processing state is a marker used to identify the current stage of knowledge entries in the index construction process, such as "to be processed", "basic index completed", "deep refinement in progress", etc. The staging area is a temporary storage area for knowledge entries before entering the index process, usually used for preprocessing and waiting for indexing. The preset threshold refers to the maximum time allowed for knowledge entries to stay in the staging area, exceeding which may trigger the cleaning mechanism. Deep semantic analysis refers to a deeper semantic understanding of knowledge entries, including entity recognition, relationship extraction, sentiment analysis, topic modeling, etc., aiming to reveal more rich and complex semantic information contained in knowledge entries. Incremental writing refers to writing only the newly added or modified deep semantic analysis results in the advanced semantic index, rather than rebuilding the index every time, to improve writing efficiency.

[0062] The core of the present application is to divide the index construction process of knowledge items into a fast basic index stage and an asynchronous deep refinement stage, and to ensure data integrity through intelligent automatic cleaning services. For example, when a new document is uploaded to the system, the metadata of the document can be automatically parsed, such as title, author, creation date, document type, etc. In addition, through text analysis technology, key summary information or core keywords can also be extracted from the document content as basic information. These basic information is then written into the basic index. The way to write into the basic index can be various, for example, the extracted basic information can be directly stored in a specific table of a relational database, or serialized into JSON format and stored in a document database. After the basic index completes the basic information writing, the processing status of the knowledge item will be updated. For example, the status of the knowledge item can be updated from "to be processed" to "basic index completed". For example, for a scientific paper item that has completed the basic index, natural language processing (NLP) technology can be used to perform entity recognition on it, identifying scientists, institutions, technical terms, and other entities mentioned in the paper; at the same time, relationship extraction is performed to identify the relationships between these entities, such as "Scientist A invented technology B". These deep semantic analysis results are then written to the advanced semantic index in an incremental manner. Incremental writing can use various strategies, such as maintaining a version control system to only record the changed part of each semantic analysis result and appending it to the advanced semantic index, or using a log-based update mechanism to write new semantic analysis results as log entries and periodically merge them into the main index.

[0063] In order to avoid valuable data being mistakenly deleted, the present application also introduces an automatic cleaning service. This service will make judgments based on the residence time of knowledge items in the staging area and the processing status of knowledge items. For example, the staging area can be periodically scanned for knowledge items, and if a knowledge item has been in the staging area for more than a predetermined threshold (e.g. 24 hours) and its processing status is still "to be processed" or "processing failed", the knowledge item can be marked as cleanable. However, when the residence time of a knowledge item exceeds the predetermined threshold, but its processing status is "basic index completed" and basic information writing is completed, the cleaning of the knowledge item is exempted. For example, a knowledge item has been in the staging area for 30 hours, exceeding the 24-hour predetermined threshold, but its processing status shows "basic index completed", which means that the item has completed the basic index and is waiting for asynchronous deep refinement. In this case, the system recognizes that the item is still in the normal processing flow, so it will not be cleaned, thus avoiding data deletion errors.

[0064] In the fast basic indexing stage, the basic information of the knowledge item can be quickly extracted and written into the basic index, and the processing status of the knowledge item is updated in time. The fast execution of this stage ensures that the newly imported knowledge can enter the preliminary searchable state at the fastest speed, significantly improving the real-time performance of knowledge publishing. For example, when the business department imports a large amount of new knowledge, the basic indexing stage can be quickly completed, so that users can immediately retrieve these new knowledge through basic information such as keywords, even if the deep semantic analysis has not been completed. Subsequently, the asynchronous deep refinement stage performs deep semantic analysis on the knowledge item with the processing status of basic index completion, and writes the results into the advanced semantic index in an incremental manner. Since this stage is executed asynchronously with the fast basic indexing stage, it does not block the basic indexing, thereby avoiding interference with the write hot spot and concurrent write control lock mechanism. For example, even if the deep semantic analysis requires a large amount of computing resources and time, it will not affect the subsequent basic indexing process of new knowledge, ensuring the overall throughput and response speed of the system. The incremental writing of the advanced semantic index further optimizes resource utilization and reduces unnecessary full index reconstruction overhead.

[0065] In addition, the automatic cleaning service mechanism of the present application realizes accurate protection of valuable data by comprehensively considering the residence time of the knowledge item in the staging area and its processing status. The traditional cleaning mechanism may be misdeleted due to index blocking, resulting in the knowledge item staying in the staging area for too long. In the present application, when the residence time of the knowledge item exceeds the preset threshold, but the processing status is basic index completion of basic information writing, the cleaning of the knowledge item is exempted, effectively avoiding the valuable new knowledge being processed being incorrectly identified as "import failed zombie data" and deleted, thereby ensuring the data integrity of the system and avoiding irreparable damage to the business.

[0066] Further, the step of writing the basic information into the basic index and updating the processing status of the knowledge item after the basic index completes the basic information writing comprises:

[0067] Pre-scanning the knowledge item to identify and record the specific identifiers existing in the knowledge item;

[0068] Writing the basic information into the basic index;

[0069] After the basic information is written into the basic index, performing functional verification on the specific identifiers contained in the basic index according to the recorded specific identifiers to obtain a functional verification result;

[0070] According to the functional verification result, updating the processing status of the knowledge item.

[0071] Specifically, the pre-scanning of the knowledge item to identify and record the presence of specific identifiers in the knowledge item refers to a preliminary inspection of the knowledge item before writing the basic information of the knowledge item into the basic index. This pre-scanning aims to identify and extract information fragments with specific structure, format or semantic importance in the knowledge item, such as product model, document number, author ID, specific terms or keywords, etc. These specific identifiers are considered to be key elements of the integrity and functionality of the knowledge item, and their presence or absence or correct format directly affects the accuracy of subsequent processing. The identified identifiers are temporarily recorded for subsequent verification. Among them, writing the basic information into the basic index refers to storing the core metadata of the knowledge item, such as title, abstract, creation time, source, etc., into the basic index. This step is the basis for building a knowledge index, ensuring that the knowledge item can be quickly searched and located.

[0072] In practical applications, after the basic information is written into the basic index, the functional verification of the specific identifiers contained in the basic index according to the recorded specific identifiers to obtain the functional verification result refers to verifying the specific identifiers contained in the corresponding knowledge item in the basic index using the specific identifiers recorded in the pre-scanning stage. The purpose of functional verification is to confirm whether these key identifiers are correctly and completely written into the basic index, and whether their format and content meet the expected functional requirements. For example, it can be checked whether a certain number meets a certain regular expression, or whether a certain field is non-empty. The verification result will indicate the functional status of these identifiers. According to the functional verification result, updating the processing status of the knowledge item refers to adjusting the processing status of the knowledge item accordingly according to the feedback of the functional verification. If the verification result shows that all key identifiers have been correctly written and meet the functional requirements, the knowledge item can be marked as the basic index completing the writing of the basic information, allowing it to enter the subsequent asynchronous deep refinement stage. On the contrary, if the verification fails, the knowledge item may be marked as other states, such as the basic index functional defect, in order to intervene manually or reprocess.

[0073] The present application effectively solves the problem of missing or incorrect key information in the basic index stage by introducing a pre-scanning and functional verification mechanism. By pre-scanning the knowledge entry, important specific identifiers are identified and recorded before the basic information is written, allowing the key structure and content requirements of the knowledge entry to be grasped in advance. After the basic information is written into the basic index, the pre-recorded identifiers are used to perform functional verification on the written basic index, ensuring the integrity and correctness of the specific identifiers contained in the basic index, and avoiding problems caused by incomplete or format errors. Therefore, only knowledge entries that pass the functional verification will be updated to the basic index to complete the basic information writing state, ensuring that the knowledge entries entering the deep refinement stage have high-quality basic data. The traditional processing method may directly update the state after the basic information is written, which may cause incomplete or incorrect data to be passed to the subsequent stage. However, the present application adds a pre-scanning and functional verification link before updating the processing state, ensuring the integrity and functionality of the key specific identifiers in the basic index, thereby avoiding the deviation or failure of subsequent deep semantic analysis caused by defects in the basic data. This not only improves the accuracy of data processing, but also reduces the maintenance cost and system error rate caused by data quality problems.

[0074] Further, according to the functional verification result, the step of updating the processing state of the knowledge entry comprises:

[0075] When the functional verification result shows that the specific identifier is completely contained, the processing state of the knowledge entry is updated to the basic index to complete the basic information writing;

[0076] When the functional verification result shows that the specific identifier is not completely contained, the processing state of the knowledge entry is updated to the basic index with functional defects.

[0077] Specifically, the functional verification result is an output of evaluating the integrity of the specific identifier in the basic index. The specific identifier can be understood as key information with unique semantic or structural characteristics in the knowledge item, such as specific entity names, professional terms, codes or reference links, etc. When the functional verification result shows that the specific identifier is completely contained, it means that in the process of building the basic index, all expected or identified specific identifiers have been correctly extracted and written into the basic index, and their structure and content meet the preset functional requirements, and the processing state of the knowledge item is updated to “basic index completes basic information writing”, which clearly indicates that the knowledge item has successfully completed the basic index stage and is ready for the subsequent asynchronous deep refinement stage. On the contrary, when the functional verification result shows that the specific identifier is not completely contained, for example, some key specific identifiers are missing in the basic index, or the structure or association of these identifiers is defective, it indicates that the basic index is functionally incomplete, and the processing state of the knowledge item is updated to “basic index functional defect”, which is intended to clearly indicate that the knowledge item in the basic index stage does not fully meet the functional requirements, and may need further manual intervention, reprocessing or isolation, to avoid passing incomplete or incorrect data to the subsequent deep refinement stage, thereby ensuring the data quality of the entire knowledge retrieval system.

[0078] The present application realizes fine management of the quality of knowledge items in the basic index stage by introducing two different processing states. When the functional verification result confirms that the specific identifier is completely contained, the knowledge item is marked as “basic index completes basic information writing”, which indicates that its basic index has reached the standard of usability and can smoothly enter the asynchronous deep refinement stage. On the contrary, when the functional verification result shows that the specific identifier is not completely contained, the knowledge item is marked as “basic index functional defect”, which can identify and isolate those knowledge items with defects in the basic index stage, ensuring that only high-quality basic index data can enter the subsequent deep processing flow, thereby avoiding subsequent processing errors or resource waste due to basic data quality problems. The present application can more accurately evaluate and manage the quality of the basic index of knowledge items, and by distinguishing between the two states of “basic index completes basic information writing” and “basic index functional defect”, it can clearly identify which knowledge items have the conditions for deep refinement and which need further attention or correction. This fine state management helps to improve the data quality and processing efficiency of the entire knowledge retrieval system, reduces downstream problems caused by incomplete or incorrect basic index, and improves the accuracy and reliability of knowledge retrieval.

[0079] Further, the step of pre-scanning the knowledge item to identify and record the specific identifier existing in the knowledge item includes:

[0080] The step of pre-scanning the knowledge entry to identify and record the specificity identifiers present in the knowledge entry includes:

[0081] Selecting one or more matching rules from a pre-set rule library based on the content features of the knowledge entry or the source information of the knowledge entry;

[0082] Text scanning the knowledge entry using the selected matching rules to identify and extract text segments that match the matching rules as specificity identifiers.

[0083] The content features of the knowledge entry can refer to the internal attributes such as the text structure, keyword distribution, and topic classification of the knowledge entry. The source information of the knowledge entry can refer to the creator, publishing platform, and timestamp of the knowledge entry. The pre-set rule library can store various rules for identifying different types of specificity identifiers, such as regular expressions, keyword lists, and named entity recognition models, which can be customized and updated according to different application scenarios and knowledge fields. The selection of matching rules is based on the specific characteristics of the knowledge entry. For example, if the knowledge entry comes from a certain technical forum, the matching rules for the forum content format and common terms can be selected; if the knowledge entry involves legal documents, rules for identifying legal clause numbers, case numbers, and other specificity identifiers can be selected. By text scanning the knowledge entry, text segments that match the selected matching rules can be efficiently identified and extracted. These text segments are confirmed as specificity identifiers, which are usually unique or highly distinctive and can serve as key identification information for the knowledge entry.

[0084] The present application can select the most suitable matching rules from the rule library based on the analysis of the content features or source information of the knowledge entry. This rule selection mechanism based on features or sources ensures the accuracy and efficiency of the identification process. By text scanning the knowledge entry using these selected matching rules, text segments with specific formats or semantics can be accurately identified and extracted, thereby being confirmed as specificity identifiers. This step-by-step and strategic identification process enables the system to effectively capture unique identification information that is crucial for subsequent indexing and management from a large amount of knowledge entries. It can achieve accurate identification and extraction of specificity identifiers in knowledge entries, avoid blind scanning and unnecessary computational overhead, improve the efficiency and accuracy of pre-scanning by selecting matching rules based on the content features or source information of the knowledge entry. The identified specificity identifiers provide reliable basis for subsequent basic indexing functionality verification, ensuring the quality and integrity of the basic index, and improving the efficiency and reliability of data management.

[0085] Further, the step of pre-scanning the knowledge entry to identify and record the specificity identifiers present in the knowledge entry further comprises:

[0086] Contextual semantic verification of the extracted specificity identifiers is performed to retain the specificity identifiers that pass the contextual semantic verification, to confirm the semantic specificity of the specificity identifiers in the knowledge entry.

[0087] Specifically, the contextual semantic verification refers to in-depth analysis of the initially identified specificity identifiers in combination with their surrounding textual environment in the knowledge entry, to determine whether the identifiers truly have their intended semantic meanings and importance in the specific context. For example, a word may be a specificity identifier in a general context, but in a certain knowledge entry, it may just be a common word without specificity. Ensuring that the identified identifiers not only meet the formal rules, but also are closely related to the core content of the knowledge entry in terms of semantics. Semantic specificity can be understood as the uniqueness, importance or representativeness of a certain identifier in a specific knowledge entry or field. An identifier with high semantic specificity can accurately point to a specific concept, entity or event, and is not easily confused with other concepts in the current context. Confirming semantic specificity aims to filter out those text fragments that are identified but have little actual meaning, thereby improving the quality of the identifiers. In practical applications, contextual semantic verification of the extracted specificity identifiers can be performed using various technical means. For example, natural language processing techniques such as word vector models, topic models or pre-trained language models (such as BERT, GPT, etc.) can be used to analyze the semantic similarity or relevance of the identifiers and their context. It is also possible to construct a domain knowledge graph, match the extracted identifiers with entities in the knowledge graph, and evaluate their association strength and centrality in the graph. It is also possible to combine human annotation or expert rules to perform more detailed semantic verification on specific types of identifiers.

[0088] The application can effectively solve the problem of inaccurate semantics or insufficient specificity of identifiers caused by relying only on matching rules by performing context semantic verification on the preliminarily extracted specific identifiers. After a text segment is preliminarily identified as a specific identifier, the specific context of the text segment in the knowledge entry is further analyzed. For example, if a word is identified under the matching rules, but the surrounding text content shows that it is only a general descriptive word, rather than a key identifier with unique meaning, the word will be determined to be insufficiently semantically specific and filtered out through context semantic verification. Compared with relying only on preset matching rules for identification, the application introduces context semantic verification to effectively filter out text segments that meet the rules in form but lack semantic specificity, thereby reducing the generation of invalid or noise identifiers. The subsequent basic index functional verification will be more accurate, and the input data quality of deep semantic analysis is also improved, making the index construction process of the knowledge entry more efficient and accurate, and providing users with higher-quality knowledge retrieval services.

[0089] Further, in the asynchronous deep refinement stage, the step of performing deep semantic analysis on the knowledge entry in the processing state of the basic index completing the basic information writing and writing the deep semantic analysis result into the advanced semantic index in an incremental manner comprises:

[0090] In the asynchronous deep refinement stage, the knowledge entry in the processing state of the basic index completing the basic information writing is called according to the corresponding specialized semantic analysis module according to the field or theme to which the knowledge entry belongs;

[0091] The specialized semantic analysis module is used to perform entity recognition and relationship extraction on the knowledge entry;

[0092] The entity and relationship are associated with the existing knowledge structure, and the existing knowledge structure is updated;

[0093] In the process of updating the existing knowledge structure, the active semantic association area in the knowledge structure is identified, and local semantic reconstruction is performed on the active semantic association area to obtain the deep semantic analysis result;

[0094] The deep semantic analysis result is written into the advanced semantic index in an incremental manner.

[0095] Specifically, in the asynchronous deep refining stage, after the basic index is marked as complete for writing the basic information, the specific field or topic to which the knowledge entry belongs is determined according to the content characteristics of the knowledge entry, such as its keywords, classification tags or source metadata. For example, if the knowledge entry relates to medical literature, a specialized semantic analysis module for the medical field will be called; if it relates to financial reports, a specialized semantic analysis module for the financial field will be called, aiming to ensure the professionalism and accuracy of semantic analysis and avoid understanding biases that may occur in specific fields with general models. Among them, the entity recognition and relationship extraction of the knowledge entry by the specialized semantic analysis module means that through a pre-trained natural language processing model optimized for a specific field, entities with specific meanings (such as names, place names, organizations, professional terms, etc.) and semantic relationships between these entities (such as "A is the author of B", "C belongs to D organization", etc.) are automatically identified from the text content of the knowledge entry, converting unstructured text information into structured semantic data, laying the foundation for subsequent knowledge graph construction and updating.

[0096] In practical applications, associating entities and relationships with existing knowledge structures and updating existing knowledge structures can be understood as comparing and fusing new extracted structured semantic data with existing knowledge graphs or ontologies in the system. Newly identified entities and relationships will be tried to be mapped to corresponding nodes and edges in the existing knowledge structure, or added as new nodes and edges to the knowledge structure, this process aims to maintain the consistency and integrity of the knowledge structure and continuously expand its content. In the process of updating the existing knowledge structure, identify the active semantic association area in the knowledge structure, and for the active semantic association area, perform local semantic reconstruction to obtain the deep semantic analysis result. The active semantic association area refers to a local subgraph or semantic network in the knowledge structure that has close contact with newly added or updated entities and relationships. For example, if a new entity has multiple strong association relationships with a certain concept node in the existing knowledge structure, then the concept node and the entities and relationships within a certain range around it constitute an active semantic association area. Performing local semantic reconstruction on these areas means reanalyzing, verifying and optimizing only the affected local knowledge structure, rather than performing global reconstruction on the entire vast knowledge structure, which can improve the efficiency of knowledge structure updating, reduce unnecessary computational overhead, and ensure the accuracy of local updating. Writing the deep semantic analysis result to the advanced semantic index in an incremental manner means that the structured data reflecting the deep semantics of the knowledge entry obtained after local semantic reconstruction is efficiently stored in the advanced semantic index in the form of addition or update. This incremental writing method avoids rebuilding the entire index every time it is updated, significantly improving the speed of index updating and the response ability of the system.

[0097] The application introduces a specialized semantic analysis module, so that the deep semantic analysis can be customized according to the specific field or theme to which the knowledge item belongs, thereby effectively solving the problem of insufficient accuracy that may occur when a general semantic analysis is used to process diversified knowledge content. Through entity recognition and relationship extraction, the unstructured text is converted into structured semantic data, laying a foundation for subsequent accurate association with the existing knowledge structure. In the process of associating the newly extracted entities and relationships with the existing knowledge structure and updating, the application innovatively identifies active semantic association areas in the knowledge structure, and only these local areas are subjected to semantic reconstruction. This local reconstruction mechanism avoids global and time-consuming recalculation of the entire large knowledge structure, significantly improving the efficiency of knowledge structure updating. At the same time, by focusing on the affected local areas, the fusion of new and old knowledge can be processed more finely, reducing potential semantic conflicts and inconsistencies, and ensuring the accuracy and consistency of knowledge structure updating. The deep semantic analysis results obtained through local semantic reconstruction are written into the advanced semantic index in an incremental manner, ensuring the efficiency of index updating, enabling a quick response to dynamic changes in knowledge, and maintaining the timeliness and accuracy of the advanced semantic index.

[0098] Further, the step of associating the entities and relationships with the existing knowledge structure and updating the existing knowledge structure comprises:

[0099] associating the entities and relationships with the existing knowledge structure;

[0100] In the process of associating the entities and relationships with the existing knowledge structure, the semantic consistency of the newly extracted entities or relationships and the corresponding entities or relationships in the existing knowledge structure is checked, and the semantic conflicts or inconsistencies between the newly extracted entities or relationships and the existing knowledge structure are identified;

[0101] According to the semantic conflicts or inconsistencies, the source information or time information of the newly extracted entities or relationships is combined to evaluate the semantic conflicts or inconsistencies;

[0102] According to the evaluation results, the corresponding entities or relationships in the existing knowledge structure are updated.

[0103] Specifically, the semantic consistency verification refers to comparing the newly extracted entity or relationship with the possible corresponding entity or relationship in the existing knowledge structure to determine whether they are consistent in semantics. For example, methods such as ontology matching, word vector similarity calculation, or rule matching can be used for verification to preliminarily screen out semantic information that may have conflicts or inconsistencies. Among them, identifying semantic conflicts or inconsistencies between the newly extracted entity or relationship and the existing knowledge structure can be understood as further specifying the specific type and degree of conflict on the basis of semantic consistency verification. For example, if the attribute value of the newly extracted entity is contradictory to the attribute value in the existing knowledge structure, or the relationship type of the newly extracted entity is inconsistent with the relationship definition in the existing knowledge structure, it can be identified as a semantic conflict or inconsistency. In practical applications, the evaluation of semantic conflicts or inconsistencies is specifically to combine the source information or time information of the newly extracted entity or relationship to judge the severity and credibility of the conflict or inconsistency. For example, if the newly extracted information comes from an authoritative and up-to-date data source, its credibility may be higher than that of the old information in the existing knowledge structure; on the contrary, if the source is unreliable, its credibility is lower. Time information can be used to judge the timeliness of the information, for example, newer information may more accurately reflect the current state and provide a basis for subsequent update decisions. Updating the corresponding entity or relationship in the existing knowledge structure according to the evaluation result means that the update operation is no longer unconditional coverage, but an intelligent decision based on the evaluation result. For example, if the evaluation result shows that the new information has higher credibility or timeliness, the existing information can be replaced; if the conflict is serious and the new information is not highly credible, the update can be selected or marked as pending manual processing.

[0104] The application effectively solves the semantic conflicts and inconsistencies that may occur when associating entities and relationships with the existing knowledge structure and updating the existing knowledge structure by introducing the mechanism of semantic consistency checking, conflict identification, evaluation and conditional updating. By performing semantic consistency checking on the newly extracted entities or relationships, potential semantic differences can be initially discovered. Once a semantic conflict or inconsistency is identified, it will be further evaluated in combination with the source information or time information of the newly extracted entities or relationships, enabling intelligent judgment of the reliability and timeliness of the new and old information. According to the evaluation results, the corresponding entities or relationships in the existing knowledge structure are selectively updated to avoid knowledge structure degradation caused by blind updating, thereby ensuring the accuracy and consistency of the knowledge structure during the dynamic updating process. Compared to simply associating and updating, the application introduces the mechanisms of semantic consistency checking and conflict evaluation, which have stronger judgment ability when handling conflicts between new and old information, effectively avoiding the pollution of existing knowledge structure by false or low-quality information, and combining source information or time information for evaluation to make the updating decision more intelligent and refined, ensuring that the knowledge structure can continuously maintain high accuracy and timeliness, thereby providing a more solid and reliable data foundation for subsequent knowledge retrieval, reasoning and application.

[0105] Further, in the process of updating the existing knowledge structure, an active semantic association area in the knowledge structure is identified, and for the active semantic association area, a step of performing local semantic reconstruction includes:

[0106] In the process of updating the existing knowledge structure, a plurality of active semantic association areas in the knowledge structure are identified;

[0107] For the plurality of active semantic association areas, a plurality of local semantic reconstruction tasks are started;

[0108] Analyzing the plurality of local semantic reconstruction tasks, identifying whether there are shared entities or relationships and whether there are pre-dependence relationships between the plurality of local semantic reconstruction tasks to determine the associativity between the plurality of local semantic reconstruction tasks;

[0109] According to the associativity, the plurality of local semantic reconstruction tasks are scheduled;

[0110] According to the scheduling result, the active semantic association area is executed for local semantic reconstruction.

[0111] Specifically, in the process of updating the existing knowledge structure, multiple active semantic association regions in the knowledge structure can be identified, which are subgraphs or node sets in the knowledge structure with high semantic association degree and recent changes or expected changes. For example, when a knowledge entry is updated, multiple semantic regions directly or indirectly associated with it can be considered as active regions. For the identified multiple active semantic association regions, an independent local semantic reconstruction task is started for each region. Each local semantic reconstruction task aims to fine-tune the respective corresponding active semantic association region, such as recalculating the relationship strength between entities, verifying semantic consistency, or introducing new semantic associations. After starting these tasks, in-depth analysis of these local semantic reconstruction tasks is performed to identify whether there are shared entities or relationships between these tasks, i.e., whether multiple tasks need to access or modify the same part of the knowledge structure. At the same time, it is also identified whether there is a pre-dependence relationship between the tasks, for example, the completion of a task is a prerequisite for the start of another task. By identifying these shared entities or relationships and pre-dependence relationships, the association between these local semantic reconstruction tasks can be determined comprehensively. In practical applications, according to the determined association, the local semantic reconstruction tasks are scheduled, and the scheduling strategy can be dynamically adjusted according to the priority of the task, resource demand, dependence relationship and shared resource situation, to optimize the overall execution efficiency and resource utilization. For example, tasks without dependence can be executed in parallel, while tasks with dependence need to wait for the completion of their pre-task before starting. According to the scheduling result, local semantic reconstruction is performed on each active semantic association region, and each task will be performed according to the predetermined order and resource allocation, ensuring that the updating process of the knowledge structure is both efficient and accurate.

[0112] The application effectively solves the efficiency and consistency problems that may occur when processing multiple active semantic association regions by introducing a correlation analysis and scheduling mechanism for multiple local semantic reconstruction tasks. When there are multiple active semantic association regions that need to be reconstructed in the knowledge structure, they are no longer simply processed independently or sequentially. Instead, the internal relationships between these reconstruction tasks are first identified, including whether they share data resources (shared entities or relationships) and whether there is a logical sequence (predecessor dependency relationship) between them. By scheduling tasks according to their correlation, such as prioritizing the execution of independent tasks, parallel processing of independent tasks, or synchronous control of tasks that share resources, the application can avoid resource contention, deadlocks, or data inconsistencies caused by blind parallelism or improper sequential execution, thereby ensuring the stability and correctness of knowledge structure updates. By analyzing and intelligently scheduling multiple local semantic reconstruction tasks, the application can effectively avoid resource conflicts and unnecessary waiting, maximize parallel processing capabilities, and thus shorten the overall time of knowledge structure updates. By strictly following the dependency relationships between tasks and coordinating access to shared resources, the application ensures the consistency and integrity of knowledge structure data in complex update scenarios, avoids errors or inaccurate semantic associations introduced by improper concurrent operations, and thus improves the quality and reliability of knowledge retrieval.

[0113] Further, the step of scheduling multiple local semantic reconstruction tasks includes:

[0114] Prioritize scheduling tasks that have no dependencies or whose dependencies have been satisfied;

[0115] For parallel tasks involving shared entities or relationships, a resource locking mechanism is used to ensure that only one task modifies the shared entity or relationship at the same time.

[0116] Specifically, the priority scheduling of tasks without dependency or dependency satisfied refers to that when starting multiple local semantic restructuring tasks, the scheduler first identifies those tasks that do not depend on the results of other tasks, or all the pre-dependence tasks of which have been completed, and these tasks can be executed immediately, thereby improving the overall processing efficiency. For example, the tasks with in-degree of zero can be identified by constructing a task dependency graph, or the dependency state is dynamically updated during task execution, and when the dependency conditions of a certain task are all met, it is added to the executable queue. Among them, for parallel tasks involving shared entities or relationships, the resource locking mechanism can be understood as follows: when multiple local semantic restructuring tasks need to access or modify the same entity or relationship at the same time, in order to avoid data conflicts and inconsistencies, a synchronization control mechanism is introduced. Specifically, when a task starts to modify a shared entity or relationship, the entity or relationship will be "locked", and other tasks that try to access or modify the entity or relationship will be temporarily blocked until the current task is completed and the lock is released, ensuring the atomicity and consistency of data operations and preventing data damage or logical errors caused by concurrent modification. In practical applications, the resource locking mechanism can be implemented by various technologies, such as mutex, read-write lock or semaphore, and the appropriate locking strategy is selected according to the access mode and concurrency requirements of shared resources.

[0117] The present application effectively solves the efficiency and data consistency problems that may occur when multiple local semantic restructuring tasks are executed in parallel by introducing task priority scheduling and resource locking mechanism. The priority scheduling of tasks without dependency or dependency satisfied can maximize the parallelism and avoid resource waste caused by unnecessary waiting, thereby speeding up the entire deep refinement process. At the same time, for parallel tasks involving shared entities or relationships, the resource locking mechanism is adopted to ensure that only one task can modify the shared data at any given time. This fundamentally avoids data conflicts, dirty reads or lost updates that may be caused by concurrent modification, ensures the accuracy and reliability of knowledge structure updating, and enables complex local semantic restructuring tasks to be executed stably and efficiently in a high-concurrency environment. The present application effectively reduces the waiting time between tasks by prioritizing tasks without dependency or dependency satisfied, optimizes the task execution process, and thereby speeds up the updating speed of knowledge structure. The introduction of the resource locking mechanism completely eliminates the data conflicts and inconsistency problems that may be caused by parallel tasks when modifying shared entities or relationships, ensures the accuracy and reliability of knowledge structure updating, not only improves the ability of the system to process large-scale knowledge entries, but also enhances the accuracy and timeliness of knowledge retrieval results, providing users with better knowledge services.

[0118] In a second aspect, referring to Figure 2The application also discloses a dynamic knowledge retrieval system data management system, which is applied to an index of knowledge entries, and an index construction process of the knowledge entries comprises a fast basic index stage and an asynchronous deep refinement stage, the asynchronous deep refinement stage is executed asynchronously with the fast basic index stage, and the system comprises:

[0119] A basic index processing module 210 is used for extracting basic information of the knowledge entries in the fast basic index stage, writing the basic information into a basic index, and updating a processing state of the knowledge entries after the basic index completes the writing of the basic information;

[0120] A deep refinement processing module 220 is used for performing deep semantic analysis on the knowledge entries with the processing state of the basic index completing the writing of the basic information in the asynchronous deep refinement stage, and writing a deep semantic analysis result into an advanced semantic index in an incremental manner;

[0121] An automatic cleaning service module 230 is used for performing automatic cleaning service according to a stay time of the knowledge entries in a temporary area and the processing state of the knowledge entries, and the automatic cleaning service comprises the following: when the stay time of the knowledge entries exceeds a preset threshold value and the processing state of the knowledge entries is the basic index completing the writing of the basic information, the knowledge entries are exempted from cleaning.

[0122] The dynamic knowledge retrieval system data management system can realize parallelization and fine processing of index construction of the knowledge entries, ensure that the knowledge entries can rapidly enter a preliminary searchable state, improve the accuracy of retrieval by performing deep semantic analysis and constructing an advanced semantic index, intelligently manage temporary area data according to the processing state and the stay time of the knowledge entries, avoid valuable data from being mistakenly deleted, effectively alleviate the performance bottleneck of a traditional system in processing large-scale knowledge import, and significantly improve the real-time performance, accuracy and data integrity of the system. The application can identify and exempt valuable knowledge entries that have completed basic index but are still waiting for deep refinement from cleaning, guarantee the data integrity of the system, and avoid losses caused by mistakenly deleted data to the business.

[0123] The above merely describes the embodiments of the application and is not used to limit the protection scope of the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A data management method for a dynamic knowledge retrieval system, applied in the indexing of knowledge entries, characterized in that, The indexing process for the knowledge entries includes a fast basic indexing stage and an asynchronous deep refinement stage. The asynchronous deep refinement stage is executed asynchronously with the fast basic indexing stage. The method includes the following steps: In the rapid basic indexing stage, the basic information of the knowledge entry is extracted, the basic information is written into the basic index, and the processing status of the knowledge entry is updated after the basic index has completed writing the basic information. In the asynchronous deep refinement stage, deep semantic analysis is performed on the knowledge entries whose processing status has completed the writing of basic information on the basic index, and the results of the deep semantic analysis are written to the advanced semantic index in an incremental manner. An automatic cleanup service is performed based on the dwell time of the knowledge entry in the temporary storage area and the processing status of the knowledge entry, including: when the dwell time of the knowledge entry exceeds a preset threshold and the processing status of the knowledge entry is that the basic information has been written to the basic index, the cleanup of the knowledge entry is waived; when the dwell time of the knowledge entry exceeds the preset threshold and the processing status of the knowledge entry is pending or processing failed, the knowledge entry is cleaned up.

2. The data management method for a dynamic knowledge retrieval system according to claim 1, characterized in that, The steps of writing the basic information into the basic index and updating the processing status of the knowledge entry after the basic index has completed writing the basic information include: The knowledge entries are pre-scanned to identify and record the specific identifiers present in the knowledge entries; Write the basic information into the basic index; After the basic information is written into the basic index, the specific identifiers contained in the basic index are functionally verified according to the specific identifiers of the records to obtain the functional verification results. Based on the functional verification results, update the processing status of the knowledge entry.

3. The data management method for a dynamic knowledge retrieval system according to claim 2, characterized in that, The step of updating the processing status of the knowledge entry based on the functional verification result includes: When the functional verification result shows that the specific identifier is fully included, the processing status of the knowledge entry is updated to complete the basic information writing based on the basic index; When the functionality verification result shows that the specific identifier is not fully included, the processing status of the knowledge entry is updated to basic index functionality incompleteness.

4. The data management method for a dynamic knowledge retrieval system according to claim 2, characterized in that, The step of pre-scanning the knowledge entries, identifying and recording the specific identifiers present in the knowledge entries includes: Based on the content characteristics of the knowledge entry or the source information of the knowledge entry, select one or more matching rules from a preset recognition rule base; The selected matching rules are used to scan the text of the knowledge entries, and text fragments that match the matching rules are identified and extracted as specific identifiers.

5. The data management method for a dynamic knowledge retrieval system according to claim 4, characterized in that, The step of pre-scanning the knowledge entries, identifying and recording the specific identifiers present in the knowledge entries further includes: The extracted specific identifiers are subjected to contextual semantic verification, and the specific identifiers that pass the contextual semantic verification are retained to confirm the semantic specificity of the specific identifiers in the knowledge entry.

6. The data management method for a dynamic knowledge retrieval system according to claim 1, characterized in that, The step of performing deep semantic analysis on the knowledge entries whose processing state has completed the basic information writing of the base index during the asynchronous deep refinement stage, and incrementally writing the deep semantic analysis results into the advanced semantic index includes: In the asynchronous deep refinement stage, for knowledge entries whose processing status has been used to write basic information based on the basic index, the corresponding specialized semantic analysis module is invoked according to the domain or topic to which the knowledge entry belongs. The specialized semantic analysis module is used to perform entity recognition and relation extraction on the knowledge entries; Associate entities and relationships with the existing knowledge structure, and update the existing knowledge structure; In the process of updating the existing knowledge structure, active semantic association regions in the knowledge structure are identified, and local semantic reconstruction is performed on the active semantic association regions to obtain deep semantic analysis results. The deep semantic analysis results are written to the advanced semantic index incrementally.

7. The data management method for a dynamic knowledge retrieval system according to claim 6, characterized in that, The steps of associating entities and relationships with the existing knowledge structure and updating the existing knowledge structure include: Associate entities and relationships with the existing knowledge structure; In the process of associating the entities and relations with the existing knowledge structure, a semantic consistency check is performed on the newly extracted entities or relations with the corresponding entities or relations in the existing knowledge structure to identify semantic conflicts or inconsistencies between the newly extracted entities or relations and the existing knowledge structure. Based on the semantic conflict or inconsistency, and in conjunction with the source or time information of the newly extracted entity or relationship, the semantic conflict or inconsistency is evaluated. Based on the evaluation results, update the corresponding entities or relationships in the existing knowledge structure.

8. The data management method for a dynamic knowledge retrieval system according to claim 6, characterized in that, The step of identifying active semantic association regions in the knowledge structure and performing local semantic reconstruction for the active semantic association regions during the process of updating the existing knowledge structure includes: During the process of updating the existing knowledge structure, multiple active semantically related regions in the knowledge structure are identified; For the aforementioned multiple active semantically associated regions, multiple local semantic reconstruction tasks are initiated; Analyze the multiple local semantic reconstruction tasks to identify whether there are shared entities or relationships among the multiple local semantic reconstruction tasks, and whether there are pre-existing dependencies, so as to determine the correlation between the multiple local semantic reconstruction tasks; Based on the aforementioned correlation, the multiple local semantic reconstruction tasks are scheduled; According to the scheduling results, local semantic reconstruction is performed on the active semantic association region.

9. A data management method for a dynamic knowledge retrieval system according to claim 8, characterized in that, The step of scheduling the plurality of local semantic reconstruction tasks includes: Prioritize scheduling tasks with no dependencies or whose dependencies are already satisfied; For parallel tasks involving shared entities or relationships, a resource locking mechanism is employed to ensure that only one task modifies the shared entity or relationship at any given time.

10. A dynamic knowledge retrieval system data management system, applied in the indexing of knowledge entries, characterized in that, The indexing process for the knowledge entries includes a fast basic indexing stage and an asynchronous deep refinement stage. The asynchronous deep refinement stage is executed asynchronously with the fast basic indexing stage. The system includes: The basic index processing module is used to extract the basic information of the knowledge entry in the fast basic indexing stage, write the basic information into the basic index, and update the processing status of the knowledge entry after the basic index has completed writing the basic information. The deep refinement processing module is used to perform deep semantic analysis on the knowledge entries whose processing status has been completed by writing basic information on the basic index during the asynchronous deep refinement stage, and to write the deep semantic analysis results into the advanced semantic index in an incremental manner. An automatic cleanup service module is used to perform automatic cleanup services based on the dwell time of the knowledge entry in the temporary storage area and the processing status of the knowledge entry. This includes: when the dwell time of the knowledge entry exceeds a preset threshold and the processing status of the knowledge entry is that the basic information has been written to the basic index, the cleanup of the knowledge entry is waived; when the dwell time of the knowledge entry exceeds the preset threshold and the processing status of the knowledge entry is pending or processing failed, the knowledge entry is cleaned.

Images