A big data knowledge construction and management system

Abstract

The application relates to the technical field of data processing, and particularly discloses a big data knowledge construction and management system. The system comprises an incremental data sensing and preprocessing module, a dynamic index hierarchical construction module, a consistency coordination and transaction management module, and a query routing and result fusion module. The application decouples the heavy operation of full index reconstruction into periodic asynchronous update of a base index and real-time additional writing of an incremental index by introducing a dynamic index hierarchical architecture. The system only needs to maintain a light incremental index block in most time, thereby completely avoiding query service interruption and performance sharp drop caused by data update. The knowledge graph can continuously integrate new data with near real-time delay, and simultaneously provide stable, efficient and data consistent query services to the outside, thereby meeting the strict requirements of high timeliness scenes such as financial risk control and public opinion monitoring.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a big data knowledge construction and management system. Background Technology

[0002] As a key infrastructure of the information age, big data technology's core value lies in extracting structured knowledge from massive, multi-source, and heterogeneous data to support intelligent decision-making and analysis. Knowledge graphs, as a semantic network, can represent entities and their relationships in a graph structure, and are an important technological means to realize the knowledge-based and intelligent applications of big data.

[0003] Big data knowledge construction and management systems aim to automatically extract knowledge from raw data, build and maintain large-scale knowledge graphs, and provide efficient knowledge query, reasoning, and application services. Such systems typically involve multiple technical aspects, including data acquisition, information extraction, knowledge fusion, and graph storage and retrieval.

[0004] In existing technologies, the construction and updating process of knowledge graphs faces significant bottlenecks. When new incremental data needs to be integrated into an existing knowledge graph, traditional systems often need to completely rebuild the entire graph's index to maintain query efficiency and data consistency. This complete reconstruction process is computationally expensive and time-consuming, leading to a sharp increase in the average response latency of the knowledge graph during the update period, and the system is unable to provide stable real-time query services.

[0005] In applications such as financial risk control and public opinion monitoring, where timeliness is extremely important, service interruptions and delays caused by data updates severely hinder real-time analysis and decision-making processes based on the latest knowledge, becoming a key challenge restricting the practicality of knowledge management systems. Summary of the Invention

[0006] The purpose of this invention is to provide a big data knowledge construction and management system to solve the technical contradiction in the prior art where incremental updates of knowledge graphs require full index reconstruction, resulting in system service interruption, a surge in query latency, and an inability to meet real-time application requirements.

[0007] This invention provides a big data knowledge construction and management system, which includes an incremental data perception and preprocessing module, a dynamic index hierarchical construction module, a consistency coordination and transaction management module, and a query routing and result fusion module.

[0008] The incremental data perception and preprocessing module is used to continuously monitor and capture new data entities and relation triples from multi-source data streams.

[0009] This module first cleans and standardizes the captured incremental data, and then performs entity linking operations to accurately match the entity references in the incremental data with existing entity nodes in the knowledge graph or determine them as new entities.

[0010] For entities that are successfully matched, the module generates entity update operation instructions;

[0011] For new entities and new relationships that cannot be matched, the module generates entity and relationship insertion operation instructions.

[0012] All operation instructions are encapsulated into transaction units with globally unique sequence numbers, and are accompanied by timestamps and data source identifiers.

[0013] The dynamic index hierarchical building module is used to provide an index structure that supports efficient incremental updates for knowledge graphs.

[0014] The index system built by this module consists of a base-state index layer and an incremental log layer.

[0015] The base state index layer adopts the native index structure of graph databases to index the full data state of the knowledge graph at a certain stable moment. The update cycle of this index is a preset long interval.

[0016] The incremental log layer consists of a series of incremental index blocks organized in chronological order, with each incremental index block corresponding to all data operation transaction units that occur within a specific time window.

[0017] The incremental index block uses a memory-optimized hash and inverted index structure to record only the entity identifiers, relationship identifiers and their operation types that have changed within the time window.

[0018] After receiving a transaction unit from the incremental data perception and preprocessing module, the dynamic index hierarchical construction module does not directly modify the base state index layer. Instead, it appends the transaction unit to the currently active incremental index block according to its timestamp.

[0019] When the size of an active incremental index block reaches a preset threshold or its time window expires, the module will seal the current active block as a read-only historical incremental index block and create a new active incremental index block to receive subsequent transactions.

[0020] The consistency coordination and transaction management module is used to ensure the consistency between the knowledge graph data state and query results during incremental updates.

[0021] This module maintains a global version vector, which consists of the base state index version number and the identifier of the current largest incremental index block.

[0022] Each time the ground state index layer is rebuilt or a new incremental index block is sealed, the atomic increment of the version vector is triggered.

[0023] For each write transaction, the module assigns the current version vector as the snapshot version before distributing it to the dynamic index hierarchical building module.

[0024] This module also implements a multi-version concurrency control mechanism to ensure that read queries can be executed based on historically consistent version snapshots without blocking concurrent write operations.

[0025] When the system decides to trigger the reconstruction of the base state index layer, this module is responsible for coordinating the reconstruction process: First, it locks the current version vector, preventing new write transactions from obtaining the updated snapshot version;

[0026] Next, it guides the system to asynchronously calculate a new full data state that integrates all incremental updates in the background based on the data of the current base state index layer and the operation logs in all sealed incremental index blocks, and to build a new base state index layer based on this state.

[0027] After the new ground-state index layer is built, the module atomically updates the ground-state index version number in the version vector, clears all merged incremental index blocks, and then releases the write lock, allowing the system to resume normal writing.

[0028] This process is transparent to the front-end query service and is uninterrupted.

[0029] The query routing and result fusion module is used to process knowledge query requests submitted by users.

[0030] After receiving a query request, this module first obtains the latest version vector from the consistency coordination and transaction management module, and uses this as a read snapshot for the query.

[0031] Subsequently, the query routing and result fusion module parses and rewrites the query request.

[0032] For conditions involving filtering by entity or relation attributes, the module breaks down the query into two parts:

[0033] One part points to the base state index layer, and the other part points to all incremental index blocks related to the read snapshot version.

[0034] The module sends the base state query sub-request to the base state index layer for retrieval, and simultaneously sends the incremental query sub-request to the corresponding incremental index block for retrieval.

[0035] Incremental index block retrieval returns records of data operations that have changed since the base-state index version.

[0036] The query routing and result fusion module finally executes the result fusion logic:

[0037] It performs online correction on the original result set retrieved from the base-state index layer based on the type of incremental operation record.

[0038] For insert operations, the new entity or relation is added to the result set;

[0039] For update operations, the old value is overwritten with the new value; for delete operations, the corresponding entry is removed from the result set.

[0040] The final result set after fusion processing reflects the complete and consistent state of the knowledge graph under the query snapshot version and is returned to the user.

[0041] As one embodiment of the present invention, the memory-optimized hash and inverted index structure of the incremental index block in the dynamic index hierarchical construction module is constructed as follows:

[0042] For each incremental index block, maintain a main hash table of entity identifiers to operation record lists, and an inverted index table of relation type to entity identifier pairs.

[0043] When a new insert or update transaction unit is added, the module first extracts the entity identifier from the transaction unit, looks up or creates the corresponding entry in the main hash table, and adds the operation details of the transaction to the operation record list of the entity. The operation record includes the operation type, relation object, attribute value and timestamp.

[0044] At the same time, if the transaction involves a relationship, the module extracts the relationship type and records the entity pairs under that relationship type in the inverted index table.

[0045] This structure enables incremental change retrieval for specific entities or relationships to be completed in constant or near-constant time complexity.

[0046] Furthermore, the specific execution flow of the result fusion logic of the query routing and result fusion module is as follows:

[0047] The module first obtains the initial result set returned from the ground state index layer.

[0048] Then, it iterates in parallel through the list of operation records returned by all relevant incremental index blocks.

[0049] For each operation record, the module checks whether the entity identifier of the operation exists in the initial result set or whether it meets the query conditions.

[0050] If the operation type is insert and the query conditions are met, then the entity and its relationship information are constructed as a new result item and added to the set.

[0051] If the operation type is update and the entity is already in the collection, the old attribute value corresponding to the entity is replaced with the new value from the operation record. If the operation type is delete and the entity is in the collection, the entity is removed from the result collection.

[0052] After the traversal is complete, the module reorders the final result set according to the sorting rules required by the query, applies pagination restrictions, and generates the final query response.

[0053] Furthermore, the implementation details of the multi-version concurrency control mechanism in the consistency coordination and transaction management module are as follows:

[0054] This module maintains multiple version chains for each data item entity or relationship. Each version contains the data value, the snapshot version number of the transaction that created that version, and a deletion marker.

[0055] When a write operation is performed, the transaction creates a new version of the data item based on its snapshot version number and links the new version to the head of the version chain, while the old version is retained.

[0056] When performing a read operation, the transaction searches the version chain for the latest version that is less than or equal to the read snapshot version number and has not been marked as deleted, based on its read snapshot version number.

[0057] This mechanism allows read operations to obtain a consistent data view without locking, while write operations are implemented by creating new versions, thus achieving high concurrency for read and write operations.

[0058] Furthermore, the system also includes an index performance monitoring and adaptive adjustment module.

[0059] This module continuously collects performance metrics such as query response time, index block size, memory usage, and incremental merging trigger frequency for the base-state index layer and each incremental index block.

[0060] The module has a built-in performance analysis model. When it detects that the latency of the fusion query exceeds the preset threshold due to the excessive number of incremental index block layers within multiple consecutive time windows, the performance analysis model will generate a base state index reconstruction suggestion and submit the suggestion weight, estimated reconstruction time, and impact assessment on the current service to the consistency coordination and transaction management module.

[0061] The consistency coordination and transaction management module determines whether to initiate an asynchronous base-state index reconstruction process immediately or during off-peak hours based on the system's current load strategy, thereby achieving adaptive optimization of the index maintenance strategy.

[0062] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0063] 1. This invention introduces a dynamic index hierarchical architecture, decoupling the cumbersome operation of full index reconstruction into periodic asynchronous updates of the base index and real-time appending to the incremental index. The system only needs to maintain lightweight incremental index blocks most of the time, thus completely avoiding query service interruptions and performance drops caused by data updates. The knowledge graph can continuously incorporate new data with near real-time latency, while providing stable, efficient, and consistent query services, meeting the stringent requirements of time-sensitive scenarios such as financial risk control and public opinion monitoring.

[0064] 2. The consistency coordination and transaction management module designed in this invention provides the system with strong consistency guarantees and high concurrency processing capabilities through the collaboration of version vectors and multi-version concurrency control mechanisms. Read queries are executed based on version snapshots, completely unaffected by background index merging or reconstruction processes, ensuring the accuracy and timeliness of query results. Write operations are recorded in an orderly manner through transaction management and version chains, guaranteeing the atomicity and durability of data updates. This design enables the system to maintain good responsiveness and data reliability even when dealing with massive concurrent read and write operations.

[0065] 3. The query routing and result fusion module of this invention realizes intelligent decomposition of the query process and online fusion of results. This module efficiently routes a single query to the base state index and multiple incremental indexes, and uses incremental operation logs to accurately correct the basic results. This method avoids repeated scanning of the entire dataset and reduces query overhead in incremental update scenarios. Combined with the index performance monitoring and adaptive adjustment module, the system can dynamically optimize the index reconstruction strategy according to the actual operating load, continuously maintaining optimal query performance and resource utilization in long-term operation, and realizing intelligent and automated system maintenance. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the overall technical architecture of the big data knowledge construction and management system proposed in this invention;

[0067] Figure 2 This is a schematic diagram of the core principle framework of the dynamic index hierarchical construction module in this invention;

[0068] Figure 3 This is a schematic diagram of the multi-level interaction relationship and data flow of the consistency coordination and transaction management module in this invention;

[0069] Figure 4 This is a logical flow diagram of the query routing and result fusion module in this invention;

[0070] Figure 5 This is a schematic diagram of the interaction framework between the index performance monitoring and adaptive adjustment module and the system core module in this invention. Detailed Implementation

[0071] Please refer to Figures 1 to 5 This invention provides a big data knowledge construction and management system, which aims to solve the technical problem of service interruption and query delay caused by incremental updates triggering full index reconstruction in existing knowledge graph systems.

[0072] This system achieves real-time incremental updates and uninterrupted high-performance query services for the knowledge graph through an innovative hierarchical indexing architecture and consistency coordination mechanism.

[0073] Please refer to the attached document. Figure 1 The overall architecture of the system includes an incremental data perception and preprocessing module, a dynamic index hierarchical construction module, a consistency coordination and transaction management module, a query routing and result fusion module, and an index performance monitoring and adaptive adjustment module.

[0074] These modules work together to form a closed-loop, adaptive knowledge management and service system.

[0075] The incremental data perception and preprocessing module is the interface between the system and external multi-source data streams. It is responsible for continuously capturing, cleaning and standardizing the new input data, and transforming it into transaction instructions that can be processed within the system.

[0076] This module is equipped with multiple data connectors that interface with different types of input sources, including but not limited to change logs from relational databases, streaming data from message queues, events pushed by application programming interfaces, and batch-uploaded data files.

[0077] Each data connector is equipped with a specific protocol parser and rate controller to ensure stable and reliable data ingestion.

[0078] When raw data flows in, the module first performs data cleaning operations, which include removing duplicate records, correcting obvious format errors, filling in null values ​​in key fields, and converting heterogeneous data formats into a unified internal representation format, such as converting all date and time fields to international standard timestamps and converting all numeric fields to double-precision floating-point numbers.

[0079] The cleaned data enters the entity linking sub-stage.

[0080] At this stage, the module calls the entity parsing engine, which loads the existing entity dictionary and alias mapping table in the knowledge graph.

[0081] For each entity reference appearing in the incremental data, such as a person's name, organization's name, or product name mentioned in a text, the entity parsing engine uses a composite algorithm based on word vector similarity calculation and context feature matching to calculate its matching confidence with existing entity nodes in the knowledge graph.

[0082] The system has a preset high confidence threshold, such as 95%.

[0083] If the matching confidence score is greater than the threshold, the entity link is considered successful. The incremental data is then associated with the corresponding existing entity node in the knowledge graph, and an entity update operation instruction is generated.

[0084] If the matching confidence is less than the threshold, it is determined to be a new entity. The module assigns a globally unique entity identifier to it and generates an entity insertion operation instruction.

[0085] For relation triples contained in the data, namely the combination of head entity, relation type, and tail entity, the module will check whether the head entity and tail entity involved in the relation already exist in the knowledge graph or have been created in the current incremental batch.

[0086] If at least one of the entities connected by the relation is a new entity, or if the relation schema does not exist in the existing graph, then a corresponding relation insertion operation instruction is generated.

[0087] All generated operation instructions, including entity insertion, entity update, and relation insertion, will be encapsulated into independent transaction units.

[0088] Each transaction unit is assigned a globally unique and monotonically increasing sequence number, the uniqueness and order of which are guaranteed by a distributed sequence generator.

[0089] The transaction unit also includes an intake timestamp accurate to microseconds and a data source identifier that identifies the origin of the original data.

[0090] After being encapsulated, the transaction unit is submitted to a highly reliable, high-throughput transaction message bus, awaiting consumption and processing by subsequent modules.

[0091] The dynamic index layering module is the core of the system to achieve efficient incremental updates. It builds and maintains a two-layer index system consisting of a base-state index layer and an incremental log layer.

[0092] Please refer to the attached document. Figure 2 The diagram clearly illustrates the structural relationship and data flow between these two levels of indexes.

[0093] The base state index layer is the cornerstone of the system. It adopts a deeply optimized native index structure of graph database to index the full data state of the knowledge graph at a specific stable moment.

[0094] This index structure typically includes B+ tree indexes for entity attributes, hash indexes for entity types, adjacency list indexes for relation types, and inverted indexes that support full-text search.

[0095] The base-state index layer is persistently stored on a high-speed solid-state drive array to provide stable read performance. Its updates are not real-time, but are triggered strategically according to a preset long period, such as every 24 hours or when incremental data accumulates to a certain scale, such as 100 million triples.

[0096] In contrast to the base-state index layer, the incremental log layer is the component that handles real-time updates. It consists of a series of incremental index blocks organized in strict chronological order.

[0097] Each incremental index block is essentially a memory-optimized data structure for fast retrieval. It corresponds to a specific, continuous time window, such as the past 1 minute or 5 minutes, and records all data operation transaction units that occurred within that time window.

[0098] Index blocks in the incremental log layer are divided into two states:

[0099] Active blocks and historical blocks. The system maintains one and only one active incremental index block at any given time to receive the latest arriving transaction units.

[0100] When the number of transaction units accumulated by an active block reaches a preset threshold, such as 100,000, or when its time window length reaches a preset duration threshold, such as 5 minutes, the dynamic index hierarchical construction module will perform a sealing operation.

[0101] The sealing operation first changes the state of the currently active block from writable to read-only, then marks it as a historical incremental index block, and serializes its memory data structure before dumping it to non-volatile memory or high-speed solid-state drive to ensure data persistence.

[0102] Immediately afterwards, the module initializes and activates a new incremental index block in memory to receive subsequent transaction units.

[0103] The incremental index block uses a hash and inverted index combined structure to achieve efficient retrieval.

[0104] Specifically, two core index tables are maintained for each incremental index block.

[0105] The first is the entity operation master hash table, which uses the entity identifier as the key.

[0106] Each key corresponds to a list of operation records sorted by transaction sequence number. Each operation record details the specific operation performed on that entity, including whether the operation type is insert, update, or delete, the relation type and tail entity identifier involved, the updated attribute key-value pairs, and the transaction timestamp corresponding to the operation. The second is the relation inverted index table, which uses the relation type as the key.

[0107] The value corresponding to each key is a set that records all new or updated entity pairs involving this relationship type within the time window of the incremental index block, namely the combination of the head entity identifier and the tail entity identifier.

[0108] When a new insert or update transaction unit is appended to the currently active incremental index block, the module's execution flow is as follows:

[0109] First, extract the target entity identifier from the transaction unit and look it up in the entity operation master hash table.

[0110] If a corresponding entry is found, the details of this transaction will be appended to the end of the entity's operation record list as a new operation record.

[0111] If not found, a new entry with the entity identifier as the key is created in the main hash table, and an operation list containing only records of this transaction is initialized.

[0112] Secondly, if the current transaction unit contains relational operations, the relation type is extracted simultaneously, and the head entity identifier and tail entity identifier involved in the current transaction are inserted into the set of entity pairs under the relation type in the relation inverted index table.

[0113] This design allows all incremental changes for a specific entity to be quickly retrieved with a single hash lookup, while incremental change queries for a specific relation type can be quickly located with a single inverted index lookup. Both can be completed in constant or near-constant time complexity, laying a solid foundation for subsequent efficient query fusion.

[0114] The consistency coordination and transaction management module is the central nervous system of the system. It is responsible for ensuring that the data state of the knowledge graph presents strong consistency to the outside world throughout the complex process of incremental updates and index reconstruction, and for managing the concurrent execution of all read and write transactions.

[0115] Please refer to the attached document. Figure 3 This diagram illustrates the multi-level interaction between version control and transaction flow within the module.

[0116] The core of this module is maintaining a global version vector.

[0117] This version of the vector is a binary data structure, consisting of two components:

[0118] The base state index version number and the identifier of the current largest incremental index block.

[0119] The base state index version number is an integer, and it is atomically incremented by 1 each time a complete base state index layer reconstruction is completed and takes effect.

[0120] The current largest incremental index block identifier is typically a string containing the start stamp of the time window, which points to the most recently archived historical incremental index block in the incremental log layer.

[0121] Each time a new incremental index block is sealed, the current largest incremental index block identifier in the version vector is updated.

[0122] The version vector itself is stored in a distributed coordination service with atomic read and write operation capabilities, such as a key-value store based on the Raft consensus algorithm, to ensure the atomicity and global visibility of its updates.

[0123] For each write transaction unit flowing in from the incremental data awareness and preprocessing module, the consistency coordination and transaction management module performs the following operations before distributing it to the dynamic index hierarchical construction module for index appending:

[0124] First, read the current global version vector from the distributed coordination service.

[0125] Then, this version vector is used as the snapshot version of the write transaction and bound to the transaction unit.

[0126] This means that the transaction is logically considered to have occurred within the knowledge graph state represented by this particular version.

[0127] The bound transaction unit is sent to the currently active incremental index block of the dynamic index hierarchical building module.

[0128] The multi-version concurrency control mechanism implemented in this module is key to ensuring high-concurrency read and write performance.

[0129] Under this mechanism, the system maintains a version chain for each data item in the knowledge graph, that is, each entity or each relationship.

[0130] A version chain is a linked list ordered by version number, where each node represents a historical or current version of the data item.

[0131] Each version node contains the following fields:

[0132] Data values, namely the specific attributes and relationship information of entities or relationships under this version; the snapshot version number of the transaction that created this version;

[0133] And a boolean deletion flag, used to indicate whether the version has been logically deleted.

[0134] When a write transaction is executed, such as updating an entity's attribute, the transaction processor finds the head of the entity's version chain and creates a new version node for the entity based on its snapshot version number.

[0135] The new version node contains updated attribute values, its creation version number is set to the snapshot version number of the current write transaction, and the deletion flag is false.

[0136] Then, this new version node is inserted at the head of the version chain, becoming the latest version.

[0137] The existing old version nodes are retained in the linked list. Deletion operations are performed by creating a new version node with the deletion flag set to true.

[0138] When a read query is executed, the query processor requests a read snapshot version number from the consistency coordination and transaction management module. This version number is usually the global version vector at the time the query is initiated.

[0139] Subsequently, when accessing a data item, the read operation traverses the version chain of that data item, searching for the latest version node that satisfies the following conditions:

[0140] The node's creation version number is less than or equal to the snapshot version number of the read query, and the node has not been marked for deletion.

[0141] The data found in the version node is the consistent data view that the read query should see.

[0142] This mechanism allows read operations to be performed without acquiring any locks and without blocking, even if the same data item is being modified concurrently.

[0143] Write operations are implemented by creating new versions, avoiding direct conflicts with read operations, thus achieving extremely high read-write concurrency.

[0144] When the system needs to trigger the reconstruction of the base-state index layer based on preset strategies or monitoring recommendations, the consistency coordination and transaction management module is responsible for coordinating this critical process, ensuring that it is transparent and uninterrupted to the front-end query service. The reconstruction process is as follows:

[0145] The first step is for the coordinator to lock the global version vector.

[0146] During the locking period, new write transactions can still be generated, but they will obtain a frozen, unchanging snapshot version number from the version vector service. This version number corresponds to the system state before the rebuild began.

[0147] The second step involves the coordinator instructing the index rebuilding engine to perform the rebuilding task asynchronously in the background.

[0148] The working inputs of the reconstruction engine are a full data snapshot corresponding to the current base state index layer, and operation logs recorded in all sealed historical incremental index blocks since this base state index version.

[0149] The reconstruction engine replays these operation logs sequentially, applying all insert, update, and delete operations to compute a new, full knowledge graph state that incorporates all incremental updates.

[0150] The third step is to rebuild the engine and construct a completely new base state index layer based on this new full state.

[0151] This build process is performed in a separate storage space and does not affect the services of the existing base-state index layer.

[0152] Fourth, after the new ground-state index layer is constructed and verified, the coordinator performs an atomic switching operation:

[0153] Increment the base state index version number in the global version vector and reset the current largest incremental index block identifier to its initial value or clear it, indicating that all merged incremental logs have become invalid.

[0154] This operation is completed within an atomic transaction. In the fifth step, the coordinator releases the version vector lock, and the system resumes normal writes.

[0155] Subsequently, new write transactions will obtain the snapshot version based on the new version vector.

[0156] Meanwhile, all queries arriving during and after the reconstruction period will have a snapshot version number that corresponds either to the old base state index plus a partial incremental index or to a completely new base state index. Through the multi-version concurrency control mechanism, a correct and consistent view can be obtained.

[0157] The storage space occupied by the merged old incremental index blocks will be asynchronously reclaimed.

[0158] The query routing and result fusion module is the portal through which the system provides query services. It intelligently decomposes user query requests, retrieves the base state index and incremental index in parallel, and merges the results online to return the latest and consistent data view.

[0159] Please refer to the attached document. Figure 4 This framework diagram depicts the complete logical flow from receiving a query to returning a result.

[0160] This module exposes a unified query interface to receive query requests submitted by users.

[0161] Query requests are typically expressed in Structured Query Language or domain-specific query languages, and include the type of the target entity, attribute filtering conditions, relation path traversal requirements, sorting rules, and pagination parameters.

[0162] After receiving the query request, the module first initiates a read version retrieval call to the consistency coordination and transaction management module to obtain the latest global version vector.

[0163] This version vector will serve as the read snapshot version for this query, ensuring a consistent baseline for the query results.

[0164] Next, the module's query parsing and rewriting engine begins to work.

[0165] The engine parses the original query statement and generates an internal query execution plan tree.

[0166] The core task of the rewrite engine is to break down the execution plan tree into two logically independent but ultimately mergeable sub-plans based on the read snapshot version: the base query sub-plan and the incremental query sub-plan.

[0167] The ground-state query subplan is designed to directly retrieve data from the ground-state index layer.

[0168] It assumes that the state of the knowledge graph remains at the point represented by the base state index version number, i.e., without any incremental updates.

[0169] Therefore, this sub-plan will be directly compiled into a set of query operations that can be efficiently executed by the graph database's native indexes.

[0170] The incremental query subplan is specifically designed to retrieve all relevant data changes that have occurred since the base state index version up to the maximum incremental index block contained in the read snapshot version.

[0171] The sub-plan was further broken down into multiple concurrent subtasks, each targeting a related historical incremental index block.

[0172] The query routing and result fusion module determines which incremental index blocks need to be queried based on the identifier of the current largest incremental index block in the read snapshot version.

[0173] Subsequently, the module concurrently sends multiple incremental retrieval requests to the dynamic index hierarchical construction module.

[0174] Each request carries the following key parameters:

[0175] The target incremental index block identifier, the range of entity identifiers to be retrieved, or the relation type condition.

[0176] The hash and inverted index structure within the incremental index block plays a crucial role here, enabling the rapid return of a list of all operation records matching the query conditions within the time window of that block.

[0177] These operation log lists are the raw materials for result fusion. The module initiates the result fusion logic simultaneously with or after parallel retrieval. The fusion logic starts with the initial set of results returned from the base-state index layer.

[0178] This initial result set reflects the state of the knowledge graph in its base-state indexed version.

[0179] Then, the merging unit begins processing the list of operation records returned from each incremental index block.

[0180] The processing involves scanning each operation record one by one and correcting the initial result set online based on the operation type in the record.

[0181] The correction logic follows these rules: For an operation record, first determine whether the entity identifier of the operation already exists in the initial result set, or whether the entity meets the conditions of the original query.

[0182] If the operation type is entity insertion or relation insertion, and the newly inserted entity or relation meets the query conditions, the merging agent will construct a new result item based on the details in the operation record and add it to the final result set.

[0183] If the operation type is an attribute update, and the entity being updated already exists in the initial result set, the merging unit will locate the result item and overwrite the old attribute value with the new attribute value provided in the operation record.

[0184] If the operation type is deletion and the entity to be deleted exists in the initial result set, the merging unit will remove the result item from the set.

[0185] After traversing all relevant operation records, the fusion processor obtains the corrected result set, which reflects the complete state of the knowledge graph after all incremental updates in the read snapshot version.

[0186] Finally, the module reorders the final result set according to the sorting rules specified in the original query request, such as sorting by a certain attribute value in descending order.

[0187] Then, the pagination parameters are applied to extract the results for the specified page number and quantity, which are then packaged into a standard query response format and returned to the user.

[0188] The entire query process, from route decomposition to parallel retrieval and then to online fusion, is transparent to the user. What the user perceives is a fast and data-consistent query.

[0189] The index performance monitoring and adaptive adjustment module is a smart operation and maintenance component of the system. It continuously monitors the system's operating status and intelligently analyzes it to dynamically optimize index maintenance strategies, ensuring that the system operates in the optimal performance range for a long time.

[0190] Please refer to the attached document. Figure 5 This diagram illustrates how this module collects data and interacts with other core modules.

[0191] This module has a built-in high-performance metrics collector that collects runtime metrics from various key points of the system at a fixed frequency, such as once per second.

[0192] The collected metrics mainly include the following categories:

[0193] The first category is query performance metrics, including the total time taken by the query routing and result fusion module to process each query, the time taken in the base index query sub-stage, the time taken in the incremental index fusion sub-stage, and the percentage of queries that time out.

[0194] The second category is index status indicators, including the size of the currently active incremental index blocks in the dynamic index hierarchical construction module, the number of historical incremental index blocks residing in memory, the data volume of each historical incremental index block, and the disk usage of the base state index layer.

[0195] The third category is system resource metrics, including the percentage of memory used by indexing operations, the utilization rate of the central processing unit cores, and the input / output throughput.

[0196] The fourth category is the update flow indicator, which is the number of transaction units received by the incremental data perception and preprocessing module per unit time.

[0197] All these metrics are timestamped and then stored in the time-series database inside the streaming injection module.

[0198] The core of the module is the performance analysis model.

[0199] The model periodically aggregates and analyzes metric data from the past time window, for example, every 5 minutes. The analysis model incorporates multiple performance thresholds and heuristic rules.

[0200] The key rule focuses on the depth effect of incremental index chains.

[0201] The model calculates the average number of incremental index blocks that need to be merged per query over a period of time.

[0202] At the same time, it monitors the proportion of queries where the overall query latency exceeds the preset service level agreement threshold due to the excessive time spent in the incremental fusion phase.

[0203] When the model detects that the average number of incremental index blocks to be merged exceeds a threshold, such as 10, over multiple consecutive analysis periods, and the proportion of query latency exceeding the standard continues to exceed another threshold, such as 5%, the performance analysis model will determine that the current system is suffering from query performance degradation due to the excessive depth of the incremental log layer.

[0204] At this point, the model will generate a formal ground-state index reconstruction proposal.

[0205] The recommendation is a structured data object that includes not only the reconstruction recommendation itself, but also the estimated reconstruction time based on historical data, an assessment of the slight impact the reconstruction process may have on write throughput, and the recommendation execution priority weight based on the current system load and time period.

[0206] The generated reconstruction proposal is immediately submitted to the decision interface of the consistency coordination and transaction management module.

[0207] The consistency coordination and transaction management module has a policy decision-maker that receives suggestions from the index performance monitoring and adaptive adjustment module, and also considers the maintenance time window policy preset by the system administrator, such as only allowing major maintenance operations during the off-peak period from 2 am to 4 am.

[0208] The strategy decision-maker comprehensively evaluates the suggested weights, current system load, expected impact, and maintenance time strategy to make a final decision.

[0209] If the decision is to be executed immediately, the coordinator will initiate an asynchronous base-state index reconstruction according to the process described above.

[0210] If the decision is to postpone implementation, it may be recommended to queue up and wait for the next suitable off-peak period window.

[0211] Through this closed-loop monitoring, analysis, and feedback adjustment mechanism, the system achieves adaptive optimization of the index maintenance strategy. It can balance real-time update requirements and query performance in long-term operation without manual intervention, thus maintaining the system's high efficiency and stability.

[0212] This embodiment details a complete implementation plan for a big data knowledge construction and management system. The system achieves real-time access and standardization of multi-source data through an incremental data sensing and preprocessing module.

[0213] A dynamic index hierarchical construction module is used to build an index system that separates the base state and incremental states, supporting low-latency incremental writes;

[0214] The consistency coordination and transaction management modules ensure strong data consistency and high-concurrency read / write operations.

[0215] A seamless real-time query experience is provided through the query routing and result fusion module;

[0216] The system achieves self-optimization and operation through the index performance monitoring and adaptive adjustment module.

[0217] Each module strictly follows the appendix Figure 1 To be continued Figure 5 The architecture and interactions shown work together to resolve the technical contradiction between real-time incremental updates and continuous high-performance queries of the knowledge graph.

[0218] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0219] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A big data knowledge construction and management system, characterized in that, include: The incremental data perception and preprocessing module is used to continuously monitor and capture new data entities and relation triples from multi-source data streams; The dynamic index layering construction module is used to build and maintain an index system consisting of a base-state index layer and an incremental log layer; The consistency coordination and transaction management module is used to maintain a global version vector consisting of the base state index version number and the identifier of the current largest incremental index block; The query routing and result fusion module is used to receive user query requests and obtain the latest version vector from the consistency coordination and transaction management module as a read snapshot of the query. The query request is parsed and rewritten into a base state query sub-request pointing to the base state index layer and an incremental query sub-request pointing to all incremental index blocks related to the read snapshot version. The base state query sub-request is sent to the base state index layer for retrieval, and the incremental query sub-request is concurrently sent to the corresponding incremental index block for retrieval to obtain the data operation records that have changed since the base state index version. Finally, the original result set retrieved from the base state index layer is corrected online according to the type of incremental operation record. For insertion operations, new entities or relations are added to the result set; for update operations, new values ​​are overwritten with old values; for deletion operations, the corresponding entries are removed from the result set. A final result set that reflects the complete and consistent state of the knowledge graph under the query snapshot version is generated and returned to the user. The ground state index layer uses the native index structure of the graph database to index the full data state of the knowledge graph at a stable time, and the update cycle is a preset long interval. The incremental log layer consists of a series of incremental index blocks organized in chronological order. Each incremental index block corresponds to all data operation transaction units that occur within a specific time window. It adopts a memory-optimized hash and inverted index structure and only records the entity identifier, relationship identifier and operation type that change within the specific time window. After receiving a transaction unit, the dynamic index hierarchical construction module appends it to the currently active incremental index block according to the timestamp. When the size of the active incremental index block reaches a preset threshold or its time window expires, the current active block is sealed as a read-only historical incremental index block and a new active incremental index block is created. The construction process of the hash and inverted index composite index structure of the incremental index block in the dynamic index hierarchical construction module is as follows: For each incremental index block, maintain a main hash table with entity identifier as the key and list of operation records as the value, and an inverted index table with relation type as the key and set of entity identifier pairs as the value; When a new insert or update transaction unit is appended, the entity identifier in the transaction unit is extracted, the corresponding entry is looked up or created in the main hash table, and the operation details of the transaction are appended to the operation record list of the entity. The operation details include operation type, relation object, attribute value and timestamp. Additionally, if a transaction involves a relationship, the relationship type is extracted and the entity pairs under that relationship type are recorded in the inverted index table.

2. The big data knowledge construction and management system according to claim 1, characterized in that, The incremental data perception and preprocessing module cleans and standardizes the captured incremental data, performs entity linking operations to match the entity references in the incremental data with existing entity nodes in the knowledge graph, or determines them as new entities. For successfully matched entities, it generates entity update operation instructions, and for new entities and new relationships that cannot be matched, it generates entity and relationship insertion operation instructions. All operation instructions are encapsulated into transaction units with globally unique sequence numbers, timestamps, and data source identifiers.

3. The big data knowledge construction and management system according to claim 1, characterized in that, The consistency coordination and transaction management module triggers atomic increment of the version vector each time the base state index layer is rebuilt or a new incremental index block is sealed. It allocates the current version vector as its snapshot version for each write transaction before distribution and implements a multi-version concurrency control mechanism to ensure that read queries are executed based on historically consistent version snapshots without blocking concurrent write operations. When the base state index layer reconstruction is triggered, the current version vector is locked to prevent new write transactions from obtaining the updated snapshot version. The system is guided to asynchronously calculate the full data state that merges all incremental updates in the background based on the current base state index layer data and the operation logs in all sealed incremental index blocks, and build a new base state index layer. After the new base state index layer is built, the base state index version number in the version vector is atomically updated, the merged incremental index blocks are cleared, and the write lock is released.

4. The big data knowledge construction and management system according to claim 1, characterized in that, The specific execution flow of the result fusion logic of the query routing and result fusion module is as follows: Retrieve the initial result set returned from the base state index layer; iterate in parallel through the list of operation records returned by all relevant incremental index blocks; For each operation record, check whether the entity identifier of the operation exists in the initial result set or whether it meets the query conditions; If the operation type is insert and the query conditions are met, then the entity and its relationship information are constructed as a new result item and added to the set; If the operation type is update and the entity is already in the collection, then the old attribute value corresponding to the entity is replaced with the new value in the operation record; If the operation type is deletion and the entity is in the collection, then the entity is removed from the result collection. After the traversal is complete, the final result collection is reordered according to the sorting rules required by the query and pagination restrictions are applied to generate the final query response.

5. A big data knowledge construction and management system according to claim 4, characterized in that, The implementation details of the multi-version concurrency control mechanism of the consistency coordination and transaction management module are as follows: For each data item entity or relationship, maintain a version chain consisting of multiple version nodes. Each version node contains the data value, the snapshot version number of the transaction that created that version, and a deletion marker. When a write operation is performed, the transaction creates a new version node for the data item based on its snapshot version number and links the new version node to the head of the version chain; When performing a read operation, the transaction searches the version chain for the latest version node that is less than or equal to the read snapshot version number and has not been marked as deleted, based on its read snapshot version number.

6. The big data knowledge construction and management system according to claim 5, characterized in that, It also includes an index performance monitoring and adaptive tuning module; The index performance monitoring and adaptive adjustment module is used to continuously collect performance indicators such as query response time, index block size, memory usage rate and incremental merging trigger frequency of the base state index layer and each incremental index block. The built-in performance analysis model generates a base state index reconstruction suggestion containing suggested weights, estimated reconstruction time and impact assessment when the fusion query delay exceeds a preset threshold due to too many incremental index block layers in multiple consecutive time windows. This suggestion is then submitted to the consistency coordination and transaction management module. The consistency coordination and transaction management module decides whether to initiate an asynchronous base-state index reconstruction process immediately or during off-peak hours based on the system's current load strategy.

7. A big data knowledge construction and management system according to claim 6, characterized in that, The list of operation records for entities in the main hash table is organized in order of the globally unique sequence number of the transaction unit.

8. A big data knowledge construction and management system according to claim 7, characterized in that, The write operations include entity insertion, entity update, relation insertion, and entity deletion; The read operations include exact queries based on entity identifiers, range queries based on attribute conditions, and graph traversal queries based on relation types.

Images