A dynamic graph database incremental construction and index updating method based on local recalculation

Abstract

The application discloses a kind of dynamic graph database incremental construction and index updating method based on local recalculation, it is related to database technical field;The method is by collecting new data, the unmatching node in incremental entity and its associated edge are as the center in the existing graph in the preset hop number along edge outward expansion, the subgraph region covered by expansion is determined as affected subgraph;The embedding vector of the node outside affected subgraph is kept unchanged, and the initial embedding vector is propagated to the neighborhood node along the edge to update the embedding vector of each node in affected subgraph;According to the embedding vector after updating, the vector index corresponding to affected subgraph is executed local update;The embedding vector after updating and vector index are written into dynamic graph database, and incremental construction is completed;The application limits incremental update operation in the range of affected subgraph, reduces the calculation overhead and index maintenance overhead in the process of dynamic graph database incremental update.

Description

Technical Field

[0001] This invention relates to the field of database technology, specifically to a method for incremental construction and index updating of dynamic graph databases based on local recomputation. Background Technology

[0002] A dynamic graph database is a database system used to store and query graph structure data, where the graph structure consists of nodes and edges connecting the nodes. As time goes by, the dynamic graph database continuously receives new data, requiring updates to the existing graph. In practical applications of dynamic graph databases, new data constantly arrives, and the graph structure continuously evolves. Incremental updates are one of the core operations of dynamic graph databases.

[0003] In dynamic graph databases, node embedding vectors are the fundamental inputs for graph computation in algorithms such as graph neural networks. When the graph undergoes incremental updates, the addition of new nodes and edges changes the graph's topology, and the embedding vectors of affected nodes need to be recalculated to reflect the updated graph structure. Existing incremental update methods for dynamic graph databases mainly adopt the following technical paths:

[0004] One existing approach is to recalculate the embedding vectors for the entire graph with each incremental update. This method merges the new data with the existing graph and then recalculates the embedding vectors for all nodes in the entire graph. However, the full recalculation method suffers from high computational overhead. As the graph size continues to grow, the time cost and computational resources required for full recalculation increase linearly. In large dynamic graph databases, frequent full recalculations make it difficult to guarantee the real-time performance of incremental updates.

[0005] Another existing method is to divide the map into multiple snapshots according to time windows, and perform an embedding vector calculation on the whole map once in each time window. This method reduces the number of calculations by reducing the update frequency, but sacrifices the real-time performance of incremental updates. In the time window between two snapshots, new data cannot be reflected in the embedding vector of the node in a timely manner, resulting in a lag between map data and embedding vector.

[0006] In terms of vector index maintenance, dynamic graph databases typically employ graph-based vector index structures to accelerate vector similarity retrieval. However, most existing vector indexes are designed for static data scenarios and do not support data updates after index construction. When the embedding vectors of nodes in the graph are updated, the entire vector index needs to be rebuilt. Full index reconstruction requires traversing all node embedding vectors and rebuilding the index structure, resulting in high index update latency and large input / output overhead. Studies have indicated that in environments with millions of vertices, full index reconstruction can take hours. Some existing solutions employ a periodic full reconstruction strategy, performing a unified index reconstruction after accumulating a certain number of updates. However, this method results in inconsistencies between the index data and the graph data between reconstructions, compromising the accuracy of query results.

[0007] Regarding query cache maintenance, dynamic graph databases typically set up query caches to accelerate the response to high-frequency query requests. When an incremental update occurs in the graph, query results in the cache involving updated nodes become invalid. Existing methods usually employ a full cache refresh strategy upon detecting a graph update, i.e., clearing all cache entries and gradually rebuilding them in subsequent queries. A full cache refresh causes a sharp drop in cache hit rate after the refresh, requiring many query requests to re-execute complete vector retrieval and graph traversal operations, significantly increasing query response latency.

[0008] In summary, existing dynamic graph databases suffer from several problems during incremental updates: high computational overhead and poor real-time performance due to full embedding vector recalculation; high index maintenance costs and long update latency due to full vector index reconstruction; and decreased cache hit rate and increased query response latency due to full cache refresh. These issues stem from the fact that existing methods employ a full-processing strategy during incremental updates, failing to localize the update based on its actual impact on the graph. There is an urgent need in this field for a technical solution that can identify the actual impact of incremental updates and perform local updates only on the affected areas. This would reduce the computational, index maintenance, and cache refresh overhead during incremental updates in dynamic graph databases, thereby improving the efficiency, index consistency, and query response speed of incremental updates. Summary of the Invention

[0009] To overcome the problems of the prior art, this invention discloses a method for incremental construction and index update of dynamic graph databases based on local recomputation.

[0010] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0011] The first aspect provides a method for incremental construction and index update of dynamic graph databases based on local recomputation, including the following steps:

[0012] S1, collect new data, which includes structured description data of application programming interfaces, system configuration file data, and operation log data;

[0013] Entity and relation extraction are performed on the newly added data to generate an incremental entity set and an incremental relation set;

[0014] Write the incremental entity set and incremental relation set into the incremental cache of the dynamic graph database according to the timestamp;

[0015] S2, compare each entity in the incremental entity set with the nodes of the existing graph in the dynamic graph database to determine the matching nodes and the unmatched nodes;

[0016] Centered on the unmatched nodes and the associated edges involved in the incremental relationship set, extend outward along the edges in the existing graph by a preset number of hops, and determine the subgraph region covered by the extension as the affected subgraph.

[0017] S3, keep the embedding vectors of nodes in the existing graph that are outside the affected subgraph unchanged;

[0018] For unmatched nodes within the affected subgraph, perform node embedding computation to generate initial embedding vectors;

[0019] The initial embedding vector is propagated along the edges of the affected subgraph to neighboring nodes, and the embedding vectors of each node in the affected subgraph are updated.

[0020] S4, based on the updated embedding vector, perform a local update on the vector index in the dynamic graph database corresponding to the affected subgraph;

[0021] S5 writes the updated embedding vector and the updated vector index into the dynamic graph database to complete the incremental construction.

[0022] Preferably, the preset number of jumps in step S2 is 2 jumps.

[0023] Preferably, in step S2, taking the unmatched nodes and the associated edges involved in the incremental relationship set as the center, the graph is expanded outward along the edges by a preset number of hops in the existing graph, and the subgraph region covered by the expansion is determined as the affected subgraph, specifically including:

[0024] Starting from the unmatched node, traverse the one-hop neighboring nodes along the associated edges;

[0025] Continue traversing the two-hop neighbor nodes along the edge from the one-hop neighbor node;

[0026] The subgraph consisting of unmatched nodes, one-hop neighbors, two-hop neighbors, and the edges connecting these nodes is considered the affected subgraph.

[0027] Preferably, in step S3, node embedding calculation is performed on the unmatched nodes in the affected subgraph, and an initial embedding vector is generated using a relational graph convolutional network;

[0028] Relational graph convolutional networks generate initial embedding vectors by aggregating the features of unmatched nodes and their neighboring nodes.

[0029] Preferably, step S3 involves propagating the initial embedding vector along the edges of the affected subgraph to neighboring nodes, specifically including:

[0030] The initial embedding vector is passed along the edge to the one-hop neighbor node, and the embedding vector of the one-hop neighbor node is updated.

[0031] The updated embedding vector of the one-hop neighbor node is passed along the edge to the two-hop neighbor node, and the embedding vector of the two-hop neighbor node is updated.

[0032] The propagation range does not exceed the boundary of the affected subgraph.

[0033] Preferably, the vector index in step S4 is an HNSW index or an IVF index.

[0034] Preferably, step S4 involves performing a local update on the vector index corresponding to the affected subgraph in the dynamic graph database, specifically including:

[0035] Only the vector index entries of nodes within the affected subgraph are recalculated and written; the vector index entries of nodes outside the affected subgraph remain unchanged.

[0036] Preferably, it further includes an index pre-construction step:

[0037] Based on the historical access frequency and node growth rate of each subgraph in the dynamic graph database, predict the hot subgraph regions within the future time window;

[0038] Vector indexes and caches are pre-established for hotspot subgraph regions.

[0039] Preferably, it also includes a query cache update step:

[0040] Retrieve the cache entries in the query cache that correspond to the nodes within the affected subgraph, and mark the cache entries as pending updates;

[0041] After the affected subgraph is updated, the cache entry is recalculated and written.

[0042] Preferably, it further includes a map incremental update step:

[0043] When new data is collected, repeat steps S1 to S5;

[0044] The updated entities and relationships are incrementally written to the dynamic graph database to generate the updated graph. The operation log of each incremental update is recorded, and version traceability and rollback are supported.

[0045] The beneficial effects of this invention are as follows:

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] By using unmatched nodes and their associated edges as the center, and expanding outward along the edges of the existing graph by a preset number of hops to determine the affected subgraph, the scope of incremental update operations is reduced from the entire graph to local subgraphs. This avoids recalculating the full embedding vector and reduces the computational overhead during the incremental update process of the dynamic graph database.

[0048] By keeping the embedding vectors of nodes outside the affected subgraph unchanged, and only performing embedding vector recalculation and edge propagation updates on nodes within the affected subgraph, the update range of the embedding vectors matches the actual change range of the graph, reducing the number of nodes that need to be recalculated during incremental updates and improving the real-time performance of incremental updates.

[0049] By performing local updates on the vector indexes corresponding to the affected subgraphs in the dynamic graph database, the update range of the vector indexes is kept consistent with the update range of the embedded vectors, avoiding the full reconstruction of the vector indexes and reducing index maintenance overhead and input / output overhead.

[0050] By recalculating and writing the vector index entries for nodes within the affected subgraph, while keeping the vector index entries for nodes outside the affected subgraph unchanged, the vector index maintains consistency with the embedded vector during incremental updates, thus avoiding data inconsistency issues caused by periodic full reconstruction.

[0051] By passing the initial embedding vector sequentially along the edges to the one-hop and two-hop neighbor nodes, the influence of unmatched nodes is propagated layer by layer along the graph structure to their two-hop neighborhood. The propagation range is consistent with the boundary of the affected subgraph. While ensuring that the embedding vector of the node in the two-hop neighborhood can reflect the influence of the new data, the boundedness of the propagation range is controlled.

[0052] By querying the cache update step, the cache entries corresponding to the nodes in the affected subgraph are obtained from the query cache and marked as pending update. After the affected subgraph is updated, the cache entries are recalculated and written to the cache, so that the cache update range matches the graph update range, avoiding the decrease in cache hit rate and the increase in query response latency caused by full cache refresh.

[0053] By using the index pre-construction step, hot subgraph regions within the future time window are predicted based on the historical access frequency and node growth rate of each subgraph in the dynamic graph database. Vector indexes and caches for hot subgraph regions are built in advance, so that the vector indexes and caches for hot subgraph regions are ready before the query request arrives, reducing query response latency.

[0054] By using the graph incremental update step, the incremental construction process is repeated every time new data is collected, and operation logs are recorded to support version traceability and rollback, so that the dynamic graph database has data reliability and recoverability during continuous incremental updates. Attached Figure Description

[0055] Figure 1 This is a flowchart of a method for incremental construction and index update of a dynamic graph database based on local recomputation, provided in Embodiment 1 of the present invention. Detailed Implementation

[0056] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.

[0057] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0059] Example 1

[0060] Please refer to Figure 1 This embodiment provides a method for incremental construction and index update of dynamic graph database based on local recomputation. A dynamic graph database is a database system used to store and query graph structure data, where the graph structure consists of nodes and edges connecting the nodes. As time goes by, the dynamic graph database continuously receives new data and needs to update the existing graph.

[0061] Traditional methods recalculate the entire graph with each update, resulting in a significant waste of computing resources and low update efficiency. This invention reduces the computational overhead of incremental updates in dynamic graph databases by locating the local subgraph regions affected by the new data and performing embedding vector recalculation and index updates only on those local regions.

[0062] In step S1, new data is collected, including structured description data of application programming interfaces, system configuration file data, and operation log data.

[0063] The structured description data of an application programming interface refers to the interface definition information described in a structured format, including the interface name, interface parameters, interface return value type, and calling relationships between interfaces;

[0064] System configuration file data refers to key value data used to configure the behavior of system components during the operation of a computer system, including component names, component dependencies, and component parameter configuration information;

[0065] Operation log data refers to user operation behavior data recorded during the operation of a computer system, including the operation subject, operation object, operation type, and operation timestamp;

[0066] After collecting the newly added data, entity extraction and relation extraction are performed on the newly added data. Entity extraction refers to identifying and extracting entity names with specific semantic meanings from the text content of the newly added data.

[0067] Relation extraction refers to identifying and extracting the types of relationships between entities from the text content of newly added data. Through entity extraction and relation extraction, an incremental set of entities and an incremental set of relationships are generated.

[0068] The incremental entity set contains all entities that appear in the newly added data, and the incremental relationship set contains all the relationships between the entities in the newly added data.

[0069] Write the incremental entity set and incremental relation set into the incremental cache of the dynamic graph database according to their respective timestamps;

[0070] The incremental cache is a storage area in a dynamic graph database specifically used to temporarily store newly added data. This area is independent of the main storage area of ​​the dynamic graph database.

[0071] When writing to the incremental cache, the incremental entities and incremental relationships are sorted and organized using the timestamp as an index, so that the data in the incremental cache is arranged in chronological order.

[0072] After the above processing, the newly collected data in step S1 is stored in the incremental cache in the form of an incremental entity set and an incremental relationship set, awaiting processing in subsequent steps.

[0073] In step S2, each entity in the incremental entity set is compared with the nodes of the existing graph in the dynamic graph database to determine the matching nodes and the unmatched nodes.

[0074] The existing graph in a dynamic graph database refers to the graph structure data that the dynamic graph database has stored at the current moment, including existing nodes and existing edges;

[0075] A matching node is a node in the existing graph whose semantics are consistent with those of an entity in the incremental entity set; an unmatched node is a node in the existing graph whose semantics are not consistent with those of an entity in the incremental entity set.

[0076] During the comparison, the similarity between the name or identifier of the incremental entity and the name or identifier of the existing graph node is calculated. When the similarity exceeds a preset threshold, the incremental entity is identified as a matching node; otherwise, it is identified as an unmatched node.

[0077] Centered on the unmatched nodes and the associated edges involved in the incremental relationship set, extend outward along the edges in the existing graph by a preset number of hops, and determine the subgraph region covered by the extension as the affected subgraph.

[0078] The preset number of jumps is a pre-defined positive integer representing the number of layers traversed outwards from the center node along the edge;

[0079] Starting from the unmatched node, traverse all one-hop neighbor nodes along the associated edges directly connected to the unmatched node to obtain the set of one-hop neighbor nodes.

[0080] Starting from each node in the one-hop neighbor set, continue traversing along the edges to obtain all two-hop neighbor nodes, thus obtaining the two-hop neighbor set. Continue traversing in the above manner until the number of hops reached the preset number;

[0081] The subgraph consisting of unmatched nodes, each hop neighbor node, and all edges connecting the above nodes is defined as the affected subgraph. All nodes and edges in the existing graph that are outside the affected subgraph are not included in the affected subgraph.

[0082] The affected subgraphs identified in step S2 represent the minimum range within which update operations need to be performed in subsequent steps. The portion of the existing graph located outside the affected subgraphs does not participate in subsequent embedding vector recalculation and index update operations.

[0083] In step S3, the embedding vectors of nodes located outside the affected subgraph in the existing graph are kept unchanged;

[0084] An embedding vector is a numerical vector representation obtained by mapping nodes in a graph structure to a continuous vector space. This vector is used to characterize the semantic and structural features of the nodes.

[0085] For nodes in the existing graph that are outside the affected subgraph, their embedding vectors will not be modified in this incremental update process; the embedding vector values ​​before the update will be used directly.

[0086] For unmatched nodes within the affected subgraph, node embedding calculations are performed to generate initial embedding vectors. Unmatched nodes are new entities introduced from the new data that do not have corresponding nodes in the existing graph.

[0087] For these new entities, their embedding vectors need to be calculated from scratch. The node embedding calculation is implemented using a relational graph convolutional network.

[0088] The relational graph convolutional network receives the initial feature vector of the unmatched node and the feature vector of its neighboring nodes in the affected subgraph as input. It aggregates and transforms the input features through multi-layer graph convolution operations and outputs the initial embedding vector of the unmatched node.

[0089] The initial embedding vector is propagated along the edges of the affected subgraph to neighboring nodes, and the embedding vectors of each node in the affected subgraph are updated.

[0090] Propagation refers to passing the information contained in the initial embedding vector to neighboring nodes layer by layer along the edges of the graph structure, so that the embedding vectors of the neighboring nodes can reflect the impact of the newly added entities.

[0091] The initial embedding vector of the unmatched node is passed along the edge to the one-hop neighbor node, and the one-hop neighbor node updates its own embedding vector according to the received information.

[0092] The updated embedding vector of the one-hop neighbor node is passed along the edge to the two-hop neighbor node. The two-hop neighbor node updates its own embedding vector according to the received information. The propagation is carried out layer by layer in the above manner until the propagation range covers all nodes in the affected subgraph.

[0093] The number of propagation layers does not exceed the preset number of hops determined in step S2. After the propagation update, the embedding vectors of each node in the affected subgraph are obtained.

[0094] In step S3, the embedding vectors of nodes outside the affected subgraph in the existing graph remain unchanged. Only nodes within the affected subgraph participate in the recalculation and propagation update of the embedding vectors. This local update strategy avoids the computational overhead caused by recalculating the entire graph.

[0095] In step S4, based on the updated embedding vector, a local update is performed on the vector index corresponding to the affected subgraph in the dynamic graph database. The vector index is a data structure in the dynamic graph database used to accelerate vector similarity retrieval.

[0096] The vector index is built based on the node's embedding vector and supports efficient approximate nearest neighbor search on the embedding vector. The vector index corresponding to the affected subgraph refers to the index entries in the vector index that correspond to each node in the affected subgraph.

[0097] Local update refers to modifying only the part of the vector index corresponding to the affected subgraph, while the part of the vector index corresponding to nodes outside the affected subgraph remains unchanged;

[0098] Extract the updated embedding vectors of each node in the affected subgraph, and replace the old embedding vector entries of the corresponding nodes in the vector index with the updated embedding vectors.

[0099] For unmatched nodes within the affected subgraph, add a corresponding index entry to the vector index and write the updated embedded vector of the unmatched node into the newly added index entry;

[0100] The index entries in the vector index corresponding to nodes outside the affected subgraph are not modified in any way;

[0101] Through the above local update operation, the vector index remains synchronized with the updated embedding vector, without the need to rebuild the entire vector index;

[0102] The vector index local update performed in step S4 has the same update range as the affected subgraph range determined in step S2, which further reduces the amount of computation in the incremental update process.

[0103] In step S5, the updated embedding vector and the updated vector index are written into the dynamic graph database to complete the incremental construction. The updated embedding vector refers to the updated embedding vector of each node in the affected subgraph obtained in step S3, and the updated vector index refers to the vector index after local update in step S4.

[0104] When writing the updated embedding vector to the dynamic graph database, the updated embedding vector of each node in the affected subgraph is stored in the main storage area of ​​the dynamic graph database at the storage location corresponding to that node, overwriting the original embedding vector of that node.

[0105] For unmatched nodes, a new storage location for the node is added in the main storage area of ​​the dynamic graph database, and the updated embedding vector of the node is written to the new storage location.

[0106] When writing the updated vector index to the dynamic graph database, the locally updated vector index is stored in the index storage area of ​​the dynamic graph database, overwriting the original vector index.

[0107] After the write operation is completed, the graph data, node embedding vectors, and vector indexes in the dynamic graph database are kept consistent, thus completing this incremental construction.

[0108] The steps S1 to S5 above together constitute the complete process of the dynamic graph database incremental construction and index update method based on local recomputation;

[0109] By locating the affected subgraph in step S2, the scope of incremental update is reduced from the entire graph to a local subgraph. By keeping the embedding vectors of nodes outside the affected subgraph unchanged in step S3 and by the local propagation of the embedding vectors along the edges, the recalculation of the embedding vectors of the entire graph is avoided.

[0110] The partial update of the vector index in step S4 avoids the need to rebuild the entire graph vector index. The final persistence of the incremental construction is completed by writing the updated embedded vectors and vector indexes to the dynamic graph database in step S5.

[0111] The above steps work together to reduce computational overhead and improve update efficiency during incremental updates of the dynamic graph database.

[0112] Furthermore, in step S2, the preset number of hops is set to 2 hops. The preset number of hops is a pre-set non-negative integer that represents the number of layers traversed outward from the center node along the edge. 2 hops means that from the center node, two layers of neighboring nodes are traversed along the edge, that is, one-hop neighboring nodes and two-hop neighboring nodes are obtained in sequence.

[0113] In the incremental update scenario of dynamic graph database, the impact of unmatched nodes and their associated edges introduced by the new data on the existing graph is spatially limited.

[0114] Information propagation in a graph structure follows the principle of locality, which means that the impact of a change in a node on other nodes in the graph decreases rapidly as the graph distance increases.

[0115] When a new node or edge is introduced into the graph, the impact on nodes that are far away from the new element is negligible in numerical terms, and there is no need to recalculate the embedding vector for these distant nodes.

[0116] Setting the preset number of hops to 2 hops can achieve a balance between update accuracy and computational efficiency.

[0117] In step S2, taking the unmatched nodes and the associated edges involved in the incremental relationship set as the center, the graph is expanded outward along the edges by a preset number of hops. This includes the following operations:

[0118] Starting from the unmatched node, traverse all one-hop neighbor nodes along the associated edges directly connected to the unmatched node to obtain the set of one-hop neighbor nodes. A one-hop neighbor node is a node that has a direct edge connection with the unmatched node.

[0119] Starting from each node in the one-hop neighbor set, continue traversing along the edges to obtain the two-hop neighbor set;

[0120] Two-hop neighbor nodes are nodes that are connected to unmatched nodes by exactly two edges. That is, these nodes have direct edge connections to one-hop neighbor nodes, but no direct edge connections to unmatched nodes.

[0121] The set of two-hop neighbor nodes does not contain nodes that have already appeared in the set of one-hop neighbor nodes, nor does it contain the unmatched node itself;

[0122] After the above traversal is completed, the subgraph formed by the unmatched node, all nodes in the one-hop neighbor node set, all nodes in the two-hop neighbor node set, and all edges connecting the above nodes is determined as the affected subgraph.

[0123] This subgraph contains all nodes and edges that can be reached within a 2-hop range from the central node. All nodes and edges outside this subgraph are not included in the affected subgraph and will not participate in the recalculation of the embedding vector and index update operations in subsequent steps.

[0124] The choice of a preset number of hops of 2 has a clear technical basis. In dynamic graph databases, the calculation of node embedding vectors depends on the node's own features and the feature information of its neighboring nodes.

[0125] When a new node is added to the graph, its one-hop neighbor nodes will directly receive information from the new node when updating their embedding vectors. Therefore, the embedding vectors of the one-hop neighbor nodes need to be recalculated.

[0126] The embedding vector update of the second-hop neighbor node depends on the updated embedding vector of the first-hop neighbor node, so the second-hop neighbor node also needs to participate in the update process;

[0127] Starting from nodes three hops and further away, the influence of newly added nodes has been sufficiently attenuated after two propagations. The influence of their three-hop and higher neighboring nodes is significantly reduced in magnitude. The accuracy improvement obtained by recompiling the embedding vector for these nodes is insufficient to offset the additional computational overhead.

[0128] The preset number of hops is set to 2 hops, so that the scope of the affected subgraph covers all nodes directly affected by the new data as well as the main nodes indirectly affected, while excluding distant nodes whose impact is negligible.

[0129] Compared to a scheme with a preset 1-hop count, the 2-hop scheme can capture the indirect impact of new data on second-hop neighbors after propagation through the first hop, resulting in more thorough updates. Compared to schemes with a preset 3-hop count or more, the 2-hop scheme avoids investing computing resources in distant nodes where the impact has already sufficiently decayed, resulting in lower computational overhead.

[0130] Dynamic graph databases typically have a high frequency of incremental updates, and balancing update accuracy and computational efficiency is crucial for each update. Two-hop is the preferred option that has been experimentally verified in most application scenarios.

[0131] Based on the determination of the affected subgraphs, the propagation range of the embedding vector in the subsequent step S3 does not exceed the boundary of the 2-hop subgraph. The initial embedding vector of the unmatched node is passed along the edge to the one-hop neighbor node and the two-hop neighbor node in sequence. The propagation terminates at the two-hop neighbor node and does not continue to expand outward.

[0132] In step S4, the local update range of the vector index is also consistent with the range of the 2-hop subgraph, and the update operation is only performed on the vector index entries of the nodes within the subgraph.

[0133] Specifically, by setting the preset number of hops to 2, this method reduces the number of nodes that need to be recalculated and the number of index entries that need to be updated during the incremental update process, while ensuring the accuracy of the updated graph data. This effectively reduces the computational and storage overhead of incremental updates in dynamic graph databases.

[0134] Furthermore, in step S2, taking the unmatched nodes and the associated edges involved in the incremental relationship set as the center, the graph is expanded outward along the edges by a preset number of hops in the existing graph, and the subgraph area covered by the expansion is determined as the affected subgraph.

[0135] When the preset number of hops is 2, the affected subgraph specifically includes the following nodes:

[0136] The unmatched node itself, its one-hop neighbor nodes, and its two-hop neighbor nodes;

[0137] Unmatched nodes are entities in the incremental entity set that do not have semantically consistent nodes in the existing graph.

[0138] A one-hop neighbor node is a node that has a direct edge connection to an unmatched node.

[0139] A two-hop neighbor node is a node that is connected to an unmatched node by exactly two edges. That is, a two-hop neighbor node has a direct edge connection with a certain hop neighbor node, but no direct edge connection with the unmatched node.

[0140] The two-hop neighborhood node set does not contain nodes that have already appeared in the one-hop neighborhood node set, nor does it contain the unmatched node itself. This subgraph contains all edges that connect the unmatched node, the one-hop neighborhood node, and the two-hop neighborhood node.

[0141] The subgraph consisting of unmatched nodes, one-hop neighbor nodes, two-hop neighbor nodes, and the edges connecting the above nodes is considered the affected subgraph. All nodes and edges outside this subgraph are not included in the affected subgraph.

[0142] One-hop neighbor nodes directly receive information from unmatched nodes and are directly affected by unmatched nodes when updating the embedding vector;

[0143] Two-hop neighborhood nodes indirectly receive information from unmatched nodes through one-hop neighborhood nodes, and are indirectly affected by unmatched nodes when updating the embedding vector;

[0144] Unmatched nodes, one-hop neighbor nodes, and two-hop neighbor nodes together constitute all the nodes affected by the new data in the existing graph under the two-hop condition. The edges connecting the above nodes constitute all the associated paths affected by the new data.

[0145] The subgraph formed by the above nodes and edges is taken as the affected subgraph, so that the scope of the affected subgraph matches the actual impact of the new data in the existing graph.

[0146] Based on the affected subgraph determined in the above manner, the propagation range of the embedded vector in the subsequent step S3 shall not exceed the boundary of the subgraph;

[0147] The initial embedding vector of the unmatched node is passed along the edge to the one-hop neighbor node and the two-hop neighbor node in turn, and the propagation terminates at the two-hop neighbor node.

[0148] In step S4, the local update range of the vector index is also consistent with the range of the subgraph, and the update operation is only performed on the vector index entries of the nodes within the subgraph.

[0149] The embedding vectors of nodes located outside the subgraph in the existing graph remain unchanged, and their corresponding vector index entries are also not modified.

[0150] Specifically, by defining the subgraph formed by unmatched nodes, one-hop neighbor nodes, two-hop neighbor nodes, and the edges connecting these nodes as the affected subgraph, this method accurately delineates the scope of incremental update operations under the two-hop condition. This ensures that the embedding vectors of one-hop and two-hop neighbor nodes can reflect the impact of the new data, while limiting the scope of the update operation to the local area actually affected by the new data.

[0151] Furthermore, in step S3, node embedding calculations are performed on the unmatched nodes within the affected subgraph. Initial embedding vectors are generated using the Relational Graph Convolutional Network (R-GCN). R-GCN is a graph neural network specifically designed for processing multi-relational graph data, and its calculation process is as follows:

[0152] For each node in the affected subgraph, an initial feature vector is constructed. For unmatched nodes, initial features are generated based on the entity attributes in the corresponding new data.

[0153] For existing matching nodes within the affected subgraph, the embedding vector of that node before the update is directly used as the initial feature.

[0154] The relational graph convolutional network updates the representation of all nodes layer by layer through stacked graph convolutional layers. In each graph convolutional layer, for each target node, two sub-steps are performed: neighborhood information aggregation and node update.

[0155] The neighborhood information aggregation sub-step is processed according to different relation types. For each relation type, all neighborhood nodes of the target node under the relation type are determined.

[0156] Multiply the embedding vector of each neighboring node in the current layer by the learnable weight matrix corresponding to the relation type to obtain the transformed vector of each neighboring node.

[0157] The transformed vectors of each neighboring node are normalized according to the number of neighboring nodes. The normalized vectors are then added together to obtain the aggregation result for this relation type.

[0158] After performing the above operations on all relation types, the aggregation results corresponding to each relation type are obtained;

[0159] The node update sub-step adds up the aggregation results of all the above relation types to obtain the total neighborhood aggregation, and multiplies the embedding vector of the target node itself by the learnable weight matrix corresponding to the self-loop connection to obtain the self-loop transformation result.

[0160] Add the total aggregation of the neighborhood to the result of the self-loop transformation to obtain the total aggregation of the target node in the graph convolutional layer;

[0161] The aggregated total is nonlinearly transformed using an activation function to obtain the embedding vector of the target node after updating in the graph convolutional layer;

[0162] The activation function uses the modified linear unit, which is mathematically defined as outputting the input value when the input value is greater than zero, and outputting zero when the input value is less than or equal to zero.

[0163] The complete calculation process of the above single-layer graph convolutional layer, with the overall formula as follows:

[0164] The updated embedding vector of the target node = activation function (self-loop weight matrix × current embedding vector of the target node + summation over all relation types (normalization coefficient × relation type weight matrix × current embedding vector of neighboring nodes));

[0165] Among them, the self-loop weight matrix is ​​a two-dimensional matrix that transforms the features of the target node itself, the relation type weight matrix is ​​a two-dimensional matrix that transforms the features of each neighboring node under the relation type, the normalization coefficient is the reciprocal of the number of neighboring nodes under the relation type, and the activation function is the modified linear unit.

[0166] The current layer embedding vector of the target node is used as the input of the next layer graph convolutional layer. The neighborhood aggregation and update process described above is repeated. The relational graph convolutional network contains two layers of graph convolutional layers. After two layers of graph convolutional operations, the embedding vector output by the last layer of graph convolutional layer is used as the initial embedding vector of the target node. After two layers of convolution, each node can capture the structural information of its two-hop neighborhood.

[0167] The aforementioned learnable weight matrices, namely the self-loop weight matrix and the weight matrices for each relation type, are optimized during the training phase through the link prediction task.

[0168] During training, 15% of the edges in the heterogeneous information network are randomly occluded. The occluded heterogeneous information network is then input into the relational graph convolutional network to obtain the embedding vector of each node.

[0169] Predict the existence of occluded edges based on the embedding vectors of each node, calculate the difference between the prediction results and the actual results using the cross-entropy loss function, and update the parameters of each learnable weight matrix through the backpropagation algorithm.

[0170] After training, the parameters of each learnable weight matrix are fixed and used for node embedding calculation in the actual inference stage;

[0171] Specifically, in the manner described above, the relational graph convolutional network performs node embedding computation on unmatched nodes within the affected subgraph, generating an initial embedding vector that integrates the unmatched node's own attribute features as well as its structural and semantic relationships with neighboring nodes in the affected subgraph.

[0172] Furthermore, in step S3, the specific process of propagating the initial embedding vector along the edges of the affected subgraph to neighboring nodes is as follows:

[0173] The initial embedding vector is a low-dimensional vector representation generated by performing node embedding computation on unmatched nodes within the affected subgraph using a relational graph convolutional network;

[0174] The initial embedding vector contains the initial features of the unmatched node itself, as well as its structural and semantic relationships with neighboring nodes in the affected subgraph;

[0175] The initial embedding vector is propagated along the edges so that the information introduced by the new data can spread to the neighboring nodes layer by layer through the edges in the graph structure, and update the embedding vectors of each node in the affected subgraph.

[0176] The spread proceeds in a layer-by-layer manner from the center outwards.

[0177] The initial embedding vector is passed along the edge to the one-hop neighbor node, and the embedding vector of the one-hop neighbor node is updated. A one-hop neighbor node is a node that has a direct edge connection with the unmatched node.

[0178] For each one-hop neighbor node, it receives the initial embedding vector from the unmatched node as input information, and this input information is transmitted in the following manner:

[0179] An unmatched node sends its initial embedding vector directly to the one-hop neighbor node along the edge connecting the unmatched node and the one-hop neighbor node;

[0180] After receiving the incoming information, the one-hop neighbor node merges the incoming information with its own current embedding vector to generate the updated embedding vector of the one-hop neighbor node.

[0181] The fusion operation uses a neighborhood aggregation method, which specifically adds the vector value contained in the input information to the current embedded vector value of the one-hop neighbor node element by element to obtain the fused vector.

[0182] Then, a nonlinear transformation is performed on the fused vector to obtain the embedding vector updated by the one-hop neighborhood nodes;

[0183] The current embedding vector of a one-hop neighbor node refers to the embedding vector value of that one-hop neighbor node in the existing graph before the start of this propagation.

[0184] The updated embedding vector of a one-hop neighbor node simultaneously encodes the original structural feature information of that one-hop neighbor node and the impact information brought about by the addition of unmatched nodes;

[0185] The updated embedding vector of the one-hop neighbor node is passed along the edge to the two-hop neighbor node, and the embedding vector of the two-hop neighbor node is updated.

[0186] A two-hop neighbor node is a node that is connected to an unmatched node by exactly two edges. That is, a two-hop neighbor node has a direct edge connection with a certain hop neighbor node, but no direct edge connection with the unmatched node.

[0187] After the one-hop neighbor nodes are updated, the information contained in the initial embedding vector of the unmatched node has been integrated into the updated embedding vector of the one-hop neighbor nodes.

[0188] For each two-hop neighbor node, the one-hop neighbor node that has a direct edge connection with it sends the updated embedding vector of the one-hop neighbor node to the two-hop neighbor node along the edge connecting the one-hop neighbor node and the two-hop neighbor node.

[0189] After receiving the incoming information, the two-hop neighbor node merges the incoming information with its own current embedding vector to generate the updated embedding vector of the two-hop neighbor node.

[0190] The fusion operation is the same as the update method of the one-hop neighbor node, that is, the neighborhood aggregation method is used to add the vector value contained in the incoming information to the current embedded vector value of the two-hop neighbor node element by element to obtain the fused vector.

[0191] Then, a nonlinear transformation is performed on the fused vector to obtain the embedding vector updated by the two-hop neighborhood nodes;

[0192] The current embedding vector of a two-hop neighbor node refers to the embedding vector value of that two-hop neighbor node in the existing graph before the start of this propagation.

[0193] The updated embedding vector of the two-hop neighbor node simultaneously encodes the original structural feature information of the two-hop neighbor node and the influence information of the newly added unmatched nodes passed from the one-hop neighbor node.

[0194] For any receiving node, its updated embedding vector is equal to the sum of the following two terms applied by the activation function:

[0195] The first term is the current embedding vector of the receiving node itself multiplied by the self-loop weight matrix;

[0196] The second term is the embedding vector sent by the sending node multiplied by the relation weight matrix;

[0197] Among them, the current embedding vector of the receiving node is determined before the propagation begins, the embedding vector sent by the sending node is the updated embedding vector of the sending node, the self-loop weight matrix and the relation weight matrix are fixed parameter matrices that have been determined during the training phase of the relation graph convolutional network, and the activation function is the modified linear unit.

[0198] For a one-hop neighbor node, the receiving node is an unmatched node, and the embedding vector sent by the sending node is the initial embedding vector of the unmatched node.

[0199] For a two-hop neighbor node, the receiving node is a one-hop neighbor node, and the embedding vector sent by the sending node is the updated embedding vector of the one-hop neighbor node.

[0200] The propagation range does not exceed the boundary of the affected subgraph. The boundary of the affected subgraph refers to the outer perimeter of the subgraph region determined in step S2, which is centered on the unmatched nodes and the associated edges involved in the incremental relationship set, and extends outward along the edges in the existing graph by a preset number of hops. In this invention, the preset number of hops is 2 hops, therefore the boundary of the affected subgraph is the outermost edge of the two-hop neighborhood nodes;

[0201] The propagation terminates at the two-hop neighborhood node, and the updated embedding vector of the two-hop neighborhood node is no longer propagated to the three-hop and more distant neighborhood nodes.

[0202] Nodes outside the affected subgraph do not receive any incoming information, and their embedding vectors remain unchanged;

[0203] Specifically, through the above-described layer-by-layer propagation method, the information contained in the initial embedding vector of the unmatched node is sequentially transmitted along the edges of the affected subgraph to the one-hop neighbor node and the two-hop neighbor node. The embedding vectors of the one-hop neighbor node and the two-hop neighbor node are updated sequentially. This propagation process only involves nodes within the affected subgraph and does not exceed the boundary of the affected subgraph, thus achieving effective diffusion of information within a local range and local updating of node embedding vectors.

[0204] Further, in step S4, based on the updated embedding vector, a local update is performed on the vector index corresponding to the affected subgraph in the dynamic graph database. This vector index is an HNSW index or an IVF index. HNSW index and IVF index are two widely used approximate nearest neighbor search index structures in vector databases, which are used to accelerate the similarity retrieval of high-dimensional vector data.

[0205] HNSW index is a vector index based on a hierarchical graph structure. HNSW index constructs a multi-level graph structure, where each level is a navigable small-world network graph, and different levels are connected through the mapping relationship between nodes of the upper and lower levels.

[0206] At the lowest level, the HNSW index contains the embedding vectors of all nodes in the dataset;

[0207] As the process progresses upwards layer by layer, the number of nodes in each layer gradually decreases, with the upper layer containing only a portion of the nodes sampled from the lower layer.

[0208] The upper-layer nodes are fewer in number but have a wider connection range, providing fast jumps over long distances;

[0209] The lower-level nodes are numerous and have more specific connections, providing fine-grained local search capabilities;

[0210] The process of building an HNSW index includes:

[0211] Use all the embedded vectors to be indexed as the lowest-level node set;

[0212] Sampling proceeds layer by layer from the bottom layer upwards. At each layer, a portion of nodes are randomly selected from the lower-level nodes according to a preset sampling ratio to form the upper-level node set.

[0213] For each node in each layer, edge connections are established based on the vector distance between the node and other nodes in the same layer. The maximum number of edges connected to each node is controlled by a preset connectivity parameter.

[0214] When performing vector retrieval in the HNSW index, it starts from a preset entry node at the top level, performs a greedy search at the current level, and proceeds downwards layer by layer until it reaches the bottom level and outputs the final retrieval result.

[0215] IVF index is a vector index based on clustering and inverted structure. The basic principle of IVF index is to divide all the embedding vectors to be indexed into multiple subsets, each subset is called a cluster, and each cluster is represented by a cluster center vector.

[0216] IVF indexes associate cluster center vectors with all embedding vectors belonging to a cluster by constructing an inverted index. The construction process of IVF indexes includes the following three stages.

[0217] The first stage is the cluster center training stage:

[0218] The K-means clustering algorithm is run on all embedded vectors to be indexed, dividing the high-dimensional vector space into a predetermined number of clusters. Each cluster generates a cluster center vector, and all cluster center vectors constitute a cluster center table.

[0219] The K-means clustering algorithm executes by randomly selecting a predetermined number of vectors from all the embedding vectors to be indexed as initial cluster centers.

[0220] Calculate the vector distance from each embedding vector to all cluster centers, and assign each embedding vector to the cluster corresponding to the nearest cluster center;

[0221] Then, for each cluster, calculate the average of all embedding vectors within that cluster as the new cluster center;

[0222] Repeat the above allocation and update steps until the position of the cluster center no longer changes significantly or the preset maximum number of iterations is reached;

[0223] The second stage is the vector allocation stage:

[0224] Each embedding vector to be indexed is assigned to the cluster corresponding to the cluster center with the closest vector distance, forming an inverted index. The inverted index records the identifiers of all embedding vectors belonging to each cluster and their storage location information.

[0225] The third stage is the vector compression coding stage:

[0226] Compressed encoding is performed on the embedding vectors within each cluster to reduce storage space and accelerate computation;

[0227] In step S4, a local update is performed on the vector index corresponding to the affected subgraph in the dynamic graph database. When the vector index is an HNSW index, the specific method for performing a local update on the HNSW index is as follows:

[0228] Extract the updated embedding vectors of each node in the affected subgraph and locate the nodes corresponding to each node in the affected subgraph at the lowest level of the HNSW index.

[0229] For existing matching nodes within the affected subgraph, replace the embedding vector value of the corresponding node at the lowest level of the HNSW index with the updated embedding vector of that node.

[0230] For unmatched nodes within the affected subgraph, a new node corresponding to the unmatched node is added at the bottom layer of the HNSW index, and the updated embedding vector of the unmatched node is used as the embedding vector value of the new node.

[0231] In each upper layer of the HNSW index except the bottom layer, the node sampling relationship of each upper layer is used to determine whether there is a corresponding node in each upper layer for each node in the affected subgraph. If there is, the embedding vector value of the corresponding node is replaced with the updated embedding vector. If there is, no operation is performed.

[0232] Based on the updated embedding vectors of each node in the affected subgraph and the vector distances between them and other nodes in the same layer, the edge connection relationships related to each node in the affected subgraph in each layer of the HNSW index are recalculated and updated.

[0233] In the HNSW index, all nodes corresponding to nodes outside the affected subgraph, their embedding vector values, and edge connections remain unchanged;

[0234] When the vector index is an IVF index, the specific method for performing a local update on the IVF index is as follows:

[0235] Extract the updated embedding vectors of each node in the affected subgraph, calculate the vector distance between each updated embedding vector and the cluster center vectors in the IVF index, and assign each updated embedding vector to the cluster corresponding to the cluster center with the closest vector distance.

[0236] For existing matching nodes within the affected subgraph, locate the original cluster allocation record of the node in the inverted table of the IVF index, remove the node from the original cluster, and add the updated embedding vector and its identifier of the node to the inverted table entry corresponding to the newly allocated cluster.

[0237] For unmatched nodes within the affected subgraph, add the node's identifier and its updated embedding vector to the inverted index of the IVF index, and add it to the inverted index entry corresponding to the newly assigned cluster.

[0238] Based on the updated embedding vectors of each node in the affected subgraph, the cluster center vectors of the clusters to which each updated embedding vector belongs are recalculated, and the updated cluster center vectors are written into the corresponding entries in the cluster center table of the IVF index.

[0239] Preferably, when recalculating the cluster center vectors of the clusters to which each updated embedding vector belongs, a locally constrained cluster update strategy is adopted:

[0240] Cluster center updates are performed only on the clusters to which nodes within the affected subgraphs belong, rather than updating all clusters; for any given cluster, if there is at least one node within the affected subgraph, then that cluster is identified as the cluster to be updated.

[0241] For each cluster to be updated, obtain the updated embedding vectors of all current nodes in the cluster, calculate the arithmetic mean of all updated embedding vectors in the cluster, use the arithmetic mean as the new cluster center vector of the cluster, and write it into the corresponding entry of the cluster in the cluster center table of the IVF index.

[0242] If a node in the affected subgraph moves from an original cluster to a new cluster after reallocation, both the original cluster and the new cluster are designated as clusters to be updated, and their cluster center vectors are recalculated. If a cluster does not contain any nodes from the affected subgraph, its cluster center vector remains unchanged, and no calculation or write operation is performed.

[0243] In the IVF index, the cluster assignment records and cluster center vectors of all embedding vectors corresponding to nodes outside the affected subgraph remain unchanged;

[0244] After the partial update operation of the HNSW index or IVF index is completed, the vector index corresponding to the affected subgraph in the dynamic graph database remains synchronized with the updated embedding vector;

[0245] The HNSW index achieves high-precision vector retrieval through its hierarchical graph structure, while the IVF index achieves low-memory vector retrieval through clustering and inverted index structures.

[0246] Both types of indexes can perform local updates based on the updated embedding vectors in step S4, avoiding the reconstruction of the full vector index, thereby reducing the index maintenance overhead during the incremental update process of the dynamic graph database.

[0247] Furthermore, in step S4, a local update is performed on the vector indexes in the dynamic graph database corresponding to the affected subgraphs based on the updated embedding vectors. The core principle of this local update is that only the vector index entries of nodes within the affected subgraphs are recalculated and written, while the vector index entries of nodes outside the affected subgraphs remain unchanged.

[0248] The vector index in the dynamic graph database maintains the mapping relationship between the embedding vectors of all nodes in the graph and their corresponding storage locations;

[0249] A vector index entry is the basic storage unit in a vector index. Each vector index entry records the identifier of the node corresponding to the entry and the embedding vector of that node.

[0250] When the embedding vector of a node in the graph is updated, the corresponding vector index entry of that node needs to be updated synchronously to maintain data consistency.

[0251] The nodes within the affected subgraph refer to all nodes contained in the affected subgraph determined in step S2, including unmatched nodes, one-hop neighbor nodes, and two-hop neighbor nodes;

[0252] An unmatched node is a new entity introduced by new data that does not have a corresponding node in the existing graph. This node does not yet have a corresponding index entry in the vector index.

[0253] One-hop neighborhood nodes and two-hop neighborhood nodes are nodes that already exist in the existing graph, and the corresponding index entries for these nodes already exist in the vector index.

[0254] When performing a local update, the vector index entries of nodes within the affected subgraph need to be recalculated and written. This operation specifically includes the following steps:

[0255] The first step is to determine the position of the index entry corresponding to each node in the affected subgraph in the vector index. For one-hop and two-hop neighbor nodes, locate the corresponding index entry in the vector index according to the node's identifier.

[0256] For unmatched nodes, there is no corresponding index entry for the node in the vector index yet, and a new index entry position needs to be allocated for the node in the vector index;

[0257] The second step is to obtain the updated embedding vectors of each node in the affected subgraph. The updated embedding vectors refer to the embedding vector values ​​obtained after propagation and updating in step S3.

[0258] The third step is to recalculate the vector value recorded in the index entry based on the updated embedding vector of the node, replace the original vector value of the index entry with the updated embedding vector value, and write the updated index entry into the corresponding storage location in the vector index.

[0259] For unmatched nodes, create a new index entry based on the updated embedding vector of the node, record the node's identifier and the updated embedding vector value, and write the new index entry to the newly allocated storage location in the vector index;

[0260] The vector index entries of nodes outside the affected subgraph remain unchanged. Nodes outside the affected subgraph refer to all nodes not included in the affected subgraph as determined in step S2, that is, all nodes in the existing graph located outside the boundary of the affected subgraph.

[0261] The nodes in this part did not participate in the propagation and update of the embedded vector in step S3, and their embedded vector values ​​did not change during this incremental update. Therefore, the vector index entries corresponding to these nodes do not need to be recalculated or written, and the original index entry content and storage location remain unchanged.

[0262] When the vector index is an HNSW index, the vector index entries for nodes within the affected subgraph include the corresponding node entries for that node in each layer of the HNSW index and the edge connection records associated with that node in each layer. The recalculation and write operations include:

[0263] Update the embedding vector values ​​of each node in the affected subgraph at the lowest level;

[0264] Update the embedding vector values ​​of each node in the affected subgraph in each upper layer;

[0265] Recalculate the edge connection relationships between each node in the affected subgraph and other nodes in the same layer, and update the edge connection records. The node entries and edge connection records of nodes outside the affected subgraph remain unchanged in each layer.

[0266] When the vector index is an IVF index, the vector index entries of nodes in the affected subgraph include the cluster assignment record of the node in the inverted index and the cluster center vector of the cluster to which the node belongs;

[0267] Recalculation and write operations include:

[0268] Calculate the vector distance from the updated embedding vector of each node in the affected subgraph to the vector of each cluster center, and redetermine the cluster assignment of each node;

[0269] Write the updated cluster allocation record into the corresponding entry in the inverted index;

[0270] The cluster center vector of each cluster is recalculated based on the updated embedding vector of each node in the reassigned cluster, and the updated cluster center vector is written into the cluster center table.

[0271] For nodes outside the affected subgraph, the cluster assignment records and cluster center vectors in the inverted index remain unchanged;

[0272] By limiting the scope of recalculation and write operations to vector index entries of nodes within the affected subgraph, a full reconstruction of the vector index is avoided.

[0273] The full reconstruction of the vector index requires recalculating the index entries for the embedding vectors of all nodes in the dynamic graph database and writing them to all storage locations. The computational and writing volume increases linearly with the size of the graph.

[0274] This invention only recalculates and writes the index entries of nodes within the affected subgraph. The size of the affected subgraph is much smaller than the total size of the graph, which significantly reduces the number of entries and the amount of data written involved in the index maintenance operation, and reduces the input / output overhead and computational resource consumption during the incremental update process of the dynamic graph database.

[0275] Furthermore, after step S4, this method also includes an index pre-construction step, which is executed independently of the incremental update process and is used to pre-complete the establishment of vector indexes and caches before the query request arrives, thereby reducing query response latency.

[0276] Based on the historical access frequency and node growth rate of each subgraph in the dynamic graph database, predict the hot subgraph regions within the future time window;

[0277] The graph in the dynamic graph database is divided into multiple subgraph regions, each containing a set of interconnected nodes and their associated edges;

[0278] The method of dividing the subgraph region is consistent with the method of determining the affected subgraph in step S2, that is, it is divided based on the connectivity relationship in the graph structure, and each subgraph region contains a set of nodes that are interconnected in the topology;

[0279] Historical access frequency refers to the statistical value of the number of times each sub-graph area has been queried and accessed within a past time window;

[0280] The dynamic graph database continuously records the subgraph region identifiers involved in each query operation during operation and stores these identifiers in the query log along with the query timestamp;

[0281] Extract the number of visits to each subgraph region within each historical time window from the query log to form a time series of the access frequency of each subgraph region.

[0282] The node growth rate refers to the ratio of the number of new nodes added to each subgraph region within a past time window to the original number of nodes in that subgraph region.

[0283] The dynamic graph database records the subgraph region identifier to which the newly added node belongs during each incremental update process, and stores the identifier in the update log in association with the newly added timestamp;

[0284] Extract the number of new nodes for each sub-graph region in each historical time window from the update log, and divide the number of new nodes by the total number of nodes in the sub-graph region at the beginning of the time window to obtain the node growth rate of the sub-graph region in the time window.

[0285] For each subgraph region, obtain the historical access frequency sequence and node growth rate sequence of that subgraph region over a predetermined number of time windows. Input the historical access frequency sequence and node growth rate sequence of that subgraph region into the prediction model, and the prediction model outputs the predicted access frequency and predicted node growth rate of that subgraph region in future time windows.

[0286] The prediction model uses a linear regression model, which fits the trend of access frequency and node growth rate over time based on historical data, and obtains the predicted value of future time windows through trend extrapolation.

[0287] For each subgraph region, the predicted access frequency and predicted node growth rate of the subgraph region are weighted and summed to obtain the popularity score of the subgraph region.

[0288] The weights for access frequency and node growth rate are pre-set positive numbers, and their sum is 1. Sub-graph regions whose popularity scores exceed the preset popularity threshold are identified as hot sub-graph regions.

[0289] Pre-build vector indexes and caches for hotspot subgraph regions. Pre-build means that the vector indexes and caches are built in advance before the query request arrives, rather than being built temporarily when the query request arrives.

[0290] The specific method for pre-establishing vector indices for hotspot subgraph regions is as follows:

[0291] Extract the current embedding vector of each node within the hotspot subgraph region, and construct the local vector index of the hotspot subgraph region based on the current embedding vector of each node within the hotspot subgraph region;

[0292] This local vector index is independent of the full vector index of the dynamic graph database and only covers nodes within the hotspot subgraph region;

[0293] When the vector index is an HNSW index, the local HNSW index of the hotspot subgraph region is constructed using the current embedding vector of each node within the hotspot subgraph region as input.

[0294] When the vector index is an IVF index, the current embedding vector of each node in the hotspot subgraph region is used as input to perform clustering and inverted index construction on the embedding vectors in the hotspot subgraph region, generating a local IVF index for the hotspot subgraph region;

[0295] Local vector indexes are stored in the index storage area of ​​the dynamic graph database and are marked with the hotspot subgraph region identifier corresponding to the local vector index;

[0296] The specific method for pre-caching hotspot subgraph regions is as follows:

[0297] The embedding vectors of each node within the hotspot subgraph region and the relationships between nodes within the hotspot subgraph region are loaded into the cache of the dynamic graph database.

[0298] The cache is a storage area in a dynamic graph database used to temporarily store frequently accessed data, and its read speed is higher than that of the main storage area.

[0299] After being loaded into the cache, when a query request involves a node within a hotspot subgraph region, the embedding vector and association relationship of that node are read directly from the cache without accessing the main storage area.

[0300] The index pre-construction step is repeated at a preset time period. At the beginning of each period, the historical access frequency and node growth rate of each subgraph region are re-acquired, the hot subgraph regions in the future time window are re-predicted, and the local vector index and cache content are updated for the re-determined hot subgraph regions.

[0301] Local vector indices created in the previous period but no longer belonging to the hotspot subgraph region in the current period are removed from the index storage area, and the corresponding cached content is removed from the cache area;

[0302] Specifically, through the index pre-construction step, the vector index and cache of the hot subgraph region are ready before the query request arrives. When the query request involves the hot subgraph region, the vector retrieval can be performed directly using the established local vector index, and the node data can be read directly from the cache without waiting for the index construction and cache loading process, thus reducing query response latency.

[0303] Furthermore, this method also includes a query cache update step, which is executed after the dynamic graph database write operation is completed in step S5;

[0304] The dynamic graph database includes a query cache, which is used to temporarily store historical query requests and their corresponding query results.

[0305] When the dynamic graph database receives a query request, it first checks the query cache to see if there is already a cached query result. If there is, it directly returns the cached query result without performing a complete vector retrieval and graph traversal operation.

[0306] Each cached entry in the query cache contains the identifier information of the query request, the query result data corresponding to the query request, and the status flag of the cached entry;

[0307] Retrieve the cache entries in the query cache corresponding to the nodes within the affected subgraph. Nodes within the affected subgraph refer to all nodes contained in the affected subgraph determined in step S2, including unmatched nodes, one-hop neighbor nodes, and two-hop neighbor nodes. The specific method for retrieving these entries is as follows:

[0308] Iterate through all cached entries in the query cache area, parse the query result data for each cached entry, and extract all node identifiers involved in the query result data;

[0309] The extracted node identifiers are compared one by one with the identifiers of each node in the affected subgraph;

[0310] If the query result data of a cache entry contains the identifier of at least one node in the affected subgraph, then the cache entry is identified as the cache entry corresponding to the node in the affected subgraph.

[0311] Mark the cached entry as pending update. Pending update is one of the status markers for a cached entry, indicating that the query result data recorded in the cached entry is invalid at the current moment and needs to be recalculated. The specific method of marking is as follows:

[0312] The status flag field of the cached entry is changed from valid to pending update. This flagging operation does not delete the cached entry, modify the query request identification information in the cached entry, or modify the original query result data in the cached entry. It only changes the value of the status flag field.

[0313] After the marking operation, when the dynamic graph database receives a query request with the same query request identifier information as the cached entry, it reads the status flag field of the cached entry and identifies it as pending update. Then, it will not directly return the original query result data in the cached entry, but waits for the recalculation to be completed before returning it.

[0314] After the affected subgraph is updated, the cache entries are recalculated and written. The update of the affected subgraph is completed when the dynamic graph database writing operation in step S5 is completed, the updated embedding vector and the updated vector index have been successfully written to the dynamic graph database, and the graph data and index data in the dynamic graph database are in a consistent state.

[0315] The specific method for recalculation is as follows:

[0316] For each cached entry marked as pending update, obtain the query request identifier information recorded in the cached entry, determine the query conditions and query parameters involved in the query request based on the query request identifier information, and re-execute the query operation corresponding to the query request using the updated dynamic graph database as the execution object to generate updated query result data.

[0317] The query operation includes a vector retrieval operation and a graph traversal operation. The vector retrieval operation is performed based on the updated vector index written in step S5, and the graph traversal operation is performed based on the updated embedded vector written in step S5.

[0318] The specific method for writing cache entries is as follows:

[0319] The recalculated and updated query result data is written to the query result data field of the cache entry, overwriting the original query result data in the cache entry;

[0320] The status flag field of the cache entry is changed from pending update to valid. After the write operation is completed, the cache entry records query result data consistent with the updated dynamic graph database, which can be directly read and used by subsequent query requests.

[0321] No operation is performed on the cache entries in the query cache that correspond to nodes outside the affected subgraph. The status flag of the cache entries remains valid, and the query result data recorded by them remains unchanged. This is because the nodes involved in these cache entries have not changed during this incremental update, and their query result data is consistent with the updated dynamic graph database.

[0322] Specifically, through the query cache update step, cache entries in the query cache involving nodes within the affected subgraph are recalculated and written after the affected subgraph is updated, ensuring that the data in the query cache is consistent with the latest state of the dynamic graph database. Cache entries in the query cache that do not involve nodes within the affected subgraph continue to retain their original content without recalculation. This step ensures cache data consistency while limiting the cache update operation to the affected scope, avoiding the computational overhead caused by a full refresh of the query cache.

[0323] Furthermore, this method also includes a graph incremental update step, which continues to run after the dynamic graph database has completed an incremental construction, in response to the arrival of subsequent new data;

[0324] When new data is collected, repeat steps S1 to S5. Collecting new data means that the dynamic graph database continuously receives new data sent by external systems through the application programming interface or data import channel.

[0325] The new data includes structured description data of application programming interfaces, system configuration file data, and operation log data. Whenever new data arrives at the dynamic graph database, the execution of steps S1 to S5 is automatically triggered.

[0326] The complete execution process of steps S1 to S5 is as follows:

[0327] Step S1 extracts entities and relationships from the newly added data, generates an incremental entity set and an incremental relationship set, and writes the incremental entity set and incremental relationship set into the incremental cache of the dynamic graph database according to the timestamp.

[0328] Step S2 compares each entity in the incremental entity set with the nodes of the existing graph in the dynamic graph database to determine the matching nodes and unmatched nodes. Taking the unmatched nodes and the associated edges involved in the incremental relationship set as the center, it expands outward along the edges in the existing graph by a preset number of hops, and determines the subgraph area covered by the expansion as the affected subgraph.

[0329] Step S3: Keep the embedding vectors of nodes outside the affected subgraph in the existing graph unchanged, perform node embedding calculation on the unmatched nodes in the affected subgraph to generate initial embedding vectors, propagate the initial embedding vectors along the edges of the affected subgraph to the neighboring nodes, and update the embedding vectors of each node in the affected subgraph.

[0330] Step S4 performs a local update on the vector indexes in the dynamic graph database corresponding to the affected subgraphs based on the updated embedding vectors;

[0331] Step S5 writes the updated embedding vector and the updated vector index into the dynamic graph database;

[0332] The updated entities and relationships are incrementally written into the dynamic graph database to generate the updated graph. The updated entities refer to the entities corresponding to the unmatched nodes introduced in this new data. These entities did not exist in the dynamic graph database before this incremental update.

[0333] The updated relationships refer to the relationships included in the incremental relationship set introduced in this new data, which did not exist in the dynamic graph database before this incremental update;

[0334] The updated entity is written as a new node to the graph storage area of ​​the dynamic graph database. A new node identifier is assigned to the entity in the graph storage area, and the attribute information of the entity is recorded.

[0335] The updated association is written as a new edge to the graph storage area of ​​the dynamic graph database. A new edge identifier is assigned to the association in the graph storage area, and the type of the association, the target node identifier, and the source node identifier are recorded.

[0336] After the writing is completed, the existing graph in the dynamic graph database, together with the newly written entities and relationships, constitute the updated graph;

[0337] Records the operation log for each incremental update. The operation log is a sequentially written persistent record used to record detailed information about each incremental update operation in the dynamic graph database. Each incremental update operation corresponds to one operation log record, which contains the following information:

[0338] The operation type for this incremental update operation is incremental write operation;

[0339] The timestamp of this incremental update operation, i.e., the start time of the operation;

[0340] The identification information of the affected subgraphs involved in this incremental update operation, namely the list of node identifiers and the list of edge identifiers contained in the affected subgraphs;

[0341] The embedding vector values ​​of each node in the affected subgraph before this incremental update operation was performed;

[0342] The updated embedded vector values ​​of each node in the affected subgraph after this incremental update operation;

[0343] The contents of the vector index entries corresponding to the affected subgraphs in the dynamic graph database before this incremental update operation was performed;

[0344] After this incremental update operation is executed, the content of the vector index entries corresponding to the affected subgraphs in the dynamic graph database will be updated.

[0345] Operation log records are stored in the log storage area of ​​the dynamic graph database in an append-only manner. Once written, operation log records cannot be modified or deleted.

[0346] Operation logs support version tracing and rollback. Version tracing refers to tracing back the data state of the dynamic graph database to any historical point in time based on the timestamp information in the operation log. The specific method of version tracing is as follows:

[0347] Determine the target time point to be traced, filter out all operation log records with timestamps earlier than or equal to the target time point from the log storage area, and read the affected subgraph identification information, the embedded vector value before the update, and the vector index entry content before the update recorded in each operation log record in the order of timestamp from earliest to latest, and restore the dynamic graph database data snapshot corresponding to the target time point;

[0348] Rollback refers to restoring the current data state of a dynamic graph database to the data state at a specified historical point in time. The specific rollback method is as follows:

[0349] Determine the target time point to be rolled back. Filter all operation log records with timestamps later than the target time point from the log storage area. Read the affected subgraph identifier information, the embedded vector value before the update, and the vector index entry content before the update from each operation log record in reverse order of timestamp from latest to earliest. Restore the embedded vector of each node in the affected subgraph to the embedded vector value before the update. Restore the vector index entry corresponding to each affected subgraph to the vector index entry content before the update. Delete the entities and relationships newly added within the rollback range in the graph storage area. Restore the entities and relationships updated within the rollback range in the graph storage area to their state before the update.

[0350] After the rollback is completed, the data state of the dynamic graph database is restored to the state at the target time point;

[0351] Specifically, through the graph incremental update steps, when new data arrives, the dynamic graph database automatically repeats steps S1 to S5 to complete the incremental construction, and persists the updated entities and relationships to the graph storage area. The recording of operation logs makes each incremental update operation traceable and rollbackable, ensuring the data reliability and data consistency of the dynamic graph database during the incremental update process.

[0352] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0353] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. A method for incremental construction and index update of a dynamic graph database based on local recomputation, characterized in that, Includes the following steps: S1, collect new data, including structured description data of application programming interfaces, system configuration file data, and operation log data; The newly added data is subjected to entity extraction and relation extraction to generate an incremental entity set and an incremental relation set; The incremental entity set and the incremental relation set are written into the incremental cache of the dynamic graph database according to the timestamp; S2, compare each entity in the incremental entity set with the nodes of the existing graph in the dynamic graph database to determine matching nodes and unmatched nodes; Centered on the unmatched node and the associated edge involved in the incremental relationship set, extend outward along the edge of the existing graph by a preset number of hops, and determine the subgraph region covered by the extension as the affected subgraph; S3, keep the embedding vectors of nodes in the existing graph that are outside the affected subgraph unchanged; For unmatched nodes within the affected subgraph, perform node embedding calculations to generate initial embedding vectors; The initial embedding vector is propagated along the edges of the affected subgraph to neighboring nodes, and the embedding vector of each node in the affected subgraph is updated. S4, perform a local update on the vector index corresponding to the affected subgraph in the dynamic graph database based on the updated embedding vector; S5. Write the updated embedding vector and the updated vector index into the dynamic graph database to complete the incremental construction.

2. The method according to claim 1, characterized in that, The preset number of jumps in step S2 is 2 jumps.

3. The method according to claim 1, characterized in that, Step S2, which involves expanding outwards along the edges of the existing graph by a preset number of hops, centered on the unmatched node and the associated edges involved in the incremental relationship set, and determining the subgraph region covered by the expansion as the affected subgraph, specifically includes: Starting from the unmatched node, traverse the one-hop neighbor nodes along the associated edge; Continue traversing the two-hop neighbor nodes along the edge from the one-hop neighbor node; The subgraph formed by the unmatched node, the one-hop neighbor node, the two-hop neighbor node, and the edges connecting the above nodes is considered as the affected subgraph.

4. The method according to claim 1, characterized in that, In step S3, node embedding calculation is performed on the unmatched nodes in the affected subgraph, and an initial embedding vector is generated using a relational graph convolutional network. The relational graph convolutional network generates the initial embedding vector by aggregating the features of the unmatched nodes and their neighboring nodes.

5. The method according to claim 1, characterized in that, Step S3, which involves propagating the initial embedding vector along the edges of the affected subgraph to neighboring nodes, specifically includes: The initial embedding vector is passed along the edge to the one-hop neighbor node, and the embedding vector of the one-hop neighbor node is updated. The updated embedding vector of the one-hop neighbor node is passed along the edge to the two-hop neighbor node, and the embedding vector of the two-hop neighbor node is updated. The propagation range does not exceed the boundary of the affected subgraph.

6. The method according to claim 1, characterized in that, The vector index mentioned in step S4 is either an HNSW index or an IVF index.

7. The method according to claim 1, characterized in that, Step S4, which involves performing a local update on the vector index corresponding to the affected subgraph in the dynamic graph database, specifically includes: Only the vector index entries of nodes within the affected subgraph are recalculated and written; the vector index entries of nodes outside the affected subgraph remain unchanged.

8. The method according to claim 1, characterized in that, It also includes the index pre-build step: Based on the historical access frequency and node growth rate of each subgraph in the dynamic graph database, predict the hot subgraph regions within the future time window; Vector indexes and caches are pre-established for the hotspot subgraph regions.

9. The method according to claim 1, characterized in that, It also includes the query cache update step: Retrieve the cache entry in the query cache that corresponds to the node in the affected subgraph, and mark the cache entry as pending update; After the affected subgraph is updated, the cache entry is recalculated and written.

10. The method according to claim 1, characterized in that, It also includes the incremental map update step: When new data is collected, repeat steps S1 to S5; The updated entities and relationships are incrementally written into the dynamic graph database to generate the updated graph. The operation log for each incremental update is recorded, and version traceability and rollback are supported.

Images