A method, device, and medium for managing HNSW index data

Abstract

The application discloses a kind of HNSW index data management method, device, equipment and medium, comprising: obtaining the target node corresponding to the transaction to be executed, according to the transaction to be executed, target node corresponding storage location is operated, generates the target redo log corresponding to target operation, target undo log and the version transaction data of the target node for target operation of the transaction to be executed, before the transaction to be executed is completed, the line lock is executed to the corresponding storage location in target node, according to the transaction to be executed, the neighbor relationship of neighbor node and target node is updated, generates the neighbor redo log corresponding to update operation and / or neighbor undo log, before the transaction to be executed is completed, neighbor relationship of neighbor node is not limited to access.The HNSW index data is decoupled and classified, and a differentiated transaction processing mechanism is used, a good balance is achieved between ensuring global strong consistency and reducing neighbor relationship transaction management overhead, effectively balancing performance and consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a HNSW index data management method and device, equipment and medium. BACKGROUND

[0002] With the development of intelligent applications, vector databases have become a key infrastructure for processing unstructured data such as images, speech, and text embeddings. Hierarchical Navigable Small World (HNSW) has become a mainstream vector indexing algorithm due to its efficient retrieval performance. However, integrating HNSW into a transaction-enabled database system presents significant challenges: the unordered and dynamically updated nature of HNSW graph structures fundamentally conflicts with the ordered and stable transaction management mechanisms of databases, leading to performance degradation under high concurrency and difficulties in ensuring consistency.

[0003] Prior art, implementation scheme of PGVector (based on PostgreSQL). The technical scheme is: 1) vector data is stored as ordinary table data in PostgreSQL, enjoying complete ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantee; 2) HNSW index is implemented through the index access method interface of PostgreSQL, and is persisted through Write-Ahead Logging (WAL) together with the data; 3) when the source table data is modified, the associated HNSW index content (including nodes and connection relationships) is updated in the background after the source data transaction is successfully committed. The index itself does not support multiple versions, and its consistency is guaranteed through the transaction state of the source data.

[0004] There are still certain limitations for this technical scheme: 1) HNSW index without transaction management: the HNSW index data of PGVector does not support transaction management, any change to HNSW index data will be immediately visible, and its transaction visibility depends on the transaction management capability of the source data table, and fine-grained and incremental transaction management cannot be performed at the index level; 2) unable to efficiently handle deletions: after deleting data, "zombie nodes" in the index cannot be immediately cleaned up, and the entire index needs to be manually rebuilt to maintain retrieval accuracy and performance, which cannot meet the continuity requirements of online businesses; 3) essentially "eventual consistency": the update of the index lags behind the transaction submission of the source data, and during the time window from the transaction submission to the completion of the index update, the latest data cannot be seen by the query, which does not strictly meet the strong consistency requirement of isolation (such as read committed).

[0005] In actual application scenarios, a user needs to perform high-concurrency data addition, deletion, modification and query operations in a vector database, and requires that these operations meet the strong consistency (ACID) transaction standard of a traditional relational database, while guaranteeing high performance and low latency of vector retrieval. The current mainstream scheme cannot simultaneously meet the three requirements of strong consistency, high-concurrency update and high-performance retrieval.

[0006] Therefore, how to manage HNSW index data, support incremental update, guarantee strong consistency of operations, and avoid performance bottlenecks caused by lock competition and full-amount log recording has become a problem to be solved. SUMMARY

[0007] Embodiments of the present application provide a HNSW index data management method, device, equipment and medium to solve the problem of how to manage HNSW index data, support incremental update, guarantee strong consistency of operations, and avoid performance bottlenecks caused by lock competition and full-amount log recording.

[0008] A HNSW index data management method comprises:

[0009] An acquisition module is configured to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node.

[0010] A node data management module is configured to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log and version transaction data of the to-be-executed transaction performing the target operation on the target node, and perform a row lock on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0011] The node data management module is further configured to perform an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation, and not limit access to the neighbor relationship of the neighbor node before the to-be-executed transaction is completed.

[0012] A HNSW index data management device comprises:

[0013] An acquisition module is configured to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node.

[0014] A node data management module is configured to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log and version transaction data of the to-be-executed transaction performing the target operation on the target node, and perform a row lock on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0015] The neighbor relationship management module is used to update the neighbor relationship between the neighbor node and the target node according to the transaction to be executed, and generate a neighbor redo log and / or a neighbor revocation log corresponding to the update operation. Before the transaction to be executed is completed, access to the neighbor relationship of the neighbor node is not restricted.

[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for managing HNSW index data.

[0017] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for managing HNSW index data.

[0018] The above-mentioned HNSW index data management method obtains the target node corresponding to the transaction to be executed, determines the neighbor nodes of the corresponding target node, performs target operations on the corresponding storage location in the target node according to the transaction to be executed, and generates target redo logs, target undo logs, and version transaction data of the target operations performed by the transaction to be executed on the target node. Before the transaction to be executed is completed, row locks are executed on the corresponding storage location in the target node. According to the transaction to be executed, the neighbor relationship between the neighbor nodes and the target node is updated, and neighbor redo logs and / or neighbor undo logs of the corresponding update operations are generated. Before the transaction to be executed is completed, access to the neighbor relationship of the neighbor nodes is not restricted.

[0019] By decoupling and classifying the HNSW index data, the complex and frequently changing neighbor relationship data updates are decoupled from the stable and easily controllable node data. Different transaction processing mechanisms are used for different data types, thereby achieving compatibility between strong consistency and high-concurrency read / write operations. Specifically, for the node data storing the vector ontology, a mature database strong consistency mechanism is used to ensure its ACID properties. For frequently changing neighbor relationship data, its atomicity, consistency, and isolation are delegated to the transaction mechanism of the node data it depends on. Through this classification and governance approach, a good balance is achieved between ensuring global strong consistency and reducing the overhead of neighbor relationship transaction management (such as the overhead of locking and recording complete rollback logs), effectively balancing performance and consistency. Attached Figure Description

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings described below are only some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative labor based on these drawings are within the scope of protection of the present application.

[0021] Figure 1 is a schematic diagram of an application environment of the HNSW index data management method in an embodiment of the present application.

[0022] Figure 2 is a flowchart of the HNSW index data management method in an embodiment of the present application.

[0023] Figure 3 is another flowchart of the HNSW index data management method in an embodiment of the present application.

[0024] Figure 4 is another flowchart of the HNSW index data management method in an embodiment of the present application.

[0025] Figure 5 is another flowchart of the HNSW index data management method in an embodiment of the present application.

[0026] Figure 6 is another flowchart of the HNSW index data management method in an embodiment of the present application.

[0027] Figure 7 is another flowchart of the HNSW index data management method in an embodiment of the present application.

[0028] Figure 8 is a schematic diagram of the HNSW index data management device in an embodiment of the present application.

[0029] Figure 9 is a schematic diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions of the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0031] The HNSW index data management method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 , including Figure 1As shown in the client and the server, the client and the server communicate through a network, and are used to solve the problem of how to manage HNSW index data, support incremental update, guarantee strong consistency of operation, and avoid performance bottleneck caused by lock competition and full-amount log recording. The client, also called user end, is a program that provides local service for the client corresponding to the server. The client can be installed on, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0032] In an embodiment, as shown in Figure 2 A method for managing HNSW index data is provided, which is applied to the server in 1 and includes the following steps:

[0033] Step S201: Obtain a target node corresponding to a to-be-executed transaction, and determine neighbor nodes of the target node.

[0034] In this embodiment, the HNSW index can refer to a navigable small-world hierarchical graph, which is an index algorithm for high-dimensional vector approximate nearest neighbor search, and has a multi-layer graph structure. In this embodiment, the HNSW index data is physically divided into two types of storage structures: graph node data, which is used to store each vector data itself (node identifier, vector value, metadata, etc.), and the table is in row or column storage structure; neighbor relationship data, which is used to independently store the neighbor relationship between nodes, and each record format can be (source node identifier, target node identifier, layer number). The to-be-executed transaction can refer to a transaction to be performed on the HNSW index data (such as addition, deletion, modification, etc.), the target node can refer to a node in the HNSW index directly operated by the to-be-executed transaction, and the neighbor node can refer to a node having a neighbor relationship with the target node.

[0035] Specifically, when the to-be-executed transaction starts, the transaction manager allocates a transaction identifier for the to-be-executed transaction. If the to-be-executed transaction performs a write operation on the HNSW index data, a graph node is constructed for the newly added vector data according to the to-be-executed transaction, the graph node is the target node corresponding to the write operation of the to-be-executed transaction, and the position of the target node inserted into the HNSW graph and its neighbor nodes in each layer are calculated according to the HNSW algorithm. If the to-be-executed transaction performs a deletion operation on the HNSW index data, the graph node corresponding to the to-be-deleted vector data and its neighbor nodes are located from the HNSW graph according to the to-be-executed transaction, the graph node is the target node corresponding to the deletion operation of the to-be-executed transaction, and the neighbor nodes of the target node are the neighbor nodes of the target node.

[0036] Step S202: performing a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, to generate a target redo log corresponding to the target operation, a target undo log, and version transaction data of the target operation performed by the to-be-executed transaction on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0037] In this embodiment, the target operation can be a data operation performed by the to-be-executed transaction on the target node (such as inserting the target node or deleting the target node), the target redo log can be a redo log used to persistently record the content and execution result of the target operation, to ensure that the target operation can be re-executed when the system recovers from a failure, the target undo log can be an undo log used to record the data state before the target operation is executed, to ensure that the node data can be restored to the state before the operation according to the log when the transaction is rolled back, and the version transaction data can be an independent data version with a timestamp and a transaction identifier created for each modification of the target node based on a multi-version concurrency control (MVCC) mechanism.

[0038] Specifically, a target operation is performed on a corresponding storage location in the target node according to the to-be-executed transaction, a target redo log, a target undo log, and version transaction data of the target node of the target operation are recorded, and a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0039] Step S203: performing an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, to generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation, wherein the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed.

[0040] In this embodiment, the update operation can be a data operation performed by the to-be-executed transaction on the neighbor relationship between the neighbor node and the target node (such as inserting a new neighbor relationship or deleting a neighbor relationship), the neighbor redo log can be a redo log used to persistently record the content and result of the update operation performed on the neighbor relationship data, to ensure that the relationship update can be re-executed when the system recovers from a failure, and the neighbor undo log can be an undo log used to record the state before the update operation is performed on the neighbor relationship data, to ensure that the relationship data can be restored to the state before the operation according to the log when the transaction is rolled back.

[0041] Specifically, after a target operation is performed on the storage location corresponding to the target node according to the to-be-executed transaction, an update operation is performed on the neighbor relationship between the target node and the neighbor node according to the to-be-executed transaction, and a neighbor redo log and / or a neighbor undo log of the update operation are recorded, where the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed, that is, dirty reading of the updated neighbor relationship corresponding to the update operation is allowed before the to-be-executed transaction is completed.

[0042] As shown in Figure 3 An HNSW index data management architecture diagram is provided. The HNSW index data is divided into graph node data and neighbor relationship data. For transaction management of the graph node data, a strong consistency transaction processing mechanism is adopted, redo logs and undo logs corresponding to operations are created in the operation process of the transaction on the node data, the MVCC mechanism is followed, version transaction data of the node data corresponding to the operation of the transaction is generated, and a row lock is performed on the operated node data before the transaction is completed. For the neighbor relationship data, a weak consistency transaction processing mechanism is adopted, redo logs and / or undo logs corresponding to operations are created in the operation process of the transaction on the neighbor relationship data, and a row lock is not performed on the operated neighbor relationship data before the transaction is completed, dirty reading of the neighbor relationship data of the neighbor node is allowed, when the neighbor relationship data needs to be accessed when dirty reading of the neighbor relationship data, if the transaction operation corresponding to the specific node has been committed, the specific node data can be normally accessed, if the transaction operation corresponding to the specific node has not been committed, the data operated by the uncommitted transaction on the specific node cannot be normally accessed, so as to ensure the isolation. In the process of rolling back the index data, the undo log of the graph node data is used to drive the rollback of the neighbor relationship data, that is, in the process of writing rollback of the index data, the specific node data is rolled back through the undo log, the neighbor node data corresponding to the specific node is rolled back based on the rolled back specific node data, in the process of deleting rollback of the index data, the specific node data and the node neighbor relationship data are rolled back through the undo log, and the neighbor node data corresponding to the specific node is rolled back based on the rolled back node neighbor relationship data.

[0043] For example, suppose a real-time risk control system needs to process a large number of transaction behavior vectors with high concurrency and update the vector database in real time to detect fraud. There are three high-concurrency write transactions: Transaction T1 (TxID=101): inserts the behavior vector of transaction A; Transaction T2 (TxID=102): inserts the behavior vector of transaction B; Transaction T3 (TxID=103): updates the behavior vector of an existing transaction C. When building the HNSW index, the newly inserted nodes resulting from the updates in T1, T2, and T3 may be connected by the HNSW algorithm to some existing popular nodes (e.g., a typical fraud pattern vector node X). In traditional models, this leads to intense competition for the connection relationship of node X. This embodiment overcomes the problem of drastic performance drops in high-concurrency writes caused by "global lock contention for connection relationships" and "consistent transaction logs" in other vector database solutions. The specific implementation method adopted can be:

[0044] 1. Concurrent writes in T1, T2, and T3

[0045] Step 1 (Parallel Write Nodes):

[0046] 1) T1 inserts a new node record into the graph node data of HNSW: (Node ID=201, Vector data Vector_1, Transaction txid=101);

[0047] 2) T2 inserts a new node record into the graph node data of HNSW: (Node ID=202, Vector data Vector_2, Transaction txid=102);

[0048] 3) T3 updates the vector value, which is split into two operations on the HNSW index:

[0049] a. Delete the old node; (Node ID=Old_ID, Vector data Vector_3_old, Transaction txid=103);

[0050] b. Insert a new node: (Node ID=203, Vector data Vector_3_new, Transaction txid=103);

[0051] 4) The above operations only involve changes to the node data of the corresponding transaction. The relevant modifications are protected by the MVCC mechanism, and transactions do not block each other.

[0052] 2. Neighbor relationship data is updated in parallel during T1, T2, and T3.

[0053] Step 2 (Parallel insertion and update of neighbor relationship data):

[0054] 1) HNSW algorithm calculates the neighbors of node 201 in T1, inserts the neighbor record of node 201 into each layer, and updates the neighbor relationship data of the neighbor;

[0055] 2) For example, node 201 calculates the neighbors as node X and node Y in layer 0:

[0056] a. First, insert the neighbor relationship data of node 201 into: [(201, X), (201, Y)];

[0057] b. Update the neighbor relationship of neighbor X (assuming that node 201 is closer than other existing neighbors of X) to [(old neighbor data of node X), (X, 201)];

[0058] 3) Node Y does the same: [(old neighbor data of node Y), (Y, 201)];

[0059] 4) At the same time, the HNSW algorithm calculates the neighbors of node 202 in T2, inserts the neighbor record of node 202 into each layer, and updates the neighbor relationship data of the neighbor:

[0060] a. For example, node 202 calculates the neighbors as node X and node Z in layer 0;

[0061] b. First, insert the neighbor relationship data of node 202 into: [(202, X), (202, Z)];

[0062] c. Update the neighbor relationship of neighbor X (assuming that node 202 is closer than other existing neighbors of X) to [(old neighbor data of node X), (X, 201), (X, 202)];

[0063] d. Node Z does the same: [(old neighbor data of node Z), (Z, 202)];

[0064] 5) T3 also has the same processing;

[0065] The above T1 and T2 need to update the neighbor relationship data of the common neighbor X, in the present application, the update operation of the neighbor relationship data does not need to be protected by the MVCC mechanism (i.e. no row lock is applied, and no undo is recorded), only the redo log is recorded, therefore the update of the neighbor relationship data of node X by T1 and T2 is parallel and will not cause blocking and waiting.

[0066] 3. Transaction submission and atomic row guarantee

[0067] Step 3 (submission): T1, T2, and T3 respectively submit their respective transactions. When submitting, the MVCC mechanism updates the transaction state of the insertion and update of the node data in the graph node data in step 1;

[0068] Step 4: If T1 is committed successfully, T2 is failed to roll back:

[0069] 1) When T2 is rolled back, the insertion operation of the node 202 of T2 needs to be rolled back through the undo log of the rollback;

[0070] 2) When the graph node data of the node 202 is rolled back, the neighbor relationship data of the node 202 node that has been written, such as the 0-layer [(202, X), (202, Z)], is deleted (the redo needs to be written, and the uplink lock is not needed);

[0071] 3) When the graph node data of the node 202 is rolled back, the neighbor relationship data of the node X and the node Z needs to be updated, for example, for the 0-layer neighbor relationship data of the node X, the data is [(old neighbor data of the node X), (X, 201), (X, 202)] when rolled back, a new neighbor node N is selected to replace, and the data is updated to [(old neighbor data of the node X), (X, 201), (X, N)];

[0072] In subsequent query transactions, because the node 202 has been rolled back and deleted, the node cannot be queried, and the deleted node is ignored in the graph search algorithm, so the data modified by the uncommitted T2 is not queried, thereby ensuring atomicity.

[0073] In the embodiment, by decoupling and classifying the HNSW index data, the complex and frequently changed neighbor relationship data update is decoupled from the stable and easily controlled node data, different transaction processing mechanisms are used for different data, thereby achieving compatibility of strong consistency and high concurrency read and write. Specifically, for the node data of the storage vector body, a mature database strong consistency mechanism is used to ensure its ACID characteristics, and for the frequently changed neighbor relationship data, its atomicity, consistency and isolation are delegated to the transaction mechanism of the node data attached thereto to ensure. Through this classification management mode, a good balance is achieved between guaranteeing global strong consistency and reducing neighbor relationship transaction management overhead (such as locking, recording complete rollback log overhead), and performance and consistency are effectively balanced.

[0074] In an embodiment, as shown in Figure 4 the management method of the HNSW index data is provided. The step S202 includes the following steps:

[0075] Step S401: If the target operation is a write operation, the target node is written into the corresponding storage location according to the to-be-executed transaction.

[0076] Step S402: generating target redo log, target undo log corresponding to the write operation and version transaction data of the target node by the to-be-executed transaction.

[0077] Specifically, if the target operation is a write operation, the target node is written into the corresponding storage location in the form of a new row according to the to-be-executed transaction, and the target redo log, the target undo log corresponding to the write operation of the target node and the version transaction data of the target node by the to-be-executed transaction are generated in the process following the standard MVCC mechanism of the database.

[0078] According to the to-be-executed transaction, the neighbor relationship of the neighbor node and the target node is updated, and the neighbor redo log and / or the neighbor undo log corresponding to the update operation are generated, including:

[0079] If the target operation is a write operation, the neighbor relationship of the target node and the neighbor node is written into the corresponding storage location of the target node and the corresponding storage location of the neighbor node, respectively, according to the to-be-executed transaction.

[0080] The neighbor redo log for the write operation is generated.

[0081] Specifically, after the target node is written into the corresponding storage location according to the to-be-executed transaction, the neighbor relationship of the target node and the neighbor node is written into the corresponding storage location of the target node in the form of a new row according to the to-be-executed transaction, and the neighbor relationship of the target node and the neighbor node is written into the corresponding storage location of the neighbor node, the neighbor redo log corresponding to the write operation of the neighbor relationship is generated, and the neighbor undo log is not generated, and the version transaction data is not generated, wherein before the to-be-executed transaction is completed, the neighbor relationship of the neighbor node is not limited to be accessed, that is, before the to-be-executed transaction is completed, other transactions are allowed to dirty read the neighbor relationship updated by the write operation, and when the to-be-executed transaction is completed and committed, only the target node data is committed, and the neighbor relationship data does not need to be committed.

[0082] In the embodiment, by classifying the HNSW index data into two categories of graph node data and neighbor relationship data, different transaction processing mechanisms are applied to different characteristics when writing transactions, strong consistency is adopted for transaction operations on target node data, row locks are executed on target node data before transaction completion, target redo logs, target undo logs and version transaction data corresponding to target node write operations are generated, weak consistency is adopted for neighbor relationship data between target node and neighbor node, only neighbor redo logs are generated, dirty reading of the neighbor relationship by other transactions is allowed, so that multiple concurrent transactions can update neighbor relationship data without blocking, thereby eliminating transaction lock competition for frequently changing neighbor relationship data. Through this classification management method, a good balance between global strong consistency and reduction of neighbor relationship transaction management overhead (such as locking and recording complete undo logs) is achieved, effectively balancing performance and consistency.

[0083] In an embodiment, as shown in Figure 5 A method for managing HNSW index data is provided, and the step S202 of generating target redo logs, target undo logs and version transaction data of the target node corresponding to the target operation of the to-be-executed transaction on the target node corresponding storage location according to the to-be-executed transaction includes the following steps:

[0084] Step S501: If the target operation is a delete operation, the target node is deleted from the corresponding storage location according to the to-be-executed transaction.

[0085] Step S502: Generate target redo logs, target undo logs and version transaction data of the target node corresponding to the delete operation of the to-be-executed transaction.

[0086] Specifically, if the target operation is a delete operation, the target node is deleted from the corresponding storage location in the form of deleting a row according to the to-be-executed transaction, this process follows the standard MVCC mechanism of the database, and target redo logs, target undo logs and version transaction data of the target node corresponding to the target operation of the to-be-executed transaction are generated.

[0087] According to the to-be-executed transaction, the neighbor relationship between the neighbor node and the target node is updated, and neighbor redo logs and / or neighbor undo logs corresponding to the update operation are generated, including:

[0088] If the target operation is a delete operation, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the target node according to the to-be-executed transaction;

[0089] generating version transaction data of the target node and the neighbor node in the corresponding storage location of the target node for the delete operation of the neighbor redo log, the neighbor undo log and the to-be-executed transaction on the neighbor relationship between the target node and the neighbor node;

[0090] according to the to-be-executed transaction, marking and deleting the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node;

[0091] determining the number of the marked and deleted neighbor relationships from the corresponding storage location of the neighbor node, and if the number exceeds a threshold, performing an update operation on all the marked and deleted neighbor relationships to generate an updated neighbor relationship for the neighbor node;

[0092] generating the neighbor redo log for the marked and deleted operation and the update operation.

[0093] Specifically, after deleting the target node from the corresponding storage location according to the to-be-executed transaction, deleting the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node according to the to-be-executed transaction, generating the neighbor redo log, the neighbor undo log and the version transaction data of the target node and the neighbor node in the corresponding storage location of the target node for the delete operation of the neighbor relationship between the target node and the neighbor node, marking and deleting the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node according to the to-be-executed transaction, determining the number of the marked and deleted neighbor relationships from the corresponding storage location of the neighbor node, and if the number exceeds a threshold, performing an update operation on all the marked and deleted neighbor relationships to generate an updated neighbor relationship for the neighbor node, and generating the neighbor redo log for the marked and deleted operation and the update operation of the neighbor relationship, wherein before the to-be-executed transaction is completed, there is no limitation on accessing the neighbor relationship of the neighbor node, that is, before the to-be-executed transaction is completed, other transactions are allowed to dirty read the neighbor relationship of the neighbor node corresponding to the update operation, and when the to-be-executed transaction is completed and committed, the target node data and each layer of neighbor relationship data thereof are committed, but the updated neighbor relationship data of the neighbor node of the target node does not need to be committed.

[0094] For example, in the process of updating the neighbor relationship in the corresponding storage location of the neighbor node, the neighbor relationship data of each layer of the target node is traversed, and the neighbor relationship data of each layer of the neighbor node is refreshed: the deleted node (its data contains {neighbor node identifier, isDeleted flag}) is found from the neighbor relationship data of the neighbor node, the isDeleted flag is set to true, and it is checked whether the number of neighbor nodes with isDeleted flag set to true in the neighbor relationship data exceeds a certain proportion. If it exceeds, the neighbor of this layer needs to be selected again for the neighbor node to which the neighbor relationship data belongs. The principle is: keep the existing normal neighbor (i.e. isDeleted flag is false), and for the neighbor with isDeleted flag set to true, check whether the corresponding transaction of the graph node data is committed through the neighbor node identifier. If it is committed, the nearest one is selected from the new neighbor set to replace it. If it is not committed, it is kept.

[0095] In the embodiment, by classifying the HNSW index data into two categories of graph node data and neighbor relationship data, different transaction processing mechanisms are applied according to different characteristics when a transaction is deleted. The transaction operation on the target node data adopts strong consistency, the row lock is executed on the target node data before the transaction is completed, the target redo log, target undo log and version transaction data corresponding to the target node deletion operation are generated, the neighbor relationship data of the target node and the neighbor node adopts weak consistency, the neighbor relationship data of the target node is generated, the neighbor redo log, neighbor undo log and version transaction data corresponding to the neighbor relationship deletion operation of the target node are generated, and only the neighbor redo log is generated for the neighbor relationship data of the neighbor node. Other transactions are allowed to read the neighbor relationship dirty, so that multiple concurrent transactions can update the neighbor relationship data without blocking, thereby eliminating the transaction lock competition for the dynamically changing and frequently changed neighbor relationship data. Through this classification management method, a good balance is achieved between guaranteeing global strong consistency and reducing neighbor relationship transaction management overhead (such as locking and recording complete undo log overhead), and performance and consistency are effectively considered.

[0096] In an embodiment, as shown in Figure 6 a management method of HNSW index data is provided, and the method further includes the following steps:

[0097] Step S601: According to the query transaction, the neighbor node is accessed to obtain the neighbor relationship between the neighbor node and the target node.

[0098] Step S602: According to the neighbor relationship, when accessing the target node, the version transaction data of the target operation of the to-be-executed transaction on the target node is obtained according to the version transaction data, and the execution state of the to-be-executed transaction is determined according to the version transaction data.

[0099] Step S603: If the execution state is uncommitted, access to the corresponding storage location in the target node for the target operation is restricted.

[0100] In this embodiment, the query transaction can refer to a transaction that performs a read operation on the HNSW index data.

[0101] Specifically, the query transaction performs graph traversal based on the HNSW algorithm, and in each layer of traversal, the algorithm reads the neighbor list of the current node from the neighbor relationship data of the entry point. Since the neighbor relationship data allows dirty reading, the query can immediately obtain the latest connection relationship, including from uncommitted write or delete transactions and the relationship updates caused thereby. Therefore, the query can successfully access the neighbor node and obtain the neighbor relationship between the neighbor node and the target node. When further accessing the target node according to the neighbor relationship, the system obtains the version transaction data generated when the target operation is performed on the target node by the to-be-executed transaction, and determines the commit state of the transaction according to the version transaction data. If it is detected that the related transaction has not been committed, access to the corresponding storage location of the target node is restricted. At this time, the data of the target node that has been committed by other transactions in the historical version can be obtained through the MVCC mechanism, so that the data written to the target node or the data of the target node deleted by the to-be-executed transaction can be ignored.

[0102] In this embodiment, by adopting strong consistency for the target node and weak consistency for the neighbor relationship data during execution of the to-be-executed transaction, dirty reading by other transactions is allowed. When querying, the other transactions can not only read the node data of the committed transaction operation, but also ignore the node data of the uncommitted transaction operation when reading the node data of the uncommitted transaction operation based on the neighbor relationship, thereby ensuring isolation and making the result returned by the final query completely composed of committed data.

[0103] In an embodiment, as shown in FIG. 7, a management method of HNSW index data is provided, further comprising the following steps: Figure 7

[0104] Step S701: According to the rollback transaction, the target operation and the update operation are rolled back. If the target operation is a write operation, according to the rollback transaction, the target node is deleted from the corresponding storage location in the target node based on the target undo log, and according to the rollback transaction, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the target node and the corresponding storage location of the neighbor node, respectively.

[0105] ​Step S702: If the target operation is a delete operation, according to the rollback transaction, the target node is written into the corresponding storage location of the target node based on the target undo log, and according to the rollback transaction, the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node is rolled back based on the neighbor undo log, to obtain the neighbor relationship between the rolled-back target node and the neighbor node in the corresponding storage location of the target node.

[0106] Step S703: According to the neighbor relationship between the rolled-back target node and the neighbor node in the corresponding storage location of the target node, the marked deletion of the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node is rolled back.

[0107] In this embodiment, the rollback transaction can refer to a transaction of performing a rollback operation on the HNSW index data.

[0108] Specifically, if the target operation is a write operation, according to the rollback transaction, the target node is deleted from the corresponding storage location based on the target undo log, and according to the rollback transaction, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the neighbor node based on the neighbor relationship between the target node and the neighbor node, the neighbor relationship of the neighbor node is repaired (such as replacing the rolled-back target node with a newly found closer neighbor), and the neighbor relationship between the target node and the neighbor node at the corresponding storage location of the target node is deleted.

[0109] If the target operation is a delete operation, according to the rollback transaction, the target node is written into the corresponding storage location of the target node based on the target undo log, according to the rollback transaction, the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node is rolled back based on the neighbor undo log, to obtain the neighbor relationship between the rolled-back target node and the neighbor node in the corresponding storage location of the target node, based on the rolled-back neighbor relationship, the marked deletion of the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node is rolled back (the number of neighbor relationships marked for deletion corresponding to the neighbor node does not exceed the threshold, i.e., the neighbor node does not perform neighbor relationship update), or the neighbor relationship between the target node and the neighbor node is written into the corresponding storage location of the neighbor node (the number of neighbor relationships marked for deletion corresponding to the neighbor node exceeds the threshold, i.e., the neighbor node has performed neighbor relationship update).

[0110] In this embodiment, by adopting strong consistency for the target node and weak consistency for the neighbor relationship data during the execution of the to-be-executed transaction, when rolling back, the strong consistency mechanism of the target node data is used to reverse the neighbor relationship data of the target node and its neighbors when the target node data is rolled back to ensure the strong consistency of the HNSW graph data, thereby achieving the compatibility of strong consistency and high concurrency read and write.

[0111] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0112] In an embodiment, a HNSW index data management apparatus is provided, applied to a server in Figure 1 The HNSW index data management apparatus corresponds to the HNSW index data management method in the above embodiments. As shown in Figure 8 The HNSW index data management apparatus includes an acquisition module 81, a node data management module 82, and a neighbor relationship management module 83. The functions of each module are described in detail as follows:

[0113] The acquisition module 81 is configured to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node;

[0114] The node data management module 82 is configured to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log, and version transaction data of the to-be-executed transaction performing the target operation on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed;

[0115] The neighbor relationship management module 83 is configured to perform an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation, wherein the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed.

[0116] Optionally, the node data management module 82 includes:

[0117] A node writing unit is configured to, if the target operation is a write operation, write the target node into the corresponding storage location according to the to-be-executed transaction;

[0118] A first recording unit is configured to generate the target redo log corresponding to the write operation, the target undo log, and the version transaction data of the to-be-executed transaction performing the write operation on the target node.

[0119] Optionally, the neighbor relationship management module 83 includes:

[0120] A relationship writing unit is configured to, if the target operation is a write operation, write the neighbor relationship between the target node and the neighbor node into the corresponding storage location of the target node and the corresponding storage location of the neighbor node according to the to-be-executed transaction;

[0121] A second recording unit is configured to generate the neighbor redo log for the write operation.

[0122] Optionally, the node data management module 82 comprises:

[0123] The node deletion unit is configured to, if the target operation is a deletion operation, delete the target node from the corresponding storage location according to the to-be-executed transaction;

[0124] The third recording unit is configured to generate the target redo log corresponding to the deletion operation, the target undo log, and the version transaction data of the to-be-executed transaction performing the deletion operation on the target node.

[0125] Optionally, the neighbor relationship management module 83 comprises:

[0126] The first relationship deletion unit is configured to, if the target operation is a deletion operation, delete the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node according to the to-be-executed transaction;

[0127] The fourth recording unit is configured to generate the neighbor redo log corresponding to the deletion operation, the neighbor undo log, and the version transaction data of the to-be-executed transaction performing the deletion operation on the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node.

[0128] The second relationship deletion unit is configured to, according to the to-be-executed transaction, mark and delete the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node.

[0129] The relationship updating unit is configured to determine the number of marked and deleted neighbor relationships from the corresponding storage location of the neighbor node, and if the number exceeds a threshold, perform an updating operation on all the marked and deleted neighbor relationships to generate an updated neighbor relationship for the neighbor node.

[0130] The fifth recording unit is configured to generate the neighbor redo log corresponding to the marked deletion operation and the updating operation.

[0131] Optionally, the HNSW index data management apparatus further comprises:

[0132] The query module is configured to access the neighbor node according to a query transaction to obtain the neighbor relationship between the neighbor node and the target node.

[0133] The node access module is configured to, according to the neighbor relationship, obtain the version transaction data of the to-be-executed transaction performing the target operation on the target node when accessing the target node, and determine the execution state of the to-be-executed transaction according to the version transaction data.

[0134] The access limiting module is configured to, if the execution state is uncommitted, limit the access to the corresponding storage location in the target node performing the target operation.

[0135] Optionally, the HNSW index data management apparatus further comprises:

[0136] a write rollback module configured to, according to the rollback transaction, rollback the target operation and the update operation, and if the target operation is a write operation, delete the target node from the corresponding storage location of the target node based on the target undo log according to the rollback transaction, and delete the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node and the corresponding storage location of the neighbor node according to the rollback transaction;

[0137] a deletion rollback module configured to, if the target operation is a deletion operation, write the target node into the corresponding storage location of the target node based on the target undo log according to the rollback transaction, and rollback the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node based on the neighbor undo log according to the rollback transaction to obtain the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback;

[0138] a label rollback module configured to rollback the label deletion of the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node according to the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback.

[0139] The specific limitations of the HNSW index data management apparatus can be referred to the limitations of the HNSW index data management method in the foregoing, which will not be repeated here. Each module in the HNSW index data management apparatus described above can be realized by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0140] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 9 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store target nodes corresponding to transactions to be executed. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a HNSW index data management method.

[0141] In an embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the method for managing HNSW index data in the above embodiments when executing the computer program, for example Figure 2 as shown in S201-S203, or Figures 3 to 7 as shown, which will not be repeated here. Alternatively, the processor implements the functions of the modules / units in this embodiment of the data management apparatus when executing the computer program, for example, Figure 8 as shown in the functions of the obtaining module 81, the node data management module 82, and the neighbor relationship management module 83, which will not be repeated here.

[0142] In an embodiment, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program being executable on a processor to implement the method for managing HNSW index data in the above embodiments, for example Figure 2 as shown in S201-S203, or Figures 3 to 7 as shown, which will not be repeated here. Alternatively, the processor implements the functions of the modules / units in this embodiment of the data management apparatus when executing the computer program, for example, Figure 8 as shown in the functions of the obtaining module 81, the node data management module 82, and the neighbor relationship management module 83, which will not be repeated here.

[0143] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above embodiments. Any reference to memory, storage, database, or other medium in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM).

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0145] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for managing HNSW index data, characterized in that, include: Obtain the target node corresponding to the transaction to be executed, and determine the neighboring nodes corresponding to the target node; Based on the transaction to be executed, a target operation is performed on the corresponding storage location in the target node, generating a target redo log, a target undo log, and version transaction data of the transaction to be executed performing the target operation on the target node, wherein a row lock is executed on the corresponding storage location in the target node before the transaction to be executed is completed. Based on the pending transaction, the neighbor relationship between the neighbor node and the target node is updated, and a neighbor redo log and / or neighbor reversal log corresponding to the update operation are generated. Before the pending transaction is completed, access to the neighbor relationship of the neighbor node is not restricted.

2. The method for managing HNSW index data according to claim 1, characterized in that, The step of performing a target operation on the corresponding storage location in the target node according to the transaction to be executed, and generating a target redo log, a target undo log, and version transaction data of the target operation performed by the transaction to be executed on the target node, includes: If the target operation is a write operation, then the target node is written to the corresponding storage location according to the transaction to be executed; Generate the target redo log, target undo log, and version transaction data of the transaction to be executed for the write operation on the target node, corresponding to the write operation.

3. The method for managing HNSW index data according to claim 1, characterized in that, The step of updating the neighbor relationship between the neighbor node and the target node according to the transaction to be executed, and generating a neighbor redo log and / or a neighbor reversal log corresponding to the update operation, includes: If the target operation is a write operation, then according to the transaction to be executed, the neighbor relationship between the target node and the neighbor node is written to the corresponding storage location of the target node and the corresponding storage location of the neighbor node, respectively. Generate a neighbor redo log for the write operation.

4. The method for managing HNSW index data according to claim 1, characterized in that, The step of performing a target operation on the corresponding storage location in the target node according to the transaction to be executed, and generating a target redo log, a target undo log, and version transaction data of the target operation performed by the transaction to be executed on the target node, includes: If the target operation is a deletion operation, then the target node is deleted from the corresponding storage location according to the transaction to be executed; Generate the target redo log, target undo log, and version transaction data of the transaction to be executed for the deletion operation on the target node, corresponding to the deletion operation.

5. The method for managing HNSW index data according to claim 1, characterized in that, The step of updating the neighbor relationship between the neighbor node and the target node according to the transaction to be executed, and generating a neighbor redo log and / or a neighbor reversal log corresponding to the update operation, includes: If the target operation is a deletion operation, then according to the transaction to be executed, the neighbor relationship between the target node and the neighboring nodes is deleted from the corresponding storage location of the target node; Generate neighbor redo logs, neighbor reversal logs, and version transaction data for the deletion operation, which are stored in the corresponding storage location of the target node to perform the deletion operation on the neighbor relationship between the target node and the neighbor nodes. Based on the transaction to be executed, the neighbor relationship between the target node and the neighbor node is marked and deleted at the corresponding storage location of the neighbor node; From the corresponding storage location of the neighbor node, determine the number of neighbor relationships that have been marked for deletion. If the number exceeds the threshold, perform an update operation on all neighbor relationships that have been marked for deletion to generate updated neighbor relationships for the neighbor node. Generate neighbor redo logs for the mark deletion operation and the update operation.

6. The method for managing HNSW index data according to claim 1, characterized in that, The management method also includes: Based on the query transaction, the neighbor nodes are accessed to obtain the neighbor relationship between the neighbor nodes and the target node; Based on the neighbor relationship, when accessing the target node, the version transaction data of the transaction to be executed performing the target operation on the target node is obtained, and the execution status of the transaction to be executed is determined based on the version transaction data. If the execution status is "not committed", then access to the corresponding storage location in the target node performing the target operation is restricted.

7. The method for managing HNSW index data according to claim 1, characterized in that, The management method also includes: According to the rollback transaction, the target operation and the update operation are rolled back. If the target operation is a write operation, according to the rollback transaction and based on the target reversal log, the target node is deleted from the corresponding storage location of the target node. According to the rollback transaction, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the target node and the corresponding storage location of the neighbor node, respectively. If the target operation is a deletion operation, then according to the rollback transaction and based on the target revocation log, the target node is written into the corresponding storage location of the target node, and according to the rollback transaction and based on the neighbor revocation log, the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node is rolled back to obtain the rolled-back neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node. Based on the neighbor relationship between the target node and the neighboring node in the corresponding storage location of the target node after rollback, the deletion of the marker for the neighbor relationship between the target node and the neighboring node is rolled back in the corresponding storage location of the neighboring node.

8. A management device for HNSW index data, characterized in that, include: The acquisition module is used to acquire the target node corresponding to the transaction to be executed and determine the neighboring nodes corresponding to the target node; The node data management module is used to perform target operations on the corresponding storage location in the target node according to the transaction to be executed, and generate target redo logs, target undo logs and version transaction data of the transaction to be executed performing the target operation on the target node. Before the transaction to be executed is completed, row locks are executed on the corresponding storage location in the target node. The neighbor relationship management module is used to update the neighbor relationship between the neighbor node and the target node according to the transaction to be executed, and generate a neighbor redo log and / or a neighbor revocation log corresponding to the update operation. Before the transaction to be executed is completed, access to the neighbor relationship of the neighbor node is not restricted.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the HNSW index data management method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the HNSW index data management method as described in any one of claims 1 to 7.

Images