HNSW index data management method and device, equipment and medium

Abstract

The invention discloses an HNSW index data management method and device, equipment and a medium, and the method comprises the steps: obtaining a target node corresponding to a to-be-executed transaction, and carrying out the target operation of a storage position corresponding to the target node according to the to-be-executed transaction, and generating a target redo log and a target revocation log corresponding to the target operation and version transaction data of the to-be-executed transaction for performing the target operation on the target node, before the to-be-executed transaction is completed, executing a row lock on a corresponding storage position in the target node, and according to the to-be-executed transaction, performing an updating operation on a neighbor relationship between the neighbor node and the target node. And generating a neighbor redo log and / or a neighbor revocation log corresponding to the update operation, and before the to-be-executed transaction is completed, not limiting the access to the neighbor relationship of the neighbor node. The HNSW index data is decoupled and classified, and a differentiated transaction processing mechanism is adopted, so that good balance is achieved between guarantee of global strong consistency and reduction of neighbor relationship transaction management overhead, and performance and consistency are effectively considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a HNSW index data management method and device, equipment and medium. BACKGROUND

[0002] With the development of intelligent applications, vector databases have become a key infrastructure for processing unstructured data such as images, speech, and text embeddings. Hierarchical Navigable Small World (HNSW) has become a mainstream vector indexing algorithm due to its efficient retrieval performance. However, integrating HNSW into a transaction-enabled database system presents significant challenges: the unordered and dynamically updated nature of HNSW graph structures fundamentally conflicts with the ordered and stable transaction management mechanisms of databases, leading to performance degradation under high concurrency and difficulties in ensuring consistency.

[0003] Prior art, implementation scheme of PGVector (based on PostgreSQL). Its technical scheme is: 1) vector data is stored as ordinary table data in PostgreSQL, enjoying complete ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantee; 2) HNSW index is implemented through the index access method interface of PostgreSQL, and is persisted through Write-Ahead Logging (WAL) together with the data; 3) when the source table data is added, deleted or modified, the source data transaction is successfully submitted, and the associated HNSW index content (including nodes and connection relationships) is updated in the background. The index itself does not support multiple versions, and its consistency is guaranteed through the transaction state of the source data.

[0004] There are still some limitations for this technical scheme: 1) HNSW index without transaction management: the HNSW index data of PGVector does not support transaction management, any change of HNSW index data will be immediately visible, and its transaction visibility depends on the transaction management capability of the source data table, which cannot be fine-tuned and incrementally managed at the index level; 2) unable to efficiently handle deletions: after deleting data, "zombie nodes" in the index cannot be immediately cleaned up, and the entire index needs to be manually rebuilt to maintain retrieval accuracy and performance, which cannot meet the continuity requirements of online business; 3) essentially "eventual consistency": the update of the index lags behind the transaction submission of the source data, and during the time window from the transaction submission to the completion of the index update, the latest data cannot be seen by the query, which does not strictly meet the strong consistency requirement of isolation (such as read committed).

[0005] In actual application scenarios, users need to perform high-concurrency data addition, deletion, modification and query operations in a vector database, and require that these operations meet the strong consistency (ACID) transaction standard of traditional relational databases, while ensuring high performance and low latency of vector retrieval. The current mainstream solution cannot simultaneously meet the three requirements of strong consistency, high-concurrency update and high-performance retrieval.

[0006] Therefore, how to manage HNSW index data, support incremental updates, guarantee the strong consistency of operations, and avoid performance bottlenecks caused by lock competition and full-amount log recording has become a problem to be solved. SUMMARY

[0007] Embodiments of the present application provide a HNSW index data management method, device, equipment and medium to solve the problem of how to manage HNSW index data, support incremental updates, guarantee the strong consistency of operations, and avoid performance bottlenecks caused by lock competition and full-amount log recording.

[0008] A HNSW index data management method comprises: acquiring a target node corresponding to a to-be-executed transaction, and determining a neighbor node corresponding to the target node; performing a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generating a target redo log corresponding to the target operation, a target undo log and version transaction data of the to-be-executed transaction performing the target operation on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed; performing an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generating a neighbor redo log and / or a neighbor undo log corresponding to the update operation, wherein the neighbor relationship of the neighbor node is not limited in access before the to-be-executed transaction is completed.

[0009] A HNSW index data management device comprises: an acquisition module configured to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node; a node data management module configured to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log and version transaction data of the to-be-executed transaction performing the target operation on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed; A neighbor relationship management module is configured to perform an update operation on the neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation.

[0010] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the HNSW index data management method when executing the computer program.

[0011] A computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the HNSW index data management method.

[0012] The HNSW index data management method includes obtaining a target node corresponding to a to-be-executed transaction, determining a neighbor node corresponding to the target node, performing a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generating a target redo log and a target undo log corresponding to the target operation and version transaction data of the target operation performed on the target node by the to-be-executed transaction, performing a row lock on the corresponding storage location in the target node before the to-be-executed transaction is completed, performing an update operation on the neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generating a neighbor redo log and / or a neighbor undo log corresponding to the update operation, and not limiting access to the neighbor relationship of the neighbor node before the to-be-executed transaction is completed.

[0013] The HNSW index data is decoupled and classified, the complex and frequently changing neighbor relationship data update is decoupled from the stable and easily controlled node data, different transaction processing mechanisms are used for different data, and therefore, the compatibility of strong consistency and high concurrency read and write is achieved. Specifically, for the node data storing the vector ontology, a mature database strong consistency mechanism is used to ensure the ACID characteristics, and for the frequently changed neighbor relationship data, the atomicity, consistency and isolation are delegated to the transaction mechanism of the node data attached to the neighbor relationship data to ensure the atomicity, consistency and isolation. Through this classification management mode, a good balance is achieved between guaranteeing global strong consistency and reducing neighbor relationship transaction management overhead (such as locking and recording complete undo log overhead), and performance and consistency are effectively balanced. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative labor based on these drawings are within the protection scope of the present application.

[0015] Figure 1 is a schematic diagram of an application environment of the HNSW index data management method in an embodiment of the present application. Figure 2 is a flowchart of the HNSW index data management method in an embodiment of the present application. Figure 3 is another flowchart of the HNSW index data management method in an embodiment of the present application. Figure 4 is another flowchart of the HNSW index data management method in an embodiment of the present application. Figure 5 is another flowchart of the HNSW index data management method in an embodiment of the present application. Figure 6 is another flowchart of the HNSW index data management method in an embodiment of the present application. Figure 7 is another flowchart of the HNSW index data management method in an embodiment of the present application. Figure 8 is a schematic diagram of the HNSW index data management device in an embodiment of the present application. Figure 9 is a schematic diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor based on these embodiments are within the protection scope of the present application.

[0017] The HNSW index data management method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 , including Figure 1As shown in the client and the server, the client and the server communicate through a network, and are used to solve the problem of how to manage HNSW index data, support incremental update, guarantee strong consistency of operation, and avoid performance bottleneck caused by lock competition and full-amount log recording. The client, also called user end, is a program that provides local service for the client corresponding to the server. The client can be installed on, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0018] In an embodiment, as shown in Figure 2 A method for managing HNSW index data is provided, which is applied to the server in 1 and includes the following steps: Step S201: Obtain a target node corresponding to a to-be-executed transaction, and determine neighbor nodes of the target node.

[0019] In this embodiment, the HNSW index can refer to a navigable small-world hierarchical graph, which is an index algorithm for high-dimensional vector approximate nearest neighbor search, and has a multi-layer graph structure. In this embodiment, the HNSW index data is physically divided into two types of storage structures: graph node data, which is used to store each vector data itself (node identifier, vector value, metadata, etc.), and the table is in row or column storage structure; neighbor relationship data, which is used to independently store the neighbor relationship between nodes, and each record format can be (source node identifier, target node identifier, layer number). The to-be-executed transaction can refer to a transaction to be performed on the HNSW index data (such as addition, deletion, modification, etc.), the target node can refer to a node in the HNSW index directly operated by the to-be-executed transaction, and the neighbor node can refer to a node having a neighbor relationship with the target node.

[0020] Specifically, when the to-be-executed transaction starts, the transaction manager allocates a transaction identifier for the to-be-executed transaction. If the to-be-executed transaction performs a write operation on the HNSW index data, a graph node is constructed for the newly added vector data according to the to-be-executed transaction, the graph node is the target node corresponding to the write operation of the to-be-executed transaction, the position of the target node inserted into the HNSW graph and its neighbor nodes in each layer are calculated according to the HNSW algorithm; if the to-be-executed transaction performs a deletion operation on the HNSW index data, the graph node corresponding to the to-be-deleted vector data and its neighbor nodes are located from the HNSW graph according to the to-be-executed transaction, the graph node is the target node corresponding to the deletion operation of the to-be-executed transaction, and the neighbor nodes are the neighbor nodes of the target node.

[0021] Step S202: According to the to-be-executed transaction, a target operation is performed on a corresponding storage location in the target node, and a target redo log, a target undo log, and version transaction data of the target operation of the to-be-executed transaction are generated, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0022] In this embodiment, the target operation can be a data operation (such as inserting the target node or deleting the target node) performed by the to-be-executed transaction on the target node, the target redo log can be a redo log used for persistently recording the content and execution result of the target operation, ensuring that the target operation can be re-executed when the system recovers from a failure, the target undo log can be an undo log used for recording the data state before the target operation is executed, ensuring that the node data can be restored to the state before the operation according to the log when the transaction is rolled back, and the version transaction data can be an independent data version with a timestamp and a transaction identifier created for each modification of the target node based on a multi-version concurrency control (MVCC) mechanism.

[0023] Specifically, according to the to-be-executed transaction, a target operation is performed on a corresponding storage location in the target node, and a target redo log, a target undo log, and version transaction data of the target node are recorded, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed.

[0024] Step S203: According to the to-be-executed transaction, an update operation is performed on the neighbor relationship between the neighbor node and the target node, and a neighbor redo log and / or a neighbor undo log corresponding to the update operation are generated, wherein the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed.

[0025] In this embodiment, the update operation can be a data operation (such as inserting a new neighbor relationship or deleting a neighbor relationship) performed by the to-be-executed transaction on the neighbor relationship between the neighbor node and the target node, the neighbor redo log can be a redo log used for persistently recording the content and result of the update operation performed on the neighbor relationship data, ensuring that the relationship update can be re-executed when the system recovers from a failure, and the neighbor undo log can be an undo log used for recording the state before the update operation is performed on the neighbor relationship data, ensuring that the relationship data can be restored to the state before the operation according to the log when the transaction is rolled back.

[0026] Specifically, after a target operation is performed on the storage location corresponding to the target node according to the to-be-executed transaction, an update operation is performed on the neighbor relationship between the target node and the neighbor node according to the to-be-executed transaction, and a neighbor redo log and / or a neighbor undo log of the update operation are recorded, where the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed, that is, dirty reading of the updated neighbor relationship corresponding to the update operation is allowed before the to-be-executed transaction is completed.

[0027] As shown in Figure 3 An HNSW index data management architecture diagram is provided. The HNSW index data is divided into graph node data and neighbor relationship data. For transaction management of the graph node data, a strong consistency transaction processing mechanism is adopted, redo logs and undo logs corresponding to operations are created in the operation process of the transaction on the node data, the MVCC mechanism is followed, version transaction data of the node data corresponding to the operation of the transaction is generated, and a row lock is performed on the operated node data before the transaction is completed. For the neighbor relationship data, a weak consistency transaction processing mechanism is adopted, redo logs and / or undo logs corresponding to operations are created in the operation process of the transaction on the neighbor relationship data, and a row lock is not performed on the operated neighbor relationship data before the transaction is completed, dirty reading of the neighbor relationship data of the neighbor node is allowed, when the neighbor relationship data needs to be accessed when dirty reading of the neighbor relationship data, if the transaction operation corresponding to the specific node has been committed, the specific node data can be normally accessed, if the transaction operation corresponding to the specific node has not been committed, the data operated by the uncommitted transaction on the specific node cannot be normally accessed, so as to ensure the isolation. In the process of rolling back the index data, the undo log of the graph node data is used to drive the rollback of the neighbor relationship data, that is, in the process of writing rollback of the index data, the specific node data is rolled back through the undo log, the neighbor node data corresponding to the specific node is rolled back based on the rolled back specific node data, in the process of deleting rollback of the index data, the specific node data and the node neighbor relationship data are rolled back through the undo log, and the neighbor node data corresponding to the specific node is rolled back based on the rolled back node neighbor relationship data.

[0028] For example, assume that in a real-time fraud detection system, a large number of transaction behavior vectors need to be processed in high concurrency, and the vector database needs to be updated in real time to detect fraud. At this time, there are three high-concurrency write transactions: transaction T1 (TxID = 101): insert a transaction A behavior vector; transaction T2 (TxID = 102): insert a transaction B behavior vector; transaction T3 (TxID = 103): update a transaction C behavior vector. When building the HNSW index, the newly inserted nodes caused by T1, T2 and T3 updates may be connected to some existing popular nodes (for example, a typical fraud pattern vector node X) by the HNSW algorithm, which will cause fierce competition for the connection relationship of node X in the traditional model. The embodiment overcomes the problem of high-concurrency write performance degradation caused by "connection relationship global lock competition" and "consistency transaction log" in other vector databases and the like, and the specific implementation manner can be as follows: 1. T1, T2, T3 concurrent write Step 1 (parallel write node): 1) T1 inserts a new node record into the graph node data of HNSW: (node ID = 201, vector data Vector_1, transaction txid = 101); 2) T2 inserts a new node record into the graph node data of HNSW: (node ID = 202, vector data Vector_2, transaction txid = 102); 3) T3 updates the vector value and splits into two operations on the HNSW index: a. delete the old node; (node ID = Old_ID, vector data Vector_3_old, transaction txid = 103); b. insert a new node: (node ID = 203, vector data Vector_3_new, transaction txid = 103); 4) The above operations only involve node data changes of the corresponding transactions, and the related modifications are protected by the MVCC mechanism, and the transactions do not block each other.

[0029] 2. T1, T2, T3 parallel update neighbor relationship data Step 2 (parallel insert and update neighbor relationship data): 1) The HNSW algorithm calculates the neighbors of node 201 of T1 for each layer, inserts the neighbor records of node 201 into each layer, and updates the neighbor relationship data of the neighbors; 2) For example, node 201 calculates the neighbors as node X and node Y in layer 0: a. First, insert the neighbor relationship data of node 201: [(201, X), (201, Y)]; b. Update the neighbor relation of neighbor X (if node 201 is closer than other neighbors of X) to [(old neighbor data of node X), (X, 201)]; 3) Node Y does the same: [(old neighbor data of node Y), (Y, 201)]; 4) At the same time, the HNSW algorithm calculates the neighbors of node 202 for T2, inserts the neighbor records of node 202 into the layers, and updates the neighbor relation data of the neighbors: a. For example, node 202 calculates the neighbors as node X and node Z in layer 0; b. First, insert the neighbor relation data of node 202: [(202, X), (202, Z)]; c. Update the neighbor relation of neighbor X (if node 202 is closer than other neighbors of X) to [(old neighbor data of node X), (X, 201), (X, 202)]; d. Node Z does the same: [(old neighbor data of node Z), (Z, 202)]; 5) T3 also has the same processing; The above T1 and T2 need to update the neighbor relation data of the common neighbor X, in the present application, the update operation of the neighbor relation data does not need to be protected by the MVCC mechanism (i.e. no row lock is applied, and no undo is recorded), only the redo log is recorded, therefore the update of the neighbor relation data of node X by T1 and T2 is parallel and will not cause blocking and waiting.

[0030] 3. Transaction submission and atomic row guarantee Step 3 (submission): T1, T2, and T3 respectively submit their respective transactions. When submitting, the MVCC mechanism is used to update the transaction state of the insertion and update of the node data in the graph node data in step 1; Step 4: If T1 is successfully submitted, T2 fails to roll back: 1) When T2 rolls back, the undo log of the rollback is used to roll back the insertion operation of node 202 of T2; 2) When the graph node data of node 202 is rolled back, for the neighbor relation data of node 202 that has been written, for example, [(202, X), (202, Z)] in layer 0, deletion is performed (the redo needs to be written, and no uplink lock is needed); 3) In the graph node data rollback of node 202, the neighbor relationship data of node X and node Z needs to be updated, for example, for the 0-layer neighbor relationship data of node X, the rollback data is [(old neighbor data of node X), (X, 201), (X, 202)], a new neighbor node N is selected to replace, and the update is [(old neighbor data of node X), (X, 201), (X, N)]; In subsequent query transactions, because node 202 has been rolled back and deleted, the node cannot be queried, and the deleted nodes are ignored in the graph search algorithm, so that the data modified by uncommitted T2 is not queried, thereby ensuring atomicity.

[0031] In the embodiment, by decoupling and classifying the HNSW index data, the complex and frequently changing neighbor relationship data update is decoupled from the stable and easily controlled node data, different transaction processing mechanisms are used for different data, thereby achieving compatibility of strong consistency and high concurrency read and write. Specifically, for the node data storing the vector body, a mature database strong consistency mechanism is used to ensure its ACID characteristics, and for the frequently changed neighbor relationship data, its atomicity, consistency and isolation are delegated to the transaction mechanism of the node data attached thereto to ensure. Through this classification management method, a good balance is achieved between ensuring global strong consistency and reducing neighbor relationship transaction management overhead (such as locking, recording complete rollback log overhead), effectively balancing performance and consistency.

[0032] In an embodiment, as shown in Figure 4 The management method of the HNSW index data comprises the following steps: Step S401: If the target operation is a write operation, the target node is written into the corresponding storage location according to the to-be-executed transaction.

[0033] Step S402: The target redo log corresponding to the write operation, the target undo log and the version transaction data of the to-be-executed transaction on the target node are generated.

[0034] Specifically, if the target operation is a write operation, the target node is written into the corresponding storage location in the form of a new row according to the to-be-executed transaction, and this process follows the standard MVCC mechanism of the database, and the target redo log corresponding to the write operation of the target node, the target undo log and the version transaction data of the to-be-executed transaction on the target node are generated.

[0035] According to the to-be-executed transaction, a neighbor update operation is performed on the neighbor relationship between the neighbor node and the target node, a neighbor redo log and / or a neighbor undo log corresponding to the neighbor update operation are generated, including: If the target operation is a write operation, according to the to-be-executed transaction, the neighbor relationship between the target node and the neighbor node is written into the corresponding storage location of the target node and the corresponding storage location of the neighbor node, respectively. The neighbor redo log for the write operation is generated.

[0036] Specifically, after the target node is written into the corresponding storage location according to the to-be-executed transaction, the neighbor relationship between the target node and the neighbor node is written into the corresponding storage location of the target node in the form of a new row, and the neighbor relationship between the target node and the neighbor node is written into the corresponding storage location of the neighbor node according to the to-be-executed transaction, a neighbor redo log corresponding to the neighbor relationship write operation is generated, no neighbor undo log is generated, and no version transaction data is generated. Before the to-be-executed transaction is completed, there is no restriction on accessing the neighbor relationship of the neighbor node, that is, before the to-be-executed transaction is completed, other transactions are allowed to dirty read the neighbor relationship updated by the write operation, and when the to-be-executed transaction is completed and committed, only the target node data is committed, and there is no need to commit the neighbor relationship data.

[0037] In this embodiment, by classifying the HNSW index data into two categories of graph node data and neighbor relationship data, different transaction processing mechanisms are applied to different characteristics during transaction writing. The transaction operation on the target node data adopts strong consistency, a row lock is performed on the target node data before the transaction is completed, a target redo log, a target undo log and version transaction data corresponding to the target node write operation are generated, the neighbor relationship data between the target node and the neighbor node adopts weak consistency, only a neighbor redo log is generated, other transactions are allowed to dirty read the neighbor relationship, so that multiple concurrent transactions can update the neighbor relationship data without blocking, thereby eliminating the transaction lock competition for the neighbor relationship data which changes frequently and dynamically. Through this classification management method, a good balance is achieved between guaranteeing global strong consistency and reducing neighbor relationship transaction management overhead (such as locking and recording complete undo log overhead), and performance and consistency are effectively taken into account.

[0038] In an embodiment, as shown in Figure 5 A method for managing HNSW index data is provided. In step S202, according to the to-be-executed transaction, a target operation is performed on the corresponding storage location in the target node, a target redo log, a target undo log corresponding to the target operation, and version transaction data of the target operation performed by the to-be-executed transaction on the target node are generated, including the following steps: Step S501: If the target operation is a delete operation, the target node is deleted from the corresponding storage location according to the to-be-executed transaction.

[0039] Step S502: generating target redo log, target undo log corresponding to the delete operation and version transaction data of the target node by the to-be-executed transaction.

[0040] Specifically, if the target operation is a delete operation, the target node is deleted from the corresponding storage location in the form of deleting a row according to the to-be-executed transaction, and the process follows the standard MVCC mechanism of the database, and generates target redo log, target undo log corresponding to the node deletion operation and version transaction data of the target node by the to-be-executed transaction.

[0041] According to the to-be-executed transaction, the neighbor relationship between the neighbor node and the target node is updated, and the neighbor redo log and / or the neighbor undo log corresponding to the update operation are generated, including: If the target operation is a delete operation, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the target node according to the to-be-executed transaction; Generating neighbor redo log, neighbor undo log for the delete operation and version transaction data of the to-be-executed transaction in the corresponding storage location of the target node for the delete operation of the neighbor relationship between the target node and the neighbor node; According to the to-be-executed transaction, the neighbor relationship between the target node and the neighbor node is marked for deletion in the corresponding storage location of the neighbor node; From the corresponding storage location of the neighbor node, the number of marked deleted neighbor relationships is determined, and if the number exceeds the threshold, all the marked deleted neighbor relationships are updated, and the updated neighbor relationship of the neighbor node is generated; Generating neighbor redo log for the marked delete operation and the update operation.

[0042] Specifically, after deleting the target node from the corresponding storage location according to the to-be-executed transaction, the neighbor relationship between the target node and the neighbor node is deleted from the corresponding storage location of the target node according to the to-be-executed transaction, a neighbor redo log corresponding to the neighbor relationship deletion operation at the target node is generated, a neighbor undo log is generated, and version transaction data of the to-be-executed transaction in the corresponding storage location of the target node is generated to perform a deletion operation on the neighbor relationship between the target node and the neighbor node, the neighbor relationship between the target node and the neighbor node is marked for deletion in the corresponding storage location of the neighbor node according to the to-be-executed transaction, the number of neighbor relationships marked for deletion is determined from the corresponding storage location of the neighbor node, if the number exceeds a threshold, an update operation is performed on all the neighbor relationships marked for deletion, an updated neighbor relationship for the neighbor node is generated, a neighbor redo log for the marking deletion operation and the neighbor relationship update operation is generated, wherein before the to-be-executed transaction is completed, there is no limitation on accessing the neighbor relationship of the neighbor node, that is, before the to-be-executed transaction is completed, dirty reading of the neighbor relationship of the neighbor node corresponding to the update operation by other transactions is allowed, when the to-be-executed transaction is completed and committed, the target node data and each layer of neighbor relationship data thereof are committed, but the updated neighbor relationship data of the neighbor node of the target node does not need to be committed.

[0043] For example, in the process of updating the neighbor relationship in the corresponding storage location of the neighbor node, each layer of neighbor relationship data of the target node is traversed, and neighbor relationship data of each layer of neighbor nodes is refreshed: the deleted node (whose data includes {neighbor node identifier, isDeleted flag}) is found from the neighbor relationship data of the neighbor node, the isDeleted flag of the deleted node is set to true, and it is checked whether the number of neighbor nodes with the isDeleted flag set to true in the neighbor relationship data exceeds a certain proportion. If it exceeds, the neighbor of this layer needs to be reselected for the neighbor node to which the neighbor relationship data belongs. The principle is: the existing normal neighbor (i.e., the isDeleted flag is false) is retained, for the neighbor with the isDeleted flag set to true, it is checked whether the corresponding transaction of the graph node data has been committed through the neighbor node identifier, if yes, the nearest one is selected from the new neighbor set to replace it, and if not, it is continued to be retained.

[0044] In the embodiment, by classifying the HNSW index data into two categories of graph node data and neighbor relationship data, different transaction processing mechanisms are applied to different characteristics when a transaction is deleted. The transaction operation on the target node data adopts strong consistency, and a row lock is performed on the target node data before the transaction is completed to generate a target redo log, a target undo log and version transaction data corresponding to the target node deletion operation. The neighbor relationship data of the target node and the neighbor node adopts weak consistency. For the neighbor relationship data of the target node, a neighbor redo log, a neighbor undo log and version transaction data corresponding to the target node neighbor relationship deletion operation are generated. For the neighbor relationship data of the neighbor node, only a neighbor redo log is generated, and dirty reading of the neighbor relationship by other transactions is allowed, so that multiple concurrent transactions can update the neighbor relationship data without blocking, thereby eliminating the transaction lock competition for the neighbor relationship data which changes frequently and dynamically. Through the classification management mode, a good balance is achieved between guaranteeing global strong consistency and reducing neighbor relationship transaction management overhead (such as the overhead of recording complete undo logs), and performance and consistency are effectively considered.

[0045] In an embodiment, as shown in Figure 6 a management method of HNSW index data is provided, and the method further includes the following steps: Step S601: According to a query transaction, a neighbor node is accessed to obtain a neighbor relationship between the neighbor node and a target node.

[0046] Step S602: According to the neighbor relationship, when the target node is accessed, version transaction data of the target operation on the target node by a to-be-executed transaction is obtained according to the version transaction data, and an execution state of the to-be-executed transaction is determined according to the version transaction data.

[0047] Step S603: If the execution state is uncommitted, the corresponding storage location in the target node which is accessed to perform the target operation is limited.

[0048] In the embodiment, the query transaction can refer to a transaction of reading operation on the HNSW index data.

[0049] Specifically, the query transaction performs graph traversal based on the HNSW algorithm, and in each layer of traversal, the algorithm reads the neighbor list of the current node from the neighbor relationship data of the entry point. Since the neighbor relationship data allows dirty reading, the query can immediately obtain the latest connection relationship, including from uncommitted write or delete transactions and the relationship updates caused thereby. Thus, the query can smoothly access the neighbor node and obtain the neighbor relationship of the neighbor node and the target node. In further accessing the target node according to the neighbor relationship, the system obtains the version transaction data generated when the target operation of the to-be-executed transaction is performed on the target node, and determines the commit state of the transaction according to the version transaction data, and if it is detected that the related transaction has not been committed, access to the storage location corresponding to the target node is restricted, at which time the data of the target node that has been committed by other transactions in the historical version can be obtained through the MVCC mechanism, so that the data of the target node written or deleted by the to-be-executed transaction that has not been committed can be ignored.

[0050] In the embodiment, by adopting strong consistency for the target node and weak consistency for the neighbor relationship data during execution of the to-be-executed transaction, dirty reading of other transactions is allowed, and in the query, the node data of the committed transaction operation can be read, and when the node data of the uncommitted transaction operation is read based on the neighbor relationship, the node data of the uncommitted transaction operation can be ignored, thereby ensuring the isolation, so that the result returned by the final query is completely composed of committed data.

[0051] In an embodiment, as shown in FIG. 7, a management method of HNSW index data is provided, further comprising the following steps: Figure 7 Step S701: According to the rollback transaction, the target operation and the update operation are rolled back. If the target operation is a write operation, according to the rollback transaction, the target node is deleted from the corresponding storage location of the target node based on the target undo log, and according to the rollback transaction, the neighbor relationship of the target node and the neighbor node is deleted from the corresponding storage location of the target node and the corresponding storage location of the neighbor node.

[0052] Step S702: If the target operation is a delete operation, according to the rollback transaction, the target node is written into the corresponding storage location of the target node based on the target undo log, and according to the rollback transaction, the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node is rolled back based on the neighbor undo log to obtain the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback.

[0053] Step S703: According to the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback, the deletion of the mark of the neighbor relationship of the target node and the neighbor node in the corresponding storage location of the neighbor node is rolled back.​

[0054] In this embodiment, the rollback transaction can refer to a transaction of rollback operation on the HNSW index data.

[0055] Specifically, if the target operation is a write operation, according to the rollback transaction, the target node is deleted from the corresponding storage location based on the target undo log, and according to the rollback transaction, the neighbor relationship of the target node and the neighbor node is deleted from the corresponding storage location of the neighbor node based on the neighbor relationship of the target node and the neighbor node, the neighbor relationship of the neighbor node is repaired (such as replacing the rolled back target node with a newly found closer neighbor), and the neighbor relationship of the target node and the neighbor node at the corresponding storage location of the target node is deleted.

[0056] If the target operation is a delete operation, according to the rollback transaction, the target node is written into the corresponding storage location of the target node based on the target undo log, according to the rollback transaction, the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node is rolled back based on the neighbor undo log, to obtain the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback, and based on the neighbor relationship after rollback, the target node and the neighbor node are rolled back in the corresponding storage location of the neighbor node (the number of neighbor relationships corresponding to the neighbor node that are marked for deletion does not exceed the threshold, i.e., the neighbor node does not update the neighbor relationship), or the target node and the neighbor node are written into the corresponding storage location of the neighbor node (the number of neighbor relationships corresponding to the neighbor node that are marked for deletion exceeds the threshold, i.e., the neighbor node has updated the neighbor relationship).

[0057] In this embodiment, by adopting strong consistency for the target node and weak consistency for the neighbor relationship data during the execution of the to-be-executed transaction, when rolling back, the strong consistency mechanism of the target node data is used to reverse the neighbor relationship data of the target node and its neighbors when the target node data is rolled back to ensure the strong consistency of the HNSW graph data, thereby achieving the compatibility of strong consistency and high concurrency read and write.

[0058] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0059] In an embodiment, a HNSW index data management device is provided, which is applied to a server in Figure 1 The HNSW index data management device corresponds to the HNSW index data management method in the above embodiments. For example, Figure 8As shown, the HNSW index data management apparatus includes an acquisition module 81, a node data management module 82, and a neighbor relationship management module 83. The functions of the modules are described in detail as follows: The acquisition module 81 is configured to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node. The node data management module 82 is configured to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log, and version transaction data of the to-be-executed transaction performing the target operation on the target node, and perform a row lock on the corresponding storage location in the target node before the to-be-executed transaction is completed. The neighbor relationship management module 83 is configured to perform an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation, and the neighbor relationship of the neighbor node is not limited to be accessed before the to-be-executed transaction is completed.

[0060] Optionally, the node data management module 82 includes: The node writing unit is configured to, if the target operation is a write operation, write the target node into the corresponding storage location according to the to-be-executed transaction. The first recording unit is configured to generate the target redo log corresponding to the write operation, the target undo log, and the version transaction data of the to-be-executed transaction performing the write operation on the target node.

[0061] Optionally, the neighbor relationship management module 83 includes: The relationship writing unit is configured to, if the target operation is a write operation, write the neighbor relationship between the target node and the neighbor node into the corresponding storage location of the target node and the corresponding storage location of the neighbor node according to the to-be-executed transaction. The second recording unit is configured to generate the neighbor redo log for the write operation.

[0062] Optionally, the node data management module 82 includes: The node deleting unit is configured to, if the target operation is a delete operation, delete the target node from the corresponding storage location according to the to-be-executed transaction. The third recording unit is configured to generate the target redo log corresponding to the delete operation, the target undo log, and the version transaction data of the to-be-executed transaction performing the delete operation on the target node.

[0063] Optionally, the neighbor relationship management module 83 includes: The first relationship deleting unit is configured to, if the target operation is a delete operation, delete the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node according to the to-be-executed transaction. a fourth recording unit configured to generate a neighbor redo log for the delete operation, a neighbor undo log for the delete operation, and version transaction data of the delete operation on the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node for the to-be-executed transaction; a second relationship deleting unit configured to mark and delete the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node according to the to-be-executed transaction; a relationship updating unit configured to determine the number of the marked and deleted neighbor relationships from the corresponding storage location of the neighbor node, and perform an update operation on all the marked and deleted neighbor relationships if the number exceeds a threshold value, to generate updated neighbor relationships for the neighbor node; a fifth recording unit configured to generate a neighbor redo log for the mark and delete operation and the update operation.

[0064] Optionally, the HNSW index data management apparatus further includes: a query module configured to access the neighbor node according to a query transaction, and acquire the neighbor relationship between the neighbor node and the target node; a node accessing module configured to acquire version transaction data of a target operation on the target node by the to-be-executed transaction when accessing the target node according to the neighbor relationship, and determine an execution state of the to-be-executed transaction according to the version transaction data; a restricted accessing module configured to restrict access to the corresponding storage location in the target node for the target operation if the execution state is uncommitted.

[0065] Optionally, the HNSW index data management apparatus further includes: a write rollback module configured to roll back the target operation and the update operation according to a rollback transaction, and delete the target node from the corresponding storage location of the target node based on the target undo log according to the rollback transaction if the target operation is a write operation, and delete the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node and the corresponding storage location of the neighbor node according to the rollback transaction; a delete rollback module configured to write the target node into the corresponding storage location of the target node based on the target undo log according to the rollback transaction if the target operation is a delete operation, and roll back the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node based on the neighbor undo log according to the rollback transaction, to obtain the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback; a mark rollback module configured to roll back the mark and delete of the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node according to the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback.

[0066] The specific definition of the HNSW index data management apparatus can refer to the definition of the HNSW index data management method in the foregoing, which will not be repeated here. Each module in the HNSW index data management apparatus described above can be realized by software, hardware, and a combination thereof in whole or in part. The modules described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the modules.

[0067] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store target nodes corresponding to to-be-executed transactions. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an HNSW index data management method.

[0068] In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the HNSW index data management method in the above embodiments when executing the computer program, such as Figure 2 shown in S201-S203, or Figures 3 to 7 shown, which will not be repeated here to avoid repetition. Alternatively, the processor implements the functions of each module / unit in the data management apparatus in this embodiment when executing the computer program, such as Figure 8 shown, which will not be repeated here to avoid repetition.

[0069] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the HNSW index data management method in the above embodiments, such as Figure 2 shown in S201-S203, or Figures 3 to 7 shown, which will not be repeated here to avoid repetition. Alternatively, the processor implements the functions of each module / unit in the data management apparatus in this embodiment when executing the computer program, such as Figure 8The functions of the acquisition module 81, the node data management module 82, and the neighbor relationship management module 83 shown are not repeated here to avoid redundancy.

[0070] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program, the computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. Wherein, any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0072] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for managing HNSW index data, characterized in that, The method comprises the following steps: acquiring a target node corresponding to a to-be-executed transaction, and determining a neighbor node corresponding to the target node; performing a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generating a target redo log, a target undo log corresponding to the target operation, and version transaction data of the to-be-executed transaction performing the target operation on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed; performing an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generating a neighbor redo log and / or a neighbor undo log corresponding to the update operation, wherein the neighbor relationship of the neighbor node is not limited before the to-be-executed transaction is completed. 2.The method of Claim 1, wherein, The method of performing the target operation on the corresponding storage location in the target node according to the to-be-executed transaction, generating the target redo log, the target undo log corresponding to the target operation, and the version transaction data of the to-be-executed transaction performing the target operation on the target node comprises the following steps: if the target operation is a write operation, then writing the target node into the corresponding storage location according to the to-be-executed transaction; generating the target redo log, the target undo log corresponding to the write operation, and the version transaction data of the to-be-executed transaction performing the write operation on the target node. 3.The method of Claim 1, wherein, The method of performing the update operation on the neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generating the neighbor redo log and / or the neighbor undo log corresponding to the update operation comprises the following steps: if the target operation is a write operation, then writing the neighbor relationship between the target node and the neighbor node into the corresponding storage location of the target node and the corresponding storage location of the neighbor node respectively according to the to-be-executed transaction; generating the neighbor redo log for the write operation. 4.The method of Claim 1, wherein, The method of performing the target operation on the corresponding storage location in the target node according to the to-be-executed transaction, generating the target redo log, the target undo log corresponding to the target operation, and the version transaction data of the to-be-executed transaction performing the target operation on the target node comprises the following steps: if the target operation is a delete operation, then deleting the target node from the corresponding storage location according to the to-be-executed transaction; generating the target redo log, the target undo log corresponding to the delete operation, and the version transaction data of the to-be-executed transaction performing the delete operation on the target node. 5.The method of Claim 1, wherein, The method of performing the update operation on the neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generating the neighbor redo log and / or the neighbor undo log corresponding to the update operation comprises the following steps: if the target operation is a delete operation, then deleting the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node according to the to-be-executed transaction; generating the neighbor redo log, the neighbor undo log for the delete operation, and the version transaction data of the to-be-executed transaction performing the delete operation on the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node; According to the to-be-executed transaction, mark deletion is performed on the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node; Determine the number of marked deleted neighbor relationships from the corresponding storage location of the neighbor node, and if the number exceeds a threshold, perform an update operation on all marked deleted neighbor relationships to generate updated neighbor relationships for the neighbor node; Generate a neighbor redo log for the mark deletion operation and the update operation. 6.The method of Claim 1, wherein, The management method further comprises: According to the query transaction, access the neighbor node to obtain the neighbor relationship between the neighbor node and the target node; According to the neighbor relationship, when accessing the target node, obtain the version transaction data of the target operation of the to-be-executed transaction on the target node according to the neighbor relationship, and determine the execution state of the to-be-executed transaction according to the version transaction data; If the execution state is uncommitted, restrict access to the corresponding storage location in the target node performing the target operation. 7.The method of Claim 1, wherein, The management method further comprises: According to the rollback transaction, roll back the target operation and the update operation, if the target operation is a write operation, according to the rollback transaction, based on the target undo log, delete the target node from the corresponding storage location in the target node, and according to the rollback transaction, respectively delete the neighbor relationship between the target node and the neighbor node from the corresponding storage location of the target node and the corresponding storage location of the neighbor node; If the target operation is a delete operation, according to the rollback transaction, based on the target undo log, write the target node to the corresponding storage location in the target node, and according to the rollback transaction, based on the neighbor undo log, roll back the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node, to obtain the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback; According to the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the target node after rollback, roll back the mark deletion of the neighbor relationship between the target node and the neighbor node in the corresponding storage location of the neighbor node.

8. An apparatus for managing HNSW-indexed data, the apparatus comprising: Comprise: An acquisition module is used to acquire a target node corresponding to a to-be-executed transaction, and determine a neighbor node corresponding to the target node; A node data management module is used to perform a target operation on a corresponding storage location in the target node according to the to-be-executed transaction, generate a target redo log corresponding to the target operation, a target undo log, and version transaction data of the target operation of the to-be-executed transaction on the target node, wherein a row lock is performed on the corresponding storage location in the target node before the to-be-executed transaction is completed; A neighbor relationship management module is configured to perform an update operation on a neighbor relationship between the neighbor node and the target node according to the to-be-executed transaction, and generate a neighbor redo log and / or a neighbor undo log corresponding to the update operation, wherein before the to-be-executed transaction is completed, access to the neighbor relationship of the neighbor node is not limited.

9. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the HNSW index data management method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the HNSW index data management method according to any one of claims 1 to 7 when the processor executes the computer program.

Images