Method and device for realizing fragmented HNSW index

Abstract

The invention relates to the technical field of databases, and provides a fragmented HNSW index implementation method and device. The method comprises the following steps: determining common columns meeting column value distribution conditions in a database table; classifying the data in the database table according to the column value of the common column to obtain a plurality of fragment data sets; creating a corresponding vector index for each fragmented data set to obtain a fragmented database table; and performing data query and / or data maintenance on the fragmented database table by using the vector index. According to the method, the vector index is established through fragmentation, so that the problems of long time consumption for constructing the HNSW index and overlarge memory occupation during query are solved.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method and apparatus for implementing a sharded HNSW index. Background Technology

[0002] Vector databases represent an important area of ​​current database research. They store data sets in vector form, enabling the management, storage, and retrieval of vectors. Compared to traditional databases, vector database queries are more focused on similarity searches. Specifically, by calculating the distance between the query vector and all vectors in the stored vector database table, the nearest neighbor vector of the query vector is found within the large-scale vector data.

[0003] Vector databases typically store hundreds of millions of vector data points. Brute-force searches in these databases are extremely time-consuming, necessitating the use of vector indexes to accelerate queries. Hierarchical Navigable Small World (HNSW) is a graph-based vector retrieval algorithm primarily used for efficiently searching high-dimensional vector data. HNSW indexes utilize a hierarchical graph structure, progressively narrowing the search scope from the top level, eliminating the need to traverse all data points, resulting in a time complexity approaching logarithmic levels. Compared to traditional methods, it avoids the "curse of dimensionality" in high-dimensional spaces, offering superior performance.

[0004] However, since the initial construction of HNSW requires multiple layers of probability screening and iteration, it takes a significant amount of time on large-scale data, resulting in the problem of long construction time for HNSW indexes. Furthermore, when querying through the HNSW index, the entire HNSW index needs to be loaded into memory, which consumes too much memory when the data volume is large.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and apparatus for implementing a sharded HNSW index. The purpose is to classify the vector data in the database table according to the column value of a specified ordinary column, create a vector index on each category, and use these vector indexes for querying, thereby solving the problems of long time consumption for HNSW index construction and excessive memory consumption during querying.

[0007] The present invention adopts the following technical solution: In a first aspect, the present invention provides a method for implementing a sharded HNSW index, comprising: Identify the ordinary columns in a database table that meet the column value distribution criteria; Based on the column values ​​of the ordinary columns, the data in the database table is classified to obtain multiple fragmented data sets; Create a corresponding vector index for each of the fragmented data sets to obtain the fragmented database tables; The vector index is used to perform data queries and / or data maintenance on the sharded database tables.

[0008] Furthermore, the column value distribution condition is as follows: the column values ​​of the ordinary column are discrete values, and the number of different categories of the column values ​​is within a preset numerical range.

[0009] Furthermore, when querying data in the sharded database table, the use of the vector index to query and / or maintain data in the sharded database table includes: Use the vector index to obtain the first set to be queried; In each of the first query sets, the nearest neighbors of the target vector to be queried are searched to obtain the first initial result set; The first initial result set is merged to obtain the first global result sort and the first target result set.

[0010] Furthermore, obtaining the first query set using the vector index includes: Obtain the set of sharded column values ​​to be queried; wherein, the set of sharded column values ​​includes at least one index value of the vector index; Check if the data set corresponding to the index value exists in memory; When the fragmented data set corresponding to the index value exists in memory, the fragmented data set is directly retrieved from memory and used as the first set to be queried; When the shard data set corresponding to the index value is not in memory, the shard data set corresponding to the index value is searched in the system table; wherein, the system table is used to record the correspondence between the vector index and the shard data set; The found fragmented data set is loaded into memory so that the fragmented data set can be retrieved from memory and used as the first set to be searched.

[0011] Furthermore, when the sharded column value set is not used, and data queries are required on the sharded database table, the use of the vector index to perform data queries and / or data maintenance on the sharded database table includes: Sequentially obtain all vector indices corresponding to the sharded database tables; according to the index values ​​of the vector indices, obtain the sharded data set from the sharded database tables; Load the acquired fragmented data set into memory; The fragmented data set in memory is identified as the second set to be searched; In each second query set, search for the nearest neighbor of the target vector to be queried, until all second query sets have been searched, and obtain the second initial result set; The second initial result set is merged to obtain the second global result sort and the second target result set.

[0012] Furthermore, when the data maintenance involves inserting data, the step of using the vector index to perform data querying and / or data maintenance on the sharded database table includes: Obtain the column value to be inserted at the position corresponding to the vector index from the data to be inserted; When the value of the column to be inserted is a new value, a corresponding vector index is created for the new value, and the value of the column to be inserted is inserted according to the created vector index; When the column value to be inserted already exists, determine the vector index to which the column value to be inserted belongs, and insert the column value to be inserted according to the determined vector index.

[0013] Furthermore, when the data maintenance is for updated data, the step of using the vector index to perform data querying and / or data maintenance on the sharded database table includes: Obtain the column value to be updated at the position corresponding to the vector index from the data to be updated; use the column value to be updated as the old index value, and delete the old vector of the data to be updated from the data set of the shard corresponding to the old index value; Insert a new vector of the data to be updated into the sharded data set corresponding to the column value to be updated; When there is no corresponding sharded data set for the column value to be updated, a corresponding vector index is created for the column value to be updated, the corresponding sharded data set is obtained, and the new vector is inserted into the obtained sharded data set.

[0014] Furthermore, when the data maintenance involves deleting data, the step of using the vector index to perform data querying and / or data maintenance on the sharded database table includes: Obtain the column value to be deleted from the position corresponding to the vector index in the data to be deleted; Delete the old vector of the data to be deleted from the sharded data set corresponding to the column value to be deleted; When the old vector is the last vector in the corresponding sharded data set, the sharded data set is deleted, and the column value to be deleted is deleted from the system table.

[0015] Secondly, the present invention also provides a fragmented HNSW index implementation apparatus, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the sliced ​​HNSW index implementation method described in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the sliced ​​HNSW index implementation method described in the first aspect.

[0017] Fourthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the sliced ​​HNSW index implementation method as described in the first aspect.

[0018] Fifthly, the present invention also provides a fragmented HNSW index implementation system, including a fragmented HNSW index implementation device as described in the second aspect, and using the fragmented HNSW index implementation method as described in the first aspect to complete the interaction of the fragmented HNSW index implementation device of the second aspect.

[0019] Unlike existing technologies, the present invention has at least the following beneficial effects: This invention specifies a common column in a database table, classifies the vector data in the table according to the column value, and creates a vector index for each category. These vector indexes are used for data querying and maintenance of the database table. By classifying the data in the database table and adding vector indexes in a sharded manner, the problem of long HNSW index construction time caused by the complexity of multi-level graph construction of the entire database table is solved, and the overhead of adding vector indexes is relatively small. By dividing the database table according to the column value of the common column, the sharded vector index can directly query the first N vectors of the vector index corresponding to a certain column value, solving the problem of excessive memory consumption during queries. Because the vector index is determined by the column value, when using the vector index to query the corresponding sharded data set, compared with the existing technology using HNSW indexes, it avoids searching for data corresponding to other column values, further improving search efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 This is a flowchart illustrating a sharded HNSW index implementation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating a specific example of index construction provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a data query process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating a specific example of data querying provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for obtaining a first set to be searched, provided by an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating another specific example of data querying provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a data insertion process provided by an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating a specific example of inserting data provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a data update process provided by an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating a specific example of updating data provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of a data deletion process provided by an embodiment of the present invention; Figure 12 This is a schematic diagram illustrating a specific example of data deletion provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the architecture of a sharded HNSW index implementation device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0024] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0025] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0026] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0027] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0028] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0029] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] Example 1: HNSW indexes have some drawbacks: On the one hand, although the HNSW algorithm achieves efficient retrieval through a hierarchical graph structure and the retrieval time complexity can reach about O(log n), its initial construction requires multiple layers of probability screening and iteration, which is time-consuming under large-scale data. Compared with the time taken by brute-force search, the construction time of HNSW index increases by 50% to 200%.

[0031] The process of building an HNSW index is to insert points one by one into a multi-level graph. Because existing points in the multi-level graph and newly inserted points will affect each other, they can only be inserted one by one in sequence, which takes a long time to build.

[0032] Currently, methods to accelerate the construction of multi-layer graphs generally involve using graphics processing units (GPUs) to speed up the calculation and sorting of vector distances, but this increases hardware overhead. Alternatively, the computational load can be reduced by adjusting the parameters used to construct the multi-layer graph, but this may lead to a decrease in recall, resulting in poor database usability.

[0033] On the other hand, when using HNSW indexes, it is necessary to store the adjacency relationships of multi-level graphs; when storing adjacency relationships, the memory consumption of a single index in a 1 billion × 128 dimension scenario can reach the terabyte (TB) level; and when dynamically updating database tables, insert and delete operations on the database tables will trigger graph structure adjustments, and large-scale updates will generate cascading modifications, and frequent adjustments to the graph structure of multi-level graphs will further exacerbate memory pressure.

[0034] When querying a vector database based on an HNSW index, all the data corresponding to the entire HNSW index needs to be loaded into memory, resulting in excessive memory consumption.

[0035] The common methods to reduce memory usage are: (1) reducing the parameters for building multi-level graphs, such as reducing the maximum number of neighbors of a node, which can reduce the connection density of each node and directly reduce memory usage. However, reducing the parameters for building multi-level graphs often leads to a decrease in recall and poor practicality. (2) using quantization algorithms to compress vector dimensions to reduce memory usage, but this may also lead to a decrease in recall. (3) if the data volume is too large, it may be necessary to replace the index with an inverted file index (IVF) index, but the query speed of the inverted file index is lower than that of the HNSW index, which affects database performance.

[0036] Furthermore, in vector databases, a database table includes vector columns and regular columns. Regular columns are often related to vectors, for example, used to record the region or year to which a vector belongs. When performing mixed queries, for example, if the database table has a vector column and a regular column representing a region (e.g., province A), and we need to query the top 100 vectors closest to the target vector in the database table, and this vector belongs to province A, using existing HNSW indexes often requires querying far more than 100 closest vectors, and then filtering from these vectors to find the vectors belonging to province A. Specifically, existing technologies often use vector indexes and regular indexes for parallel queries; where the HNSW index is a vector index used for vector-type data (e.g., vector columns), and the regular index is a regular index built on a regular column, a means for the database to speed up queries; that is, a vector index is created on the vector column, and a regular index is created on the regular column, then the two indexes are used separately to obtain the corresponding result sets, and finally the two result sets are merged to obtain the final query result. However, since the HNSW index is built based on the entire database table, when using the HNSW index for querying, it is necessary to query more than 100 results, which affects the query performance of the vector database.

[0037] For example, the database table TEST_VEC is created using the following statement: "CREATE TABLE TEST_VEC(Province VARCHAR, embedding VECTOR)". This table has a vector column of type "embedding" and a regular column of type "Province". After creating an HNSW index on the "embedding" column, when querying the top 100 data entries where the "Province" column value is "Province A" and is closest to the target vector A, one query process is: first, use the HNSW index to query the top 200 data entries closest to the target vector A, and then filter from these to find the data where the province is Province A. Since querying the 200 closest data points to the target vector A takes significantly longer than querying the 100 closest data points, and there's no guarantee that 100 of the 200 retrieved data points will be from province "A", querying only the 200 closest data points to the target vector A may not yield the 100 closest data points to the target vector A whose "province" column value is "A". This indicates that conventional HNSW indexes have issues with both efficiency and accuracy in mixed query scenarios.

[0038] To solve the above problems, such as Figure 1 As shown, this embodiment of the invention provides a method for implementing a sharded HNSW index, including: Step 10: Identify the ordinary columns in the database table that meet the column value distribution conditions.

[0039] The database table is a vector database table, and the ordinary column is a column in the database table other than the vector column.

[0040] In this embodiment of the invention, users can specify a column in a database table (excluding vector columns) to categorize data based on that column. If the column name of a regular column is a region, then the column value can be a specific province. During the index building phase when creating a database table, in addition to the usual parameters for creating an HNSW index, this embodiment of the invention also requires the user to specify a regular column in the database table as the basis for data partitioning.

[0041] The following describes the applicable scenarios for the sharded HNSW index implementation method of this invention: In one embodiment, the column value distribution condition is that the column values ​​of the ordinary column are discrete values, and the number of distinct categories of the column values ​​is within a preset numerical range. The preset numerical range is selected by those skilled in the art based on the specific application scenario. The column values ​​of the ordinary column specified in this embodiment need to be discrete, and the number of categories of the column values ​​should be relatively small. Ideally, the column values ​​should be distributed as evenly as possible according to the discrete values; for example, when the column name of the ordinary column is "Region," the column values ​​include Province A, Province B, Province C, Province D, and Province E. When there are 1 million data entries in the database table, the database table contains approximately 200,000 entries with a column value of Province A, approximately 200,000 entries with a column value of Province B, approximately 200,000 entries with a column value of Province C, approximately 200,000 entries with a column value of Province D, and approximately 200,000 entries with a column value of Province E.

[0042] In one embodiment, when the column name of a regular column is "Region" and the preset value range is 10 to 50, a regular column that meets the column value distribution condition is: the column value is a province, the province is a discrete value, including 20 provinces such as province A and province B, that is, the corresponding number of different categories is 20. Since this embodiment of the invention sets multiple vector indexes for a database table by dividing it into categories, and uses each vector index to locate the corresponding part of the data, the performance of locating data using vector indexes can only be ensured when the number of categories is as reasonable as possible relative to the size of the data in the database table. For example, if there are 1 million data records in the database table, of which 990,000 are of the same category (e.g., the column value is province A), since subsequent queries are likely to be for data with the column value of province A, it is still necessary to search among the 990,000 data records. Compared to searching directly among the 1 million data records, the significance of searching among the 990,000 data records through vector indexes is smaller.

[0043] Step 20: Based on the column values ​​of the ordinary columns, classify the data in the database table to obtain multiple fragmented data sets.

[0044] After classification, the column value of a regular column corresponds to a sharded data set.

[0045] Based on the discrete values ​​of ordinary columns specified by the user (e.g., region-province, time-year), the vector data in the database table is sharded to generate data subsets, namely, multiple sharded data sets in this embodiment of the invention.

[0046] Step 30: Create a corresponding vector index for each of the sharded data sets to obtain the sharded database tables.

[0047] In this invention, one vector index corresponds to one sharded data set. The vector index in this embodiment is an HNSW index; to distinguish it from the HNSW index that vector databases originally need to create in the prior art, the term "vector index" in this document specifically refers to the vector index proposed by the sharded HNSW index implementation method of this embodiment.

[0048] The vector index in this embodiment of the invention is constructed using vector data. The specific method of creating the vector index on each shard data set is determined by those skilled in the art based on the specific use case, and is not limited here.

[0049] In one embodiment, a multi-threaded computing framework can be used to synchronously build the vector index of each shard on each sharded dataset, thereby reducing the construction time from O(nlog n) to O(n / mlog(n / m)), where m is the number of shards.

[0050] After creating a vector index, you also need to record the shard ID, column value range, and index storage path corresponding to the vector index in the system table of the vector database.

[0051] In one embodiment, vector indexes for all sharded data sets can be built in parallel; in another embodiment, vector indexes for the next batch of sharded data sets can be built after the vector indexes for a batch of sharded data sets have been built and the constructed multi-level graph has been written to disk. This avoids the need to store all the data in the database table in memory, thereby avoiding excessive memory usage.

[0052] like Figure 2 As shown, the regular parameters for creating the HNSW index and the discrete values ​​of the user-specified ordinary columns are used as the HNSW index parameters; according to the HNSW index parameters, first, according to step 20, the data is divided into sharded sets according to the discrete values ​​of the ordinary columns; then, according to step 30, a vector index is created and the sharding is completed.

[0053] Step 40: Use the vector index to perform data query and / or data maintenance on the sharded database tables.

[0054] One vector index corresponds to one shard data set. The corresponding shard data set can be located through the vector index to perform data query and / or data maintenance on the shard data set.

[0055] This invention specifies a common column in a database table, classifies the vector data in the table according to the column value, and creates a vector index for each category. These vector indexes are used for data querying and maintenance of the database table. By classifying the data in the database table and adding vector indexes in a sharded manner, the problem of long HNSW index construction time caused by the complexity of multi-level graph construction of the entire database table is solved, and the overhead of adding vector indexes is relatively small. By dividing the database table according to the column value of the common column, the sharded vector index can directly query the first N vectors of the vector index corresponding to a certain column value, solving the problem of excessive memory consumption during queries. Because the vector index is determined by the column value, when using the vector index to query the corresponding sharded data set, compared with the existing technology using HNSW indexes, it avoids searching for data corresponding to other column values, further improving search efficiency.

[0056] The data query process is explained below, such as... Figure 3 As shown, when performing a data query on the sharded database table, step 40 includes: Step 101: Use the vector index to obtain the first set to be queried.

[0057] The first set to be queried is a fragmented data set. For example... Figure 4 As shown, the method of obtaining the first query set varies depending on the input data used for the query. The following text will explain in detail the methods of obtaining the first query set under different circumstances.

[0058] Step 102: On each of the first query sets, search for the nearest neighbor of the target vector to be queried, and obtain the first initial result set.

[0059] Perform a search row-wise on the sharded dataset that meets the conditions, searching for the target vector (i.e., such as...). Figure 4 The K nearest neighbors of v in the equation.

[0060] Step 103: Merge the first initial result set to obtain the first global result sort and the first target result set.

[0061] In one embodiment, the first initial result set can be merged into a min-heap to achieve a global Top-K sort, that is, to obtain the first global result sort and the first target result set.

[0062] In one alternative embodiment, a fragment access hot queue can be maintained simultaneously; fragment data sets that are frequently used in memory can be determined according to the access hot queue, and fragment data sets that are not in the access hot queue can be automatically unloaded, thereby automatically unloading fragment data sets that have not been used for a long time in memory to free up memory space.

[0063] When a user uses a set of sharded column values ​​as input data for a query, to address the issue of high memory consumption, this embodiment of the invention uses a sharded vector index query. Only the sharded data set that meets the input data query conditions needs to be loaded into memory, without loading all the data. In step 101, as... Figure 5 As shown, obtaining the first query set using the vector index includes: Step 201: Obtain the set of sharded column values ​​to be queried; wherein the set of sharded column values ​​includes at least one index value of the vector index.

[0064] The sharded column value set is the set of column values ​​of the ordinary columns used to create the sharded index. The column values ​​of the ordinary columns are the index values ​​of the corresponding vector indexes.

[0065] This is equivalent to using the column values ​​that the user needs to query as index values, and locating the vector index according to the index values, so that the corresponding sharded data set can be located later using the vector index.

[0066] like Figure 4 As shown, in one embodiment, for the set of sharded column values ​​C that the user needs to query, each column value in the set of sharded column values ​​C (i.e., such as...) Figure 4 The vector index (col) is used as the index value to locate the vector index to be queried, and then the sharded data set with the column value of the ordinary column in the database table is found.

[0067] Step 202: Query whether there is a set of sharded data corresponding to the index value in memory.

[0068] Check if the required set of sharded data has been loaded into memory.

[0069] Step 203: When there is a set of sharded data corresponding to the index value in memory, directly retrieve the set of sharded data from memory and use it as the first set to be queried.

[0070] Locate the vector index based on the index value. By searching the vector index, determine whether the corresponding shard data set has been loaded into memory. If the vector index can be found in memory, it means that the corresponding shard data set has been loaded into memory, and the shard data set can be obtained directly.

[0071] Step 204: When there is no shard data set corresponding to the index value in memory, search for the shard data set corresponding to the index value in the system table; wherein, the system table is used to record the correspondence between the vector index and the shard data set.

[0072] When the sharded data set corresponding to the vector index is not loaded into memory, since there is a record of the vector index created in this embodiment of the invention in the system table, it is necessary to use the index value to look up the sharded data set corresponding to the vector index in the system table.

[0073] Step 205: Load the found fragmented data set into memory so that the fragmented data set can be retrieved from memory and used as the first set to be searched.

[0074] like Figure 4 As shown, if no ordinary column value set C is passed in, then p data sets of each segment are loaded into memory in a loop, and searches are performed in parallel on these segments to find the K nearest neighbors of the target vector, until the vector indices of all segments are retrieved; specifically, as Figure 6 As shown, when the sharded column value set is not used, and data queries are required on the sharded database table, step 40 includes: Step 301: Sequentially obtain all vector indexes corresponding to the sharded database tables.

[0075] like Figure 4 As shown, in one embodiment, each vector index has an ID, and the vector indexes are arranged in order of ID. All vector indexes corresponding to the database table are retrieved sequentially according to the ID order of the vector indexes.

[0076] Step 302: Obtain the sharded data set from the sharded database table according to the index value of the vector index.

[0077] In one optional embodiment, p fragments of data can be loaded in each batch to reduce memory usage; the specific value of p can be determined by those skilled in the art based on the specific use case, and is not limited here.

[0078] Step 303: Load the acquired fragmented data set into memory; determine the fragmented data set in memory as the second query set.

[0079] The second query set is a sharded data set. It should be noted that both the first and second query sets are sharded data sets, the only difference being that they are sharded data sets under different query scenarios.

[0080] Step 304: On each second query set, search for the nearest neighbor of the target vector to be queried, until all second query sets have been searched, and obtain the second initial result set.

[0081] The specific method of searching on each second set to be searched is determined by those skilled in the art based on the specific use case; in an optional embodiment, the HNSW algorithm is used to search on each second set to be searched.

[0082] Step 305: Merge the second initial result set to obtain the second global result sort and the second target result set.

[0083] In one embodiment, the second initial result set can be merged into a min-heap to achieve global Top-K sorting (i.e., second global result sorting) to obtain the second target result set.

[0084] The embodiments of the present invention alleviate memory pressure by loading sharded indexes on demand and using an eviction mechanism, and accelerate search speed through parallel search.

[0085] In one embodiment, for the database table TEST_VEC, which has a vector column "embedded vector" of vector type and a regular column "province" of character type, according to the sharded HNSW index implementation method of this invention, after creating a vector index on the regular column "province", the data in the database table TEST_VEC is divided into multiple sharded data sets according to the discrete values ​​of the "province" column. When querying the top 100 data points for which the column value of the "province" column is "Province A" and is closest to the target vector A, the query process is as follows: first, query the vector index corresponding to "Province A" and obtain the sharded data set of that vector index; then, query the 100 data points closest to the target vector A from the obtained sharded data sets. Compared to existing technologies that require querying far more than 100 data points and cannot guarantee that the queried data includes all 100 data points required, this embodiment of the invention only requires querying 100 data points in the vector index, and all 100 data points are the data to be queried. The computational resources consumed by the query are more controllable, resulting in better query performance and higher efficiency.

[0086] By establishing a sharded vector index, the data size of graph structure adjustments caused by insert and delete operations on database tables can be reduced, and cascading modifications can be minimized. Specifically, for example... Figure 7 As shown, when the data is maintained as inserted data, step 40 includes: Step 401: Obtain the column value to be inserted at the position corresponding to the vector index from the data to be inserted.

[0087] The data to be inserted includes vector columns and regular columns, and the values ​​of the columns to be inserted (i.e., such as...) Figure 8 The column value shown (col) is a regular column with a vector index.

[0088] Step 402: When the value of the column to be inserted is a new value, create a corresponding vector index for the new value, and insert the value of the column to be inserted according to the created vector index.

[0089] Step 403: When the column value to be inserted already exists, determine the vector index to which the column value to be inserted belongs, and insert the column value to be inserted according to the determined vector index.

[0090] like Figure 8 As shown, when inserting data, if the column value of the regular column in the database table does not exist, it means that the column value to be inserted is a new value. In this case, a new sharded vector index is created, and the data to be inserted is used as the data in the sharded data set of the sharded index. If the column value to be inserted is an existing value of the regular column in the database table, the data to be inserted is directly used as the data in the sharded data set of the existing value.

[0091] like Figure 9 As shown, when the data is maintained as updated data, step 40 includes: Step 501: Obtain the column value to be updated at the position corresponding to the vector index from the data to be updated.

[0092] Similarly, when inserting data, the column value to be updated (i.e., such as...) will be updated during data updates. Figure 10 The column value shown (col) is a regular column with a vector index.

[0093] Step 502: Use the column value to be updated as the old index value, and delete the old vector of the data to be updated on the shard data set corresponding to the old index value.

[0094] The index value to be updated is used as the old index value, thus locating the old vector in the database table that needs to be updated. For example... Figure 10 As shown, when updating the vector column of the old vector, it is necessary to use the vector index to locate the old vector, delete the old vector first, and then insert the new vector.

[0095] Step 503: Insert a new vector of the data to be updated into the sharded data set corresponding to the column value to be updated.

[0096] Step 504: When there is no corresponding sharded data set for the column value to be updated, create a corresponding vector index for the column value to be updated, obtain the corresponding sharded data set, and insert the new vector into the obtained sharded data set.

[0097] If the column value that does not have a vector index is the value of the column to be updated, then the corresponding vector index needs to be created first, and then the new vector can be inserted.

[0098] like Figure 11 As shown, when the data maintenance is for deleting data, step 40 includes: Step 601: Obtain the column value to be deleted at the position corresponding to the vector index from the data to be deleted.

[0099] Similarly, when inserting data, the column value to be deleted (i.e., such as...) is used when deleting data. Figure 12 The column value shown (col) is a regular column with a vector index.

[0100] Step 602: Delete the old vector of the data to be deleted from the sharded data set corresponding to the column value to be deleted.

[0101] like Figure 12 As shown, when deleting an existing old vector in a database table, the vector index corresponding to the data to be deleted is located using the column value to be deleted. The sharded data set corresponding to the column value to be deleted is found through the vector index, and then the old vector is deleted.

[0102] Step 603: When the old vector is the last vector in the corresponding sharded data set, delete the sharded data set and delete the column value to be deleted in the system table.

[0103] If the vector to be deleted is already the last vector in the shard data set corresponding to the vector index, then delete the shard data set and the vector index, and delete the corresponding record of the shard in the system table, for example, the column value to be deleted representing the vector index.

[0104] For "add," "delete," and "modify" operations in a vector database, existing technologies require waiting for the HNSW index to be fully maintained before a normal query can be performed on the entire database table. However, after creating a vector index according to step 30 of this embodiment, modifications only need to be made to the vector index corresponding to the user-specified ordinary column and the sharded data set corresponding to that vector index; other vector indexes and sharded data sets can still provide query functionality.

[0105] When the column values ​​of ordinary columns in a database table are discrete and relatively uniformly distributed, the sharded HNSW index implementation method of this invention is used to establish vector indexes. Since each vector index only involves a portion of the data in the database table, the operation is flexible, does not introduce excessive resource overhead, and does not require operation on all data in the database table. Compared with the existing technology that operates on the entire database table each time through the overall HNSW index, the operation efficiency of each vector index is better than that of the overall HNSW index. Furthermore, the data of the entire database table is classified into multiple sharded data sets according to the column values ​​of ordinary columns. Each sharded data set is relatively independent, and updating one sharded data set will not affect the use of other sharded data sets, resulting in high flexibility.

[0106] Example 2: like Figure 13The diagram shown is an architectural schematic of a fragmented HNSW index implementation device according to an embodiment of the present invention. The fragmented HNSW index implementation device of this embodiment includes one or more processors 21 and a memory 22. Figure 13 Take a processor 21 as an example.

[0107] Processor 21 and memory 22 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0108] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the segmented HNSW index implementation method in this embodiment. The processor 21 executes the segmented HNSW index implementation method by running the non-volatile software programs and instructions stored in the memory 22.

[0109] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0110] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they execute the fragmented HNSW index implementation method in the above embodiments, for example, executing each step of the fragmented HNSW index implementation method of the present invention described above.

[0111] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 13 A processor 21 can enable one or more of the processors to execute the fragmented HNSW index implementation method in the specific embodiments of the present invention, for example, to execute the various steps of the fragmented HNSW index implementation method of the embodiments of the present invention described above; it can also implement Figure 13 The various modules and units described above; or the fragmented HNSW index implementation method in the specific embodiments of the present invention, for example, executing the various steps of the fragmented HNSW index implementation method of the embodiments of the present invention described above; can also be implemented. Figure 13 The various modules and units mentioned above.

[0112] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing a sharded HNSW index, characterized in that, include: Identify the ordinary columns in a database table that meet the column value distribution conditions; Based on the column values ​​of the ordinary columns, the data in the database table is classified to obtain multiple fragmented data sets; Create a corresponding vector index for each of the fragmented data sets to obtain the fragmented database tables; The vector index is used to perform data queries and / or data maintenance on the sharded database tables.

2. The fragmented HNSW index implementation method according to claim 1, characterized in that, The column value distribution condition is as follows: the column values ​​of the ordinary column are discrete values, and the number of different categories of the column values ​​is within a preset numerical range.

3. The fragmented HNSW index implementation method according to claim 2, characterized in that, When querying data in the sharded database table, the method includes: Use the vector index to obtain the first set to be queried; In each of the first query sets, the nearest neighbors of the target vector to be queried are searched to obtain the first initial result set; The first initial result set is merged to obtain the first global result sort and the first target result set.

4. The fragmented HNSW index implementation method according to claim 3, characterized in that, The method includes: Obtain the set of sharded column values ​​to be queried; wherein, the set of sharded column values ​​includes at least one index value of the vector index; Check if the data set corresponding to the index value exists in memory; When the fragmented data set corresponding to the index value exists in memory, the fragmented data set is directly retrieved from memory and used as the first set to be queried; When the shard data set corresponding to the index value is not in memory, the shard data set corresponding to the index value is searched in the system table; wherein, the system table is used to record the correspondence between the vector index and the shard data set; The found fragmented data set is loaded into memory so that the fragmented data set can be retrieved from memory and used as the first set to be searched.

5. The fragmented HNSW index implementation method according to claim 4, characterized in that, When the sharded column value set is not used, and data queries are required on the sharded database table, the method includes: Sequentially obtain all vector indexes corresponding to the partitioned database tables; According to the index value of the vector index, obtain the sharded data set from the sharded database table; Load the acquired fragmented data set into memory; determine the fragmented data set in memory as the second query set; In each second query set, search for the nearest neighbor of the target vector to be queried, until all second query sets have been searched, and obtain the second initial result set; The second initial result set is merged to obtain the second global result sort and the second target result set.

6. The fragmented HNSW index implementation method according to claim 1, characterized in that, When the data is maintained as inserted data, the method includes: Obtain the column value to be inserted at the position corresponding to the vector index from the data to be inserted; When the value of the column to be inserted is a new value, a corresponding vector index is created for the new value, and the value of the column to be inserted is inserted according to the created vector index; When the column value to be inserted already exists, determine the vector index to which the column value to be inserted belongs, and insert the column value to be inserted according to the determined vector index.

7. The fragmented HNSW index implementation method according to claim 6, characterized in that, When the data is maintained as updated data, the method includes: Obtain the column value to be updated at the position corresponding to the vector index from the data that needs to be updated; Use the column value to be updated as the old index value, and delete the old vector of the data to be updated in the sharded data set corresponding to the old index value; Insert a new vector of the data to be updated into the sharded data set corresponding to the column value to be updated; When there is no corresponding sharded data set for the column value to be updated, a corresponding vector index is created for the column value to be updated, the corresponding sharded data set is obtained, and the new vector is inserted into the obtained sharded data set.

8. The fragmented HNSW index implementation method according to claim 6, characterized in that, When the data is maintained as deleted data, the method includes: Obtain the column value to be deleted from the position corresponding to the vector index in the data to be deleted; Delete the old vector of the data to be deleted from the sharded data set corresponding to the column value to be deleted; When the old vector is the last vector in the corresponding sharded data set, the sharded data set is deleted, and the column value to be deleted is deleted from the system table.

9. A fragmented HNSW index implementation device, characterized in that, The fragmented HNSW index implementation apparatus includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to implement the fragmented HNSW index implementation method according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the fragmented HNSW index implementation method according to any one of claims 1-8.

Images