Vector data processing method, device and system and storage medium

Abstract

The invention discloses a method, device and system for processing vector data and a storage medium, and belongs to the field of storage. The method is applied to a vector database system, the vector database system comprises a first cluster, the first cluster is used for storing a plurality of pieces of vector data, the plurality of pieces of vector data are data stored to the first cluster based on a first center, and the first center is the center of the first cluster. The method comprises the following steps: acquiring a second center based on vector data stored by a first cluster; when it is determined that the center of the first cluster is changed from the first center to the second center based on the second center, at least one piece of vector data is obtained based on the second center, the at least one piece of vector data is data which are stored by the first cluster and belong to the second cluster, and the vector database system further comprises the second cluster; and migrating at least one piece of vector data from the first cluster into the second cluster. According to the method, the vector data query precision can be improved.

Description

Technical Field

[0001] This application relates to the field of storage, and in particular to a method, apparatus, system and storage medium for processing vector data. Background Technology

[0002] A vector database is a database used to store vector data, which is typically data obtained by vectorizing content such as text, speech, images, or video. A vector database includes multiple clusters, each used to store different types of vector data.

[0003] When vector data that needs to be saved to a vector database is received, the distance between the vector data and multiple centers corresponding to the multiple clusters can be calculated. The center with the smallest distance from the vector data is selected from the multiple centers, and the vector data is saved to the cluster corresponding to that center.

[0004] As time goes on, the distribution of vector data stored in the vector database changes continuously, which means that any cluster in the vector database may include a large amount of vector data that does not belong to that cluster. This will reduce the accuracy of querying data from the vector database. Summary of the Invention

[0005] This application provides a method, apparatus, system, and storage medium for processing vector data to improve the accuracy of querying vector data. The technical solution is as follows:

[0006] Firstly, this application provides a method for processing vector data. The method is applied to a vector database system, which includes a first cluster for storing multiple vector data sets. These multiple vector data sets are stored in the first cluster based on a first center, where the first center is the center of the first cluster. In the method, a second center is obtained based on the vector data stored in the first cluster. When it is determined that the center of the first cluster changes from the first center to the second center based on the second center, at least one vector data set is obtained based on the second center. This at least one vector data set is data stored in the first cluster belonging to the second cluster, and the vector database system also includes the second cluster. The at least one vector data set is then migrated from the first cluster to the second cluster.

[0007] Based on the vector data stored in the first cluster, a second center is obtained, which is the latest center of the first cluster. When the center of the first cluster changes from the first center to the second center, it indicates that there are a large number of vector data that do not belong to the first cluster. Based on the second center, the vector data in the first cluster that belongs to the second cluster is found, and the found vector data is migrated from the first cluster to the second cluster. In this way, the vector data retained in the first cluster is the data that belongs to the first cluster, thereby improving the accuracy of vector data retrieval when querying vector data from the vector database system.

[0008] In one possible implementation, for each vector data in at least one set of vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center. Therefore, at least one vector data is closer to the center of the second cluster and belongs to the second cluster. Migrating at least one vector data ensures that the vector data retained in the first cluster belongs to the first cluster.

[0009] In another possible implementation, x vector data points are sampled from the vector data stored in the first cluster, where x is an integer greater than 1 and less than the total number of vector data points included in the first cluster. The second center is obtained by clustering based on these x vector data points. Clustering based on x vector data points reduces the amount of data to be clustered and improves the efficiency of clustering.

[0010] In another possible implementation, based on the second center, y vector data points are obtained from x vector data points, where y is a positive integer less than or equal to x. For each of the y vector data points, the distance between the vector data point and the second center is less than the distance between the vector data point and the first center. When the ratio between y and x exceeds a ratio threshold, the center of the first cluster is determined to change from the first center to the second center.

[0011] When the ratio between y and x exceeds the ratio threshold, it indicates that most of the vector data in the first cluster are closer to the second center. Therefore, it can be concluded that the center of the first cluster changes from the first center to the second center, that is, the actual center of the first cluster is the second center.

[0012] In another possible implementation, the vector database system further includes a first index graph for a first cluster. The first index graph includes nodes corresponding to each vector data in the first cluster. For any two nodes connected in the first index graph, the distance between the two nodes satisfies a distance condition. The system receives the first vector data to be stored. Based on a second center, the first vector data is saved to the first cluster. At least one second node is obtained from the first index graph whose distance to the first node corresponding to the first vector data satisfies the distance condition. The first node is added to the first index graph, and the first node is connected to at least one second node.

[0013] By obtaining at least one second node whose distance to the first node satisfies the distance condition, and connecting the first node and at least one second node, it is not necessary to rebuild the entire index graph, thus reducing the computational load of updating the index graph and improving efficiency.

[0014] In another possible implementation, multiple nodes with the smallest distance to the first node are obtained from the first index graph. At least one second node is selected from the multiple nodes, such that for each second node, the distance between the second node and the first node is less than the distance between any two of the at least one second node, or the distance between the second node and the first node is less than the product of the distance between the two second nodes and the target coefficient, where the target coefficient is greater than 1.

[0015] Since the distance between the second node and the first node is less than the distance between any two second nodes, or the distance between the second node and the first node is less than the product of the distance between the two second nodes and the target coefficient, at least one second node is closer to the first node. Connecting the first node with at least one second node can improve the connectivity of the index graph.

[0016] In another possible implementation, the vector database system also includes a correspondence between the identifier of each vector data in the first cluster and the position of the corresponding node. The identifier of the second vector data to be deleted is obtained. Based on the identifier of the second vector data, the second vector data is deleted from the first cluster, and the position of the third node corresponding to the second vector data is obtained from the correspondence. Based on the position of the third node, the third node is deleted from the first index graph, resulting in a second index graph. This way, the third node can be quickly found from the first index graph through the correspondence, improving the efficiency of deleting the third node.

[0017] In another possible implementation, a fourth node is obtained, which is a node in the first index graph that is connected to the third node. At least one fifth node is obtained from the second index graph whose distance to the fourth node satisfies a distance condition. The fourth node is then connected to at least one fifth node in the second index graph. Since the removal of the third node results in an empty neighbor node for the fourth node, the connectivity of the index graph is reduced. Therefore, obtaining at least one fifth node whose distance to the fourth node satisfies a distance condition and connecting the fourth node to at least one fifth node in the second index graph improves the connectivity of the second index graph.

[0018] In another possible implementation, the first cluster is also used to store metadata for each vector data in a plurality of vector data. A query request is received, including the third vector data to be queried and filtering conditions. Based on the third vector data and the filtering conditions, at least one cluster is determined to include the vector database system. The total number of vector data in each cluster that satisfy the filtering conditions and whose distance to the third vector data is less than a distance threshold is greater than or equal to K, where K is an integer greater than 1. From the at least one cluster, the K vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is the smallest are queried.

[0019] By first determining the query scope, i.e., identifying at least one cluster, and then querying the required vector data from at least one cluster, the query efficiency is improved compared to querying from all clusters.

[0020] In another possible implementation, the proportion corresponding to each cluster in the vector database system is obtained. This proportion is the ratio between the number of vector data whose metadata satisfies the filtering conditions and the total number of vector data in the cluster. Based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster, a first number is obtained for each cluster. This first number represents the number of vector data in the cluster whose distance to the third vector data is less than the distance threshold. Based on the first number and the proportion for each cluster, a second number is obtained for each cluster. This second number represents the number of vector data in the cluster whose metadata satisfies the filtering conditions and whose distance to the third vector data is less than the distance threshold. If the cumulative value of the second number for each cluster is greater than or equal to K, at least one cluster whose second number is greater than or equal to the count threshold is selected. This allows the second number for each cluster to be derived through the vector count prediction model, expanding the application scope and flexibility.

[0021] In another possible implementation, a vector count prediction model is trained based on multiple training samples, which include the fourth vector data, a distance threshold, the center of each cluster, and the number of vector data in each cluster whose distance to the fourth vector data is less than the distance threshold.

[0022] Secondly, this application provides an apparatus for processing vector data, used to perform the method in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus includes units for performing the method in the first aspect or any possible implementation of the first aspect.

[0023] Thirdly, this application provides a vector database system, which includes at least one computing device, each computing device including a processor and a memory;

[0024] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the vector database system to perform the method in the first aspect or any possible implementation thereof.

[0025] In one possible implementation, the vector database system is a cluster of computing devices.

[0026] Fourthly, this application provides a computer program product containing instructions that, when executed by a storage system, cause the storage system to perform the method of the first aspect or any possible implementation thereof.

[0027] Fifthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a storage system, perform the method of the first aspect or any possible implementation thereof.

[0028] In a sixth aspect, this application provides a chip including a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to perform the method in the first aspect or any possible implementation thereof. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of a vector database system provided in an embodiment of this application;

[0030] Figure 2 This is a flowchart of a method for saving vector data provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of an index diagram provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of another index diagram provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of another index diagram provided in an embodiment of this application;

[0034] Figure 6 This is a flowchart of a method for processing vector data provided in an embodiment of this application;

[0035] Figure 7 This is a flowchart of a method for deleting vector data provided in an embodiment of this application;

[0036] Figure 8 This is a flowchart of a method for querying vector data provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of a histogram provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of the device structure for processing vector data provided in Embodiment 3 of this application;

[0039] Figure 11 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of a cluster structure for processing vector data provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of another cluster structure for processing vector data provided in an embodiment of this application. Detailed Implementation

[0042] See Figure 1 This application provides a vector database system 100, which includes a coordinating node 101 and multiple data nodes 102. The coordinating node 101 can communicate with each data node 102.

[0043] The coordinating node 101 can initialize the number of clusters included in the vector database system 100 to M, where M is an integer greater than 1. That is, the vector database system 100 can include M clusters, which are distributed across multiple data nodes 102. The number of data nodes 102 is less than or equal to M, so each data node 102 includes one or more clusters.

[0044] Initially, the coordinating node 101, upon receiving vector data to be stored, randomly saves the vector data into a cluster included in the vector database system 100. In implementation,

[0045] Coordination node 101 is used to receive vector data to be stored, randomly select a cluster from M clusters, and send the vector data to the data node 102 where the selected cluster is located.

[0046] Data node 102 is used to receive vector data and save the vector data to the cluster selected by coordinating node 101.

[0047] Optionally, if the data node 102 containing the selected cluster includes multiple clusters, the coordinating node 101 is further configured to send the identifier of the selected cluster to the data node 102. The data node 102 is further configured to receive the identifier of the selected cluster and, based on the identifier, save the vector data into the selected cluster.

[0048] The coordinating node 101 can continuously receive vector data to be stored and then randomly save the vector data into a cluster included in the vector database system 100.

[0049] Optionally, vector data is a vector that includes data in multiple dimensions. For example, vector data is a vector that includes data in three dimensions, such as [1, 2, 3]. Another example is vector data that includes data in four dimensions, such as [11, 12, 43, 7].

[0050] When the number of vector data stored in the vector database system 100 exceeds the first threshold, the coordinating node 101 is also used to obtain the centers of M clusters based on the vector data stored in the M clusters, that is, a total of M centers are obtained.

[0051] Subsequently, the coordinating node 101, upon receiving vector data to be stored, obtains the distance between the vector data and the M centers, selects the center with the smallest distance from the vector data, determines the cluster corresponding to the selected center, and sends the vector data to the data node 102 where that cluster resides. The data node 102 is also used to receive the vector data and save it into that cluster.

[0052] In the relevant technology, after the coordination node 101 first obtains M centers, it will always save the vector data that needs to be stored each time it receives it into a cluster in the vector database system 100 based on the M centers.

[0053] However, as time goes on, the distribution of the vector data that needs to be stored in the vector database system 100 changes continuously, which will cause the centers of all or some of the M clusters included in the vector database system 100 to change. This change in cluster centers can also be called cluster center drift. Thus, for clusters whose centers have drifted, these clusters may contain a large amount of vector data that does not belong to them, which will reduce the accuracy of querying vector data from the vector database system 100.

[0054] The following examples will be used to describe in detail the process of storing vector data in the vector database system 100, the process of discovering clusters whose centers have changed and migrating out vector data that does not belong to the clusters, the process of querying vector data from the vector database 100, and the process of deleting vector data.

[0055] See Figure 2 This application provides a method 200 for storing vector data, which is applied to... Figure 1 The vector database system 100 is shown. The method 200 is used to receive vector data to be stored and save the vector data into a cluster within the vector database system 100. The method 200 includes the following steps.

[0056] Step 201: The coordinating node initializes the M clusters included in the vector database system. These M clusters are located on multiple data nodes included in the vector database system, where M is an integer greater than 1.

[0057] In step 201, the coordinating node initializes the vector database system with M clusters and determines the data node for each of the M clusters.

[0058] Optionally, if the vector database system includes M data nodes, then each data node includes one cluster. If the vector database system includes less than M data nodes, then some data nodes may include one cluster, while others may include multiple clusters.

[0059] Step 202: The coordinating node receives the target vector data to be stored, randomly selects a cluster, and sends the target vector data to the data node where the cluster is located.

[0060] In some embodiments, if the data node containing the cluster includes multiple clusters, the coordinating node also sends the identifier of the cluster to the data node.

[0061] Step 203: The data node receives the target vector data and saves the target vector data into the cluster.

[0062] In step 203, if a data node includes a cluster, after receiving the target vector data, the data node saves the target vector data into that cluster. If a data node includes multiple clusters, the data node also receives the identifier of the cluster and saves the target vector data into the cluster corresponding to the identifier.

[0063] In some embodiments, data nodes may also acquire metadata of the target vector data and save the metadata of the target vector data into the cluster.

[0064] Optionally, the metadata of the target vector data may include one or more of the following: the identifier of the target vector data, the type of the target vector data, or the timestamp of the target vector data.

[0065] Optionally, the timestamp of the target vector data may be the timestamp of the creation of the target vector data, and / or the timestamp of the storage of the target vector data, etc.

[0066] Optionally, the category of the target vector data may be the place of origin of the target vector data, and / or the user to which the target vector data belongs, etc.

[0067] In some embodiments, the cluster may be a data table, where data nodes save target vector data, and the identifier of the target vector data may be the row number of the row containing the target vector data. Alternatively, the cluster may be a text file, such as an Extensible Markup Language (XML) file, where data nodes save target vector data, and the identifier of the target vector data may be the sequence number of the target vector data.

[0068] The coordination node and the data nodes in the vector database system can repeat the operations of steps 201-203 above so that when the target vector data that needs to be saved is received again, the target vector data is randomly saved to a cluster in the vector database system.

[0069] When the number of vector data stored in the vector database system exceeds the first threshold, the following process can be executed.

[0070] Step 204: The coordinating node obtains M centers based on the vector data stored in the M clusters. These M centers are the centers of the M clusters.

[0071] In some embodiments, for each cluster and for the data node containing that cluster, the data node may sample z vector data points from the vector data stored in that cluster, where z is an integer greater than 1, and send these z vector data points to the coordinating node. Optionally, the data node may randomly sample z vector data points from the vector data stored in that cluster.

[0072] In this vector database system, other data nodes also send z vector data samples from each cluster to the coordinating node in the same manner. Therefore, the coordinating node may receive a total of z*M vector data, and then perform clustering based on these z*M vector data to obtain M cluster centers. The coordinating node can also notify each data node of the M cluster centers.

[0073] The number of vector data stored in M ​​clusters is much larger than z*M. If the coordinating node performs clustering based on the vector data stored in M ​​clusters, the amount of data for clustering operations becomes extremely large, reducing the efficiency of clustering operations. Therefore, z vector data can be randomly sampled from each cluster, resulting in a total of z*M vector data. The coordinating node then performs clustering based on these z*M vector data, reducing the amount of data for clustering operations and improving the efficiency of clustering operations.

[0074] Optionally, the coordinating nodes perform clustering using a clustering algorithm. For example, the clustering algorithm could be k-means or similar algorithms. The center of each cluster is a vector data point, which may be a single vector data point within that cluster or may be different from every other vector data point in that cluster.

[0075] For each of the M clusters, for ease of explanation, this cluster is referred to as the first cluster, the data node containing the first cluster is referred to as the first data node, and the center of the first cluster is referred to as the first center. After receiving the M centers, the first data node can perform a redistribution process to obtain at least one vector data belonging to the second cluster stored in the first cluster, and then migrate the at least one vector data from the first cluster to the second cluster. The M clusters other than the first cluster include the second cluster.

[0076] The redistribution process performed by the first data node may include the following steps (11)-(12).

[0077] (11): The first data node receives M centers and obtains at least one vector data belonging to the second cluster based on the M centers.

[0078] For each vector data stored in the first cluster, the first data node obtains the distance between the vector data and each of the M centers. It selects the center with the smallest distance from the M centers. If the selected center is the first center, the vector data is determined to belong to the first cluster; otherwise, the selected center is determined to belong to the second cluster, and the vector data is the vector data stored in the first cluster belonging to the second cluster.

[0079] Following the above process, the first data node can obtain at least one vector data belonging to the second cluster that is stored in the first cluster.

[0080] (12): The first data node migrates at least one vector data from the first cluster to the second cluster.

[0081] In some embodiments, the second cluster is also located on the first data node, and the first data node saves the at least one vector data into the second cluster and deletes the at least one vector data from the first cluster.

[0082] In some embodiments, the second cluster is not located on the first data node. The first data node determines the second data node where the second cluster is located, sends the at least one vector data to the second data node, and deletes the at least one vector data from the first cluster. The second data node receives the at least one vector data and saves it into the second cluster.

[0083] If the second data node includes multiple clusters, the first data node also sends the identifier of the second cluster to the second data node. The second data node receives the identifier of the second cluster and, based on the identifier, saves at least one vector data into the second cluster.

[0084] In some embodiments, after the first data node migrates all vector data that does not belong to the first cluster stored in the first cluster, the first data node can obtain the distance between any two nodes in the first cluster and establish a first index graph of the first cluster based on the distance between any two nodes.

[0085] The first index graph includes nodes corresponding to each vector data stored in the first cluster. For any two nodes in the first index graph that are connected and related, the distance between the two nodes is equal to the distance between the two vector data corresponding to the two nodes, and the distance between the two nodes satisfies the distance condition.

[0086] Optionally, in implementation, the first data node can establish the first index graph of the first cluster through the following (21)-(25) process.

[0087] (21): Obtain the node set, which includes the node corresponding to each vector data in the first cluster.

[0088] For example, see Figure 3 The obtained node set includes node1, node2, node3, node4, node5, node6, node7 and node8.

[0089] (22): Get the distance between any two nodes in the node set.

[0090] Optionally, for any two nodes in the node set, obtain the two vector data corresponding to the two nodes, calculate the distance between the two vector data, and use the calculated distance as the distance between the two nodes.

[0091] (23): For any node in the node set, the node is called the first target node, and multiple nodes with the smallest distance from the first target node are obtained from the node set.

[0092] For example, taking node 1 as the first target node, the four nodes with the smallest distance from node 1 are obtained from the node set as node 2, node 3, node 4 and node 6.

[0093] (24): Select at least one second target node from the plurality of nodes. For each second target node, the distance between the second target node and the first target node is less than the distance parameter between any two second target nodes in the plurality of nodes.

[0094] Optionally, the distance parameter between two second target nodes includes one or more of the following: the distance between the two second target nodes, or the product of the distance between the two second target nodes and the target coefficient, where the target coefficient is greater than 1. Optionally, the target coefficient is a value such as 1.1, 1.2, 1.3, 1.4, or 1.5.

[0095] For example, assuming the target coefficient is 1.5, such as Figure 3 As shown, the distances 12 between node 1 and node 2, 13 between node 1 and node 3, and 14 between node 1 and node 4 are less than 1.5 times the distance between node 2 and node 4, less than 1.5 times the distance between node 2 and node 3, and less than 1.5 times the distance between node 3 and node 4. However, the distance between node 1 and node 6 is greater than 1.5 times the distance between node 3 and node 6. Therefore, at least one second target node selected from nodes 2, 3, 4, and 6 includes nodes 2, 3, and 4.

[0096] (25): Connect the first target node and at least one second target node.

[0097] For example, see Figure 3 Connect node 1 to node 2, connect node 1 to node 4, and connect node 1 to node 3.

[0098] If there are any remaining nodes in the node set that are not the first target node, select one of the remaining nodes as the first target node, and then repeat the process of steps (23)-(25) above. If there are no remaining nodes in the node set that are not the first target node, then use the graph obtained by connecting them as the first index graph of the first cluster.

[0099] In some embodiments, the first data node further obtains the identifier of each vector data in the first cluster from the first cluster, and obtains the position of the node corresponding to each vector data from the first index graph, and stores the identifier of each vector data and the position of the node corresponding to each vector data in the correspondence between the identifier of the vector data and the position of the node.

[0100] For other data nodes in the vector database system besides the first data node, the other data nodes will also perform the above redistribution process and build an index graph for each cluster they include, just like the first data node.

[0101] After obtaining M centers, the coordinating node, upon receiving the vector data to be stored, can save the vector data based on the M centers according to the following process. Optionally, the data nodes can perform the redistribution process asynchronously; that is, the data nodes can perform the redistribution process while executing the following process.

[0102] Step 205: The coordinating node receives the first vector data to be stored, obtains the distance between the first vector data and each center, and selects the center with the smallest distance from the M centers. Assume that the selected center is the first center.

[0103] Next, the coordinating node saves the first vector data into the first cluster corresponding to the first center. The detailed implementation process is as follows.

[0104] Step 206: The coordinating node determines the first cluster corresponding to the first center and sends the first vector data to the first data node where the first cluster is located.

[0105] In some embodiments, if the first data node containing the first cluster includes multiple clusters, the coordinating node also sends the identifier of the first cluster to the first data node.

[0106] Step 207: The first data node receives the first vector data and saves the first vector data into the first cluster.

[0107] In step 207, if the first data node includes one cluster, after receiving the first vector data, the first data node saves the first vector data into the first cluster. If the first data node includes multiple clusters, the first data node also receives the identifier of the first cluster, and saves the first vector data into the first cluster based on the identifier of the first cluster.

[0108] In some embodiments, the first data node may also acquire the metadata of the first vector data and save the metadata of the first vector data into the first cluster.

[0109] Step 208: The first data node obtains at least one second node from the first index graph of the first cluster that has a distance that satisfies the distance condition between it and the first node corresponding to the first vector data.

[0110] In step 208, the first data node can obtain multiple nodes with the smallest distance from itself by greedily searching for neighbors, and then select at least one second node from these multiple nodes. In the greedy neighbor search method, the first vector data is used as the coordinates of the first node in the first index graph and placed in the first index graph, and then the first queue and the second queue are used to find multiple nodes.

[0111] Optionally, in implementation, at least one second node can be obtained through the following process 2081-2085.

[0112] 2081: Add the third target node to the second queue, dequeue the third target node from the second queue, add the third target node to the first queue, add at least one neighbor node of the third target node to the second queue, and sort the nodes in the second queue based on the distance between the first node and each node in the second queue.

[0113] In the sorted second queue, the node at the head of the queue has the smallest distance from the first node, and the node at the tail of the queue has the largest distance from the first node.

[0114] The third target node is any node in the first index graph, and at least one of its neighbors is a node in the first index graph that is connected to the third target node. That is, first, any node is selected from the first index graph as the third target node, and the third target node is added to the second queue. At this point, the third target node is at the head of the second queue, so it is dequeued from the second queue and added to the first queue.

[0115] For example, see Figure 4Assuming the third target node is node 1, first add node 1 to the second queue, then dequeue node 1 from the second queue and add it to the first queue. Obtain the neighboring nodes of node 1, which are nodes 2, 3, and 4. Add nodes 2, 3, and 4 to the second queue. Based on the distances between node 2 and the first node, the distances between node 3 and the first node, and the distances between node 4 and the first node, sort nodes 2, 3, and 4 in the second queue in ascending order of distance. The distance between node 4 and the first node is the smallest, and the distance between node 3 and the first node is smaller than the distance between node 2 and the first node. Therefore, in the sorted second queue, node 4 is at the head of the queue, and node 3 is between node 4 and node 2. That is, the node sequence stored in the second queue is node 4, node 3, and node 2.

[0116] The first queue is a fixed-length queue. Assume the length of the first queue is Q, where Q is an integer greater than 1. The first queue is used to store the Q nodes that are closest to the first node.

[0117] 2082: Dedicate a node from the head of the second queue as the fourth target node, add the fourth target node to the first queue, and sort the nodes in the first queue based on the distance between the first node and each node in the first queue.

[0118] Since the length of the first queue is Q, if adding the fourth target node to the first queue causes the number of nodes in the first queue to exceed Q, the node with the largest distance from the first node will be automatically dequeued from the first queue.

[0119] For example, the head of the second queue is node 4. Node 4 is dequeued from the head of the second queue and added to the first queue. The first queue includes nodes 1 and 4. The distance between node 4 and the first node is less than the distance between node 1 and the first node. Therefore, the sorted first queue stores the node sequence of nodes 4 and 1.

[0120] 2083: Determine if there are any nodes among the at least one neighboring nodes of the fourth target node that have not been added to the second queue. If there are nodes that have not been added to the second queue, add the nodes that have not been added to the second queue to the second queue. Sort the nodes in the second queue based on the distance between the first node and each node in the second queue. Return to execute 2082.

[0121] See Figure 4In the first index graph, at least one neighboring node of node 4 includes nodes 2, 7, and 8, which are connected to node 4. Nodes 7 and 8 are not currently in the second queue, so they are added to the second queue, which then includes nodes 3, 2, 7, and 8. Based on the distances between the first node and nodes 3, 2, 7, and 8 in the second queue, the nodes in the second queue are sorted. The sorted second queue stores the node sequence 8, 7, 3, and 2.

[0122] The head of the second queue is node 8. Node 8 is dequeued from the head of the second queue and added to the first queue. The first queue includes nodes 4, 1, and 8. Based on the distances between the first node and nodes 4, 1, and 8 in the first queue, the nodes 4, 1, and 8 in the first queue are sorted. The sorted first queue stores the node sequence as nodes 8, 4, and 1.

[0123] See Figure 4 In the first index graph, at least one neighboring node of node 8 includes nodes 4, 7, and 5, which are connected to node 8. Node 5 is not in the second queue, so it is added to the second queue, which includes nodes 7, 3, 2, and 5. Based on the distances between the first node and nodes 7, 3, 2, and 5 in the second queue, the nodes 7, 3, 2, and 5 in the second queue are sorted. The sorted second queue stores the node sequence 7, 5, 3, and 2.

[0124] The head of the second queue is node 7. Node 7 is dequeued from the head of the second queue, which includes nodes 5, 3, and 2. Node 7 is added to the first queue, which includes nodes 8, 4, 1, and 7. Nodes 8, 4, 1, and 7 in the first queue are sorted based on their distances to the first node. The sorted first queue contains the node sequence 8, 7, 4, and 1.

[0125] 2084: If there are no nodes that have not been added to the second queue, determine if the second queue is empty. If the second queue is not empty, return to execute 2082. If the second queue is empty, execute 2085.

[0126] If the second queue is empty, the multiple nodes stored in the first queue are the nodes with the smallest distance to the first node.

[0127] For example, see Figure 4In the first index graph, at least one neighboring node of node 7 includes nodes 3, 4, and 8, which are connected to node 7. Nodes 3, 4, and 8 have all been added to the second queue, so there are no nodes among at least one neighboring node of node 7 that have not been added to the second queue.

[0128] The head of the second queue is node 5. Node 5 is dequeued from the head of the second queue. The second queue includes nodes 3 and 2. Node 5 is added to the first queue, which includes nodes 8, 7, 4, 1, and 5. Nodes 8, 7, 4, 1, and 5 in the first queue are sorted based on their distances to the first node. Assuming Q = 4, after sorting the nodes in the first queue, node 1, which has the largest distance to the first node, is dequeued. Therefore, the sorted first queue contains the node sequence 8, 7, 5, and 4.

[0129] The head of the second queue is node 3. Node 3 is dequeued from the head of the second queue, which includes node 2. Node 3 is added to the first queue, which includes nodes 8, 7, 5, 4, and 3. Nodes 8, 7, 5, 4, and 3 in the first queue are sorted based on their distances to the first node. After sorting the nodes in the first queue, node 3, which has the largest distance to the first node, is dequeued. Therefore, the sorted first queue contains the node sequence 8, 7, 5, and 4.

[0130] See Figure 4 In the first index graph, at least one neighboring node of node 3 includes nodes 1, 7, and 6, which are connected to node 3. Node 6 is not in the second queue, so it is added to the second queue, which includes nodes 2 and 6. Based on the distances between the first node and nodes 2 and 6 in the second queue, nodes 2 and 6 in the second queue are sorted. The sorted second queue stores the node sequence of nodes 2 and 6.

[0131] The head of the second queue is node 2. Node 2 is dequeued from the head of the second queue, which includes node 6. Node 2 is added to the first queue, which includes nodes 8, 7, 5, 4, and 2. Nodes 8, 7, 5, 4, and 2 in the first queue are sorted based on their distances to the first node and to each of these nodes. After sorting the nodes in the first queue, node 2, which has the largest distance to the first node, is dequeued. Therefore, the sorted first queue contains the node sequence 8, 7, 5, and 4.

[0132] See Figure 4 In the first index graph, at least one neighboring node of node 2 includes nodes 1, 5, and 4, which are connected to node 2. Nodes 1, 5, and 4 are all added to the second queue. The head of the second queue is node 6. Node 6 is dequeued from the head of the second queue, making the second queue empty. Node 6 is added to the first queue, which includes nodes 8, 7, 5, 4, and 6. Nodes 8, 7, 5, 4, and 6 in the first queue are sorted based on their distances to the first node. After sorting the nodes in the first queue, node 6, which has the largest distance to the first node, is dequeued from the first queue. Therefore, the sorted first queue contains the node sequence 8, 7, 5, and 4.

[0133] Since the second queue is empty, nodes 8, 7, 5, and 4 in the first queue are the four nodes with the smallest distance to the first node.

[0134] 2085: Select at least one second node from the multiple nodes stored in the first queue. For each second node, the distance between the second node and the first node is less than the distance parameter between any two second nodes in the first queue.

[0135] In some embodiments, the plurality of nodes can be selected as a plurality of second nodes, and then the first node is added to the first index graph and connected to the plurality of second nodes.

[0136] In some embodiments, at least one second node can be selected from a plurality of nodes. Since the distance between the second node and the first node is less than the distance parameter between any two of the at least one second node, the second node is closer to the first node than other nodes. Connecting the first node to the at least one second node can increase the connectivity of the first index graph.

[0137] The distance parameter between two second nodes includes one or more of the following: the distance between the two second nodes, or the product of the distance between the two second nodes and the target coefficient.

[0138] If the distance parameter between two second nodes is simply the distance between the two second nodes, then fewer second nodes can be selected from multiple nodes. However, if the target coefficient is a value greater than 1, then the distance between two second nodes can be increased by the target coefficient, thus allowing more second nodes to be selected from multiple nodes.

[0139] Step 209: The first data node adds the first node to the first index graph and connects the first node with at least one second node.

[0140] For example, see Figure 5 The first data node is selected from nodes 8, 7, 5, and 4, and at least one second node is selected from nodes 8, 7, and 5. The first node (node ​​9) is added to the first index graph, and node 9 is associated with nodes 8, 7, and 5 respectively.

[0141] In some embodiments, the first data node also obtains the identifier of the first vector data and the position of the first node in the first index graph, and stores the identifier of the first vector data and the position of the first node in the correspondence between the identifier of the vector data and the position of the node.

[0142] When there is vector data that needs to be stored in the vector database system again, the coordination node and the data nodes in the vector database system repeat the process of steps 205-209 above to save the vector data into a cluster in the vector database system.

[0143] If the time elapsed since the last acquisition of cluster centers reaches a specified duration, or if the number of vector data stored in the vector database system since the last acquisition of cluster centers exceeds a second threshold, then the centers of some clusters in the vector database system may drift. It is necessary to reacquire the centers of these clusters and use a redistribution process to migrate the vector data that does not belong to these clusters.

[0144] Optionally, the vector database system can asynchronously obtain the cluster centers and perform a redistribution process. See the following for details. Figure 6 The embodiments shown will not be described in detail here.

[0145] In this embodiment, after saving the first vector data to be stored to the first cluster, the first data node obtains at least one second node from the first index graph of the first cluster whose distance to the first node corresponding to the first vector data satisfies the distance condition. The first node is then added to the first index graph, and the first node is connected to the at least one second node. This way, after saving the first vector data, it is not necessary to reconstruct the first index graph based on the vector data saved in the first cluster. Compared to reconstructing the first index graph, obtaining at least one second node whose distance to the first node satisfies the distance condition and connecting the first node to the at least one second node involves less computation, thereby improving the efficiency of obtaining the index graph.

[0146] See Figure 6 This application provides a method 600 for processing vector data, which is applied to... Figure 1 The vector database system shown. Method 600 is used to determine the first cluster whose center has changed (drifted), and to migrate the vector data belonging to the second cluster stored in the first cluster to the second cluster. Method 600 includes the following process.

[0147] Step 601: Obtain the second center based on the vector data saved by the first cluster.

[0148] The data node containing the first cluster is called the first data node, and the center of the first cluster is called the first center. The vector data stored in the first cluster is the vector data selected by the coordinating node based on the first center.

[0149] When the time elapsed since the last acquisition of cluster centers reaches a specified duration, or when the number of vector data points saved to the vector database system since the last acquisition of cluster centers exceeds a second threshold, each data node in the vector database system begins acquiring the centers of its respective clusters. The process by which each data node acquires its cluster centers is the same as the process by which the first data node acquires its second centers.

[0150] In some embodiments, the first data node can sample x vector data from the vector data stored in the first cluster, where x is an integer greater than 1 and less than the total number of vector data included in the first cluster, and cluster the x vector data to obtain the second center.

[0151] Optionally, the first data node can randomly sample x vector data from the vector data stored in the first cluster, so that the x vector data are distributed more evenly in the first cluster. Clustering x vector data in this way can improve the accuracy of the second centers obtained from the clustering. Furthermore, clustering x vector data reduces the amount of data to be clustered and improves clustering efficiency.

[0152] Optionally, the first data nodes are clustered using a clustering algorithm to obtain the second centers. For example, the clustering algorithm could be the k-means algorithm, etc.

[0153] Step 602: Determine whether the center of the first cluster has changed from the first center to the second center based on the second center. If the center of the first cluster has changed from the first center to the second center, proceed to step 603.

[0154] The change of the center of the first cluster to the second center can also be described as the center of the first cluster drifting from the first center to the second center.

[0155] In step 602, the first data node determines whether the center of the first cluster has changed from the first center to the second center through the following operations 6021-6022.

[0156] 6021: The first data node is based on the second center and obtains y vector data from x vector data, where y is a positive integer less than or equal to x. The distance between each of the y vector data and the second center is less than the distance between each vector data and the first center.

[0157] In 6021, for each of the x vector data points, the first data node obtains the distance between each vector data point and the first center, and the distance between each vector data point and the second center. The distances between each vector data point and the first center, and between each vector data point and the second center, are compared to determine the y vector data points whose distance to the second center is less than their distance to the first center.

[0158] 6022: When the ratio between y and x of the first data node exceeds the ratio threshold, the center of the first cluster is determined to change from the first center to the second center. When the ratio between y and x does not exceed the ratio threshold, the center of the first cluster remains unchanged and is still the first center.

[0159] Step 603: Obtain at least one vector data based on the second center, wherein the at least one vector data is data belonging to the second cluster that is stored in the first cluster.

[0160] For each data node in the vector database system other than the first data node, it determines, in the same way as the first data node, whether the centers of its own clusters have changed. If it determines that the centers of its own clusters have changed, it notifies the other data nodes and the coordinating node in the vector database system of the new cluster centers. Similarly, the first data node notifies the other data nodes and the coordinating node in the vector database system of the second centers of the first cluster. Therefore, the first data node has the centers of M-1 other clusters in the vector database system besides the first cluster.

[0161] For example, if the center of the second cluster changes, the center of the second cluster on the first data node becomes the new center. If the center of the second cluster does not change, the center of the second cluster on the first data node becomes the old center.

[0162] In step 603, for any vector data stored in the first cluster, the first data node obtains the distance between the vector data and each of the M centers, and selects the center with the smallest distance to the vector data. If the selected center is not the second center, then the vector data does not belong to the first cluster, but to the cluster corresponding to the selected center. For ease of explanation, the cluster corresponding to the selected center is called the second cluster.

[0163] Perform the same operation as described above on each of the other vector data stored in the first cluster to obtain at least one vector data belonging to the second cluster stored in the first cluster. Furthermore, the distance between each of these at least one vector data and the center of the second cluster is less than the distance between each vector data and the second cluster center.

[0164] In some embodiments, the coordinating node receives notification of the second center of the first cluster and updates the locally stored first center of the first cluster to the second center. When the coordinating node receives vector data to be stored, it obtains the distance between the vector data and each of the M locally stored centers. It selects the center with the smallest distance to the vector data; assuming the selected center is the second center. The coordinating node determines the first cluster corresponding to the second center and sends the vector data to the first data node where the first cluster resides. The first data node saves the vector data into the first cluster. For detailed implementation details, see [link to implementation details]. Figure 2 Steps 205-209 of method 200 shown will not be described in detail here.

[0165] Step 604: Transfer at least one vector data from the first cluster to the second cluster.

[0166] In some embodiments, the second cluster is also located on the first data node, and the first data node saves the at least one vector data into the second cluster and deletes the at least one vector data from the first cluster.

[0167] In some embodiments, the second cluster is not located on the first data node. The first data node determines the second data node where the second cluster is located, sends the at least one vector data to the second data node, and deletes the at least one vector data from the first cluster. The second data node receives the at least one vector data and saves it into the second cluster.

[0168] If the second data node includes multiple clusters, the first data node also sends the identifier of the second cluster to the second data node. The second data node receives the identifier of the second cluster and, based on the identifier, saves at least one vector data into the second cluster.

[0169] After migrating at least one vector data from the first cluster to the second cluster, it is also necessary to delete at least one node corresponding to at least one vector data from the first index graph of the first cluster.

[0170] In implementation, the first data node, based on the identifier of at least one vector data, obtains the position of at least one node corresponding to at least one vector data from the correspondence between the vector data identifier and the node position. Based on the position of at least one node, at least one node is deleted from the first index graph.

[0171] In some embodiments, after a node is deleted from the first index graph, a neighbor node becomes available for other nodes connected to that node. To improve the throughput of the first index graph, at least one node whose distance to the other node satisfies a distance condition can be obtained from the deleted node in the first index graph. Then, the other node and the at least one node are connected in the deleted node in the first index graph.

[0172] Optionally, for a detailed implementation process of obtaining at least one node whose distance to the other nodes satisfies the distance condition, please refer to [link to relevant documentation]. Figure 2 The steps 208 of method 200 shown include the processes 2081-2085, which will not be described in detail here.

[0173] In some embodiments, after migrating at least one vector data from the first cluster to the second cluster, it is also necessary to add at least one node corresponding to the at least one vector data to the index graph of the second cluster.

[0174] In implementation, for each vector data in the at least one vector data, and for the node corresponding to the vector data, at least one node whose distance from the node satisfies the distance condition is obtained from the index graph of the second cluster, the node is added to the index graph of the second cluster, and the node is connected to the at least one node.

[0175] Optionally, for a detailed implementation process of obtaining at least one node whose distance from this node satisfies the distance condition, please refer to [link to documentation]. Figure 2 The steps 208 of method 200 shown include the processes 2081-2085, which will not be described in detail here.

[0176] In this embodiment, the first data node obtains a second center based on the vector data stored in the first cluster. The second center is used to determine whether the center of the first cluster has changed from the first center to the second center. If the center of the first cluster changes, the vector data that does not belong to the first cluster can be identified based on the second center. This vector data is then migrated to the second cluster to which it belongs. This ensures that vector data belonging to the first cluster remains within the first cluster, preventing the first cluster from including a large amount of non-belonging vector data. This improves the accuracy of data queries when querying the vector database system.

[0177] See Figure 7 This application provides a method 700 for deleting vector data, which is applied to... Figure 1 The vector database system shown. Method 700 is used to delete vector data from a first cluster. Method 600 includes the following steps.

[0178] Step 701: Obtain the identifier of the second vector data to be deleted.

[0179] In some embodiments, the coordinating node receives a first deletion request, which includes an identifier of the second vector data, determines the first cluster in which the second vector data belongs, and sends the first deletion request to the first data node in which the first cluster belongs. The first data node receives the first deletion request and obtains the identifier of the second vector data from the first deletion request.

[0180] In some embodiments, the coordinating node receives a second deletion request, which includes deletion conditions, and sends the second deletion request to each data node in the vector database system. The first data node receives the second deletion request and retrieves the identifier of the second vector data that meets the deletion conditions from the vector data stored in the first cluster.

[0181] When other data nodes receive a second deletion request, they execute the same process as the first data node.

[0182] In some embodiments, deletion conditions may include one or more of the following: the identifier range of the vector data to be deleted, the timestamp range of the vector data to be deleted, or the target type of the vector data to be deleted.

[0183] Optionally, the first cluster includes metadata for each vector data, which may include one or more of the following: the vector data's identifier, the vector data's type, or the vector data's timestamp.

[0184] In some embodiments, the deletion criteria include the identifier range of the vector data to be deleted, and the second vector data is the vector data in the first cluster whose identifiers fall within that identifier range.

[0185] In some embodiments, the deletion criteria include the timestamp range of the vector data to be deleted, and the second vector data is the vector data in the first cluster whose timestamps fall within that timestamp range.

[0186] In some embodiments, the deletion criteria include the target type of the vector data to be deleted, and the second vector data is the vector data of the target type in the first cluster.

[0187] Step 702: Based on the identifier of the second vector data, delete the second vector data from the first cluster, and obtain the position of the third node corresponding to the second vector data from the correspondence between the identifier of the vector data and the node position.

[0188] The position of the third node is the position of the third node in the first index graph, which is the index graph of the first cluster.

[0189] Step 703: Delete the third node from the first index graph based on the position of the third node to obtain the second index graph.

[0190] Based on the location of the third node, the third node can be quickly located in the first index graph, thereby improving the efficiency of deleting the third node from the first index graph.

[0191] Step 704: Obtain the fourth node, which is a node in the first index graph that has a connection relationship with the third node.

[0192] Step 705: Obtain at least one fifth node from the second index graph whose distance to the fourth node satisfies the distance condition.

[0193] Optionally, for a detailed implementation of obtaining at least one fifth node whose distance from the fourth node satisfies the distance condition, please refer to [link to documentation]. Figure 2 The steps 208 of method 200 shown include the processes 2081-2085, which will not be described in detail here.

[0194] Step 706: Connect the fourth node to at least one fifth node in the second index graph.

[0195] In this embodiment, before deleting the third node, there is a fourth node connected to the third node in the first index graph, and the third node is a neighbor node of the fourth node. After deleting the third node from the first index graph, a second index graph is obtained. In the second index graph, the fourth node has an empty neighbor node, affecting the connectivity of the second index graph. Therefore, at least one fifth node whose distance from the fourth node meets the distance condition is obtained from the second index graph, and the fourth node is connected to at least one fifth node in the second index graph. At least one fifth node is a neighbor node of the fourth node, thereby enriching the neighbors of the fourth node, improving the connectivity of the second index graph, and thus improving query efficiency when querying data.

[0196] See Figure 8 This application provides a method 800 for querying vector data, which is applied to... Figure 1 The vector database system shown. The method 800 is used to query vector data from a first cluster. The method 800 includes the following process.

[0197] Step 801: Receive a query request, which includes the third vector data to be queried and the filtering conditions.

[0198] In some embodiments, the coordinating node receives a query request, which includes third vector data to be queried and filtering conditions.

[0199] In some embodiments, the filtering conditions may include one or more of the following: the target identifier range of the vector data to be queried, the target timestamp range of the vector data to be queried, or the target type of the vector data to be queried.

[0200] For example, the filtering condition can be to query vector data with an identifier less than 1000, or to query vector data with timestamps between t0 and t1, or the category of vector data can be the origin of the vector data, and the filtering condition can be to query vector data whose origin is a certain city.

[0201] Step 802: Based on the third vector data and the filtering conditions, determine at least one cluster included in the vector database system. The total number of vector data in the at least one cluster that satisfies the filtering conditions and whose distance to the third vector data is less than a distance threshold is greater than or equal to K, where K is an integer greater than 1.

[0202] In step 802, the coordinating node can obtain at least one cluster through the operations described in steps 8021-8026.

[0203] 8021: Obtain the percentage corresponding to each cluster in the vector database system. The percentage corresponding to each cluster is the ratio between the number of vector data whose metadata in the cluster meets the filtering conditions and the total number of vector data in the cluster.

[0204] The coordinating node includes statistics for each cluster. The statistics for a cluster may include the number of vector data within different identifier ranges in the cluster, the number of vector data within different timestamp ranges in the cluster, and / or the number of vector data of different types in the cluster.

[0205] For example, see Figure 9 The coordinating node includes a cluster histogram, meaning the cluster statistics are presented in histogram form. The horizontal axis of the histogram includes different label ranges (or different timestamp ranges or different types), and the height of the rectangle corresponding to any label range represents the number of vector data within that label range. Figure 9 In the ID: 0-100, it represents the range of identifiers that are greater than or equal to 0 and less than or equal to 100; ID: 101-200 represents the range of identifiers that are greater than or equal to 101 and less than or equal to 200; and ID: 201-300 represents the range of identifiers that are greater than or equal to 201 and less than or equal to 300.

[0206] In 8021, for each cluster in the vector database system, the coordinating node obtains the number of vector data whose metadata in the cluster satisfies the filtering conditions based on the statistical information of the cluster. Based on the number of vector data whose metadata in the cluster satisfies the filtering conditions and the total number of vector data in the cluster, the corresponding proportion of the cluster is obtained.

[0207] In some embodiments, the filtering conditions include a target identifier range, obtaining at least one identifier range that intersects with the target identifier range, and obtaining the number of vector data within the target identifier range based on the number of vector data within the at least one identifier range. The number of vector data within the target identifier range is the number of vector data whose metadata included in the cluster satisfies the filtering conditions.

[0208] For example, assuming the target identifier range is ID less than 150, the histogram of a certain cluster in the vector database system is as follows: Figure 9 The histogram shown. Identifier ranges that intersect with the target identifier range include IDs: 0-100 and IDs: 101-200. From... Figure 9 The cluster histogram shows that the number of vector data in the ID range of 0-100 is 1000, and the number of vector data in the ID range of 101-200 is 1200.

[0209] The ratio between the length of the intersection of the identifier range ID:101-200 and the target identifier range and the length of the identifier range ID:101-200 is 0.5. Therefore, the number of vector data in the target identifier range is 1000 + 0.5 * 1200 = 1600.

[0210] When the filtering conditions include a target timestamp range, the process of obtaining the number of vector data within the target timestamp range can refer to the process of obtaining the number of vector data within the target identifier range described above, and will not be explained in detail here.

[0211] In some embodiments, the filtering criteria include a target type, which includes one or more types. The number of vector data corresponding to each type can be obtained from the histogram of the cluster. The number of vector data corresponding to each type is summed to obtain the number of vector data whose metadata in the cluster satisfies the filtering criteria.

[0212] 8022: Based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster, obtain the first number corresponding to each cluster. The first number corresponding to the cluster is the number of vector data included in the cluster whose distance to the third vector data is less than the distance threshold.

[0213] The initial value for the distance threshold can be a specified threshold. For example, assuming the specified threshold is 0.2, the initial value for the distance threshold is 0.2.

[0214] In 8022, the coordinating node inputs the third vector data, a distance threshold, and the center of each cluster in the vector database system into the vector count prediction model. The vector count prediction model infers from the third vector data, the distance threshold, and the center of each cluster to obtain the number of vectors in each cluster whose distance to the third vector data is less than the distance threshold. The coordinating node then obtains the output of the vector count prediction model: the number of vectors in each cluster whose distance to the third vector data is less than the distance threshold.

[0215] 8023: Based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster is obtained. The second number corresponding to the cluster is the number of vector data whose metadata in the cluster meets the filtering conditions and whose distance to the third vector data is less than the distance threshold.

[0216] In 8023, the coordinating node multiplies the first number corresponding to the cluster with the proportion corresponding to the cluster to obtain the second number corresponding to the cluster.

[0217] 8024: Accumulate the second number corresponding to each cluster to obtain the total number of vector data in the vector database system that meet the filtering conditions and whose distance to the third vector data is less than the distance threshold. If the total number is less than K, execute 8025. If the total number is greater than or equal to K, execute 8026.

[0218] K is the number of vector data that need to be queried from the vector database system, which meet the filtering conditions and whose distance from the third vector data is less than the distance threshold. K is an integer greater than 1.

[0219] 8025: Accumulate the distance threshold and the offset value, update the distance threshold to the accumulated value, and return to execute 8022.

[0220] If the total number of vector data in the vector database system that meet the filtering conditions and whose distance to the third vector data is less than the distance threshold is less than K, it indicates that the distance threshold is small, resulting in fewer vector data that are found whose distance to the third vector data is less than the distance threshold. Therefore, it is necessary to increase the distance threshold.

[0221] For example, assuming the distance threshold is 0.2 and the offset value is 0.01, the distance threshold of 0.2 and the offset value of 0.01 are added together, and the distance threshold of 0.2 is updated to the accumulated value of 0.21, that is, the distance threshold becomes 0.21, and then the process returns to execute 8022.

[0222] 8026: Select at least one cluster where the second number is greater than or equal to the number threshold.

[0223] For example, with a threshold of 1, the coordinating node can select at least one cluster where the second number is greater than or equal to 1.

[0224] For the vector count prediction model described above, multiple training samples can be obtained. Each training sample includes the fourth vector data, a distance threshold, the center of each cluster, and the number of vector data in each cluster whose distance to the fourth vector data is less than the distance threshold. Then, the vector count prediction model is trained based on multiple training samples.

[0225] Step 803: Query the K vector data from at least one cluster that satisfy the filtering conditions and have the smallest distance to the third vector data.

[0226] In 803, the coordinating node determines the data node where the at least one cluster is located and sends a query request to the determined data node.

[0227] The identified data nodes include a first data node, and at least one cluster includes a first cluster. The first data node includes the first cluster and a first index graph of the first cluster. The first data node receives a query request, and based on the first index graph, queries the first cluster to find at least one vector data whose metadata satisfies the filtering conditions and has the smallest distance to the third vector data. It then sends the queried at least one vector data to the coordinating node. The coordinating node can receive vector data from different data nodes, and the number of received vector data is greater than or equal to K. It selects K vector data with the smallest distance to the third vector data from the received vector data and sends these K vector data.

[0228] The coordinating node identifies at least one cluster containing vector data whose metadata satisfies the filtering criteria, and among these vector data, there are one or more vector data points closest to the third vector data point. Therefore, querying from this at least one cluster narrows the query scope, improving both query efficiency and accuracy.

[0229] In some embodiments, the process by which the first data node, based on the first index graph, queries at least one vector data from the first cluster whose metadata satisfies the filtering conditions and has the smallest distance to the third vector data can be:

[0230] For the sixth node corresponding to the third vector data, the first data node uses the third vector data as the coordinates of the sixth node in the first index graph and places it in the first index graph. Then, it uses a greedy neighbor search method to find multiple nodes that are the nodes with the smallest distance to the sixth node. The multiple vector data corresponding to these nodes are the multiple vector data in the first cluster that have the smallest distance to the third vector data. From these multiple vector data, at least one vector data whose metadata satisfies the filtering conditions is obtained.

[0231] For a detailed explanation of how the first data node uses a greedy neighbor search method to find multiple nodes, please refer to [link to documentation / reference]. Figure 2 The process of steps 208 in the illustrated embodiment, including steps 2081-2084, will not be described in detail here.

[0232] In this embodiment, upon receiving a query request, the query request includes the third vector data to be queried and filtering conditions. Based on the third vector data and filtering conditions, at least one cluster is determined to be included in the vector database system. The total number of vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is less than a distance threshold is greater than or equal to K in the at least one cluster. This narrows the query scope. Since one or more vector data whose metadata satisfies the filtering conditions are closest to the third vector data in the at least one cluster, querying the K vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is the smallest from the at least one cluster not only improves query efficiency but also improves query accuracy.

[0233] See Figure 10 This application provides an apparatus 1000 for processing vector data, which can be applied to... Figure 1 The vector database system 100 shown is described. The device 1000 includes a first cluster, which stores multiple vector data sets. These multiple vector data sets are stored in the first cluster based on a first center, where the first center is the center of the first cluster. The device 1000 includes:

[0234] Acquisition unit 1001 is used to acquire the second center based on the vector data stored in the first cluster;

[0235] Processing unit 1002 is used to obtain at least one vector data based on the second center when the center of the first cluster changes from the first center to the second center. The at least one vector data is data belonging to the second cluster stored in the first cluster. The device 1000 also includes the second cluster.

[0236] The processing unit 1002 is also configured to migrate at least one vector data from the first cluster to the second cluster.

[0237] Optionally, the detailed implementation process of obtaining the vector data stored by the first cluster in the acquisition unit 1001 to obtain the second center can be found in [reference needed]. Figure 6 The details of step 601 of method 600 shown will not be explained in detail here.

[0238] Optionally, when the processing unit 1002 determines that the center of the first cluster has changed from the first center to the second center based on the second center, the detailed implementation process of obtaining at least one vector data based on the second center can be found in [reference needed]. Figure 6 The details of steps 602-603 of method 600 shown will not be explained in detail here.

[0239] Optionally, the detailed implementation process of processing unit 1002 transferring at least one vector data from the first cluster to the second cluster can be found in [reference needed]. Figure 6 The details of step 604 of method 600 shown will not be explained in detail here.

[0240] Optionally, for each vector data in at least one vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center.

[0241] Optionally, the acquisition unit 1001 is used for:

[0242] Sample x vector data from the vector data stored in the first cluster, where x is an integer greater than 1 and less than the total number of vector data included in the first cluster;

[0243] The second center is obtained by clustering based on x vector data.

[0244] Optionally, the detailed implementation process of the processing unit 1002 sampling x vector data from the vector data stored in the first cluster can be found in [reference needed]. Figure 6 The details of step 601 of method 600 shown will not be explained in detail here.

[0245] Optionally, the detailed implementation process of the processing unit 1002 clustering based on x vector data to obtain the second center can be found in [reference needed]. Figure 6 The details of step 601 of method 600 shown will not be explained in detail here.

[0246] Optionally, the processing unit 1002 is used for:

[0247] Based on the second center, y vector data are obtained from x vector data, where y is a positive integer less than or equal to x. For each vector data in the y vector data, the distance between the vector data and the second center is less than the distance between the vector data and the first center.

[0248] When the ratio between y and x exceeds the ratio threshold, the center of the first cluster is determined to change from the first center to the second center.

[0249] Optionally, the processing unit 1002, based on the second center, can obtain detailed implementation details of acquiring y vector data from x vector data. (See [link to relevant documentation]). Figure 6 The details of step 602 of method 600 shown will not be explained in detail here.

[0250] Optionally, the detailed implementation process of the processing unit 1002 determining that the center of the first cluster changes from the first center to the second center can be found in [reference needed]. Figure 6 The details of step 602 of method 600 shown will not be explained in detail here.

[0251] Optionally, the device 1000 further includes a first index graph of the first cluster, the first index graph including the nodes corresponding to each vector data in the first cluster, and for any two nodes in the first index graph that are connected, the distance between the two nodes satisfies the distance condition.

[0252] The device 1000 further includes: a first receiving unit 1003;

[0253] The first receiving unit 1003 is used to receive the first vector data to be stored;

[0254] The processing unit 1002 is also used to save the first vector data into the first cluster based on the second center;

[0255] The processing unit 1002 is further configured to obtain from the first index graph at least one second node whose distance to the first node corresponding to the first vector data satisfies the distance condition;

[0256] The processing unit 1002 is also configured to add the first node to the first index graph and connect the first node with at least one second node.

[0257] Optionally, the detailed implementation process of the first receiving unit 1003 receiving the first vector data to be stored can be found in [reference needed]. Figure 2 The details of step 205 of method 200 shown will not be explained in detail here.

[0258] Optionally, the detailed implementation process of processing unit 1002 saving the first vector data into the first cluster based on the second center can be found in [reference needed]. Figure 2 The details of step 207 of method 200 shown will not be explained in detail here.

[0259] Optionally, the detailed implementation process of the processing unit 1002 obtaining at least one second node from the first index graph whose distance to the first node corresponding to the first vector data satisfies the distance condition can be found in [reference needed]. Figure 2 The details of step 208 of method 200 shown will not be explained in detail here.

[0260] Optionally, the detailed implementation process of the processing unit 1002 adding the first node to the first index graph and connecting the first node with at least one second node can be found in [reference needed]. Figure 2 The details of step 209 of method 200 shown will not be explained in detail here.

[0261] Optionally, the processing unit 1002 is used for:

[0262] Obtain the multiple nodes with the smallest distance from the first node from the first index graph;

[0263] Select at least one second node from multiple nodes. For each second node, the distance between the second node and the first node is less than the distance between any two second nodes in the at least one second node, or the distance between the second node and the first node is less than the product of the distance between the two second nodes and the target coefficient, where the target coefficient is greater than 1.

[0264] Optionally, the detailed implementation process of processing unit 1002 obtaining multiple nodes with the smallest distance from the first node from the first index graph can be found in [reference needed]. Figure 2 The details of step 208 of method 200 shown will not be explained in detail here.

[0265] Optionally, the detailed implementation process of the processing unit 1002 selecting at least one second node from multiple nodes can be found in [reference needed]. Figure 2 The details of step 208 of method 200 shown will not be explained in detail here.

[0266] Optionally, the device 1000 further includes a correspondence between the identifier of each vector data in the first cluster and the position of the node corresponding to each vector data. The processing unit 1002 is further configured to:

[0267] Get the identifier of the second vector data to be deleted;

[0268] Based on the identifier of the second vector data, the second vector data is deleted from the first cluster, and the position of the third node corresponding to the second vector data is obtained from the correspondence.

[0269] The third node is deleted from the first index graph based on its position to obtain the second index graph.

[0270] Optionally, the detailed implementation process of the processing unit 1002 obtaining the identifier of the second vector data to be deleted can be found in [reference needed]. Figure 7 The details of step 701 of method 700 shown will not be explained in detail here.

[0271] Optionally, the detailed implementation process of the processing unit 1002 deleting the second vector data from the first cluster based on the identifier of the second vector data, and obtaining the position of the third node corresponding to the second vector data from the correspondence, can be found in [reference needed]. Figure 7 The details of step 702 of method 700 shown will not be explained in detail here.

[0272] Optionally, the processing unit 1002 deletes the third node from the first index graph based on the position of the third node to obtain the second index graph. For a detailed implementation process, please refer to [link to relevant documentation]. Figure 7 The details of step 703 of method 700 shown will not be explained in detail here.

[0273] Optionally, the processing unit 1002 is further configured to:

[0274] Obtain the fourth node, which is a node in the first index graph that is connected to the third node;

[0275] From the second index graph, obtain at least one fifth node whose distance from the fourth node satisfies the distance condition;

[0276] Connect the fourth node to at least one fifth node in the second index graph.

[0277] Optionally, the detailed implementation process of the processing unit 1002 acquiring the fourth node can be found in [reference needed]. Figure 7 The details of step 704 of method 700 shown will not be explained in detail here.

[0278] Optionally, the detailed implementation process of the processing unit 1002 obtaining at least one fifth node from the second index graph whose distance to the fourth node satisfies the distance condition can be found in [reference needed]. Figure 7 The details of step 705 of method 700 shown will not be explained in detail here.

[0279] Optionally, the detailed implementation process of the processing unit 1002 connecting the fourth node and at least one fifth node in the second index graph can be found in [reference needed]. Figure 7 The details of step 706 of method 700 shown will not be explained in detail here.

[0280] Optionally, the first clustering is also used to store metadata for each of the multiple vector data, and the device 1000 further includes: a second receiving unit 1004;

[0281] The second receiving unit 1004 is used to receive a query request, which includes the third vector data to be queried and the filtering conditions.

[0282] The processing unit 1002 is further configured to determine, based on the third vector data and filtering conditions, at least one cluster included in the device 1000, wherein the total number of vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is less than a distance threshold is greater than or equal to K, where K is an integer greater than 1.

[0283] The processing unit 1002 is also configured to query K vector data from at least one cluster that satisfy the filtering conditions and have the smallest distance to the third vector data.

[0284] Optionally, the detailed implementation process of the second receiving unit 1004 receiving the query request can be found in [reference needed]. Figure 8 The details of step 801 of method 800 shown will not be explained in detail here.

[0285] Optionally, the detailed implementation process of the processing unit 1002 determining at least one cluster included in the device 1000 based on the third vector data and filtering conditions can be found in [reference needed]. Figure 8 The details of step 802 of method 800 shown will not be explained in detail here.

[0286] Optionally, the detailed implementation process of the processing unit 1002 querying the K vector data from at least one cluster that satisfy the filtering conditions and have the smallest distance to the third vector data can be found in [reference needed]. Figure 8 The details of step 803 of method 800 shown will not be explained in detail here.

[0287] Optionally, the processing unit 1002 is used for:

[0288] The proportion corresponding to each cluster included in the device 1000 is obtained. The proportion corresponding to each cluster is the ratio between the number of vector data whose metadata in the cluster satisfies the filtering conditions and the total number of vector data in the cluster.

[0289] Based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster, the first number corresponding to each cluster is obtained. The first number corresponding to each cluster is the number of vector data included in the cluster whose distance to the third vector data is less than the distance threshold.

[0290] Based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster is obtained. The second number corresponding to each cluster is the number of data vectors whose metadata included in the cluster meets the filtering conditions and whose distance to the third vector data is less than the distance threshold.

[0291] If the cumulative value of the second number corresponding to each cluster is greater than or equal to K, select at least one cluster where the second number is greater than or equal to the number threshold.

[0292] Optionally, the detailed implementation process of the processing unit 1002 obtaining the proportion corresponding to each cluster included in the device 1000 can be found in [reference needed]. Figure 8 The details of step 8021 of method 800 shown will not be explained in detail here.

[0293] Optionally, the processing unit 1002 obtains the first number corresponding to each cluster based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster. For a detailed implementation process, please refer to [link to relevant documentation]. Figure 8 The details of step 8022 of method 800 shown will not be explained in detail here.

[0294] Optionally, the processing unit 1002 obtains the second number for each cluster based on the first number for each cluster and the proportion for each cluster. For a detailed implementation process, please refer to [link to relevant documentation]. Figure 8 The details of step 8023 of method 800 shown will not be explained in detail here.

[0295] Optionally, the detailed implementation process of the processing unit 1002 selecting at least one cluster where the second number is greater than or equal to the number threshold can be found in [reference missing]. Figure 8 The details of step 8026 of method 800 shown will not be explained in detail here.

[0296] Optionally, the processing unit 1002 is further configured to:

[0297] The model predicts the number of vectors based on multiple training samples. The training samples include the fourth vector data, the distance threshold, the center of each cluster, and the number of vector data in each cluster whose distance to the fourth vector data is less than the distance threshold.

[0298] In this embodiment, the acquisition unit acquires the second center based on the vector data stored in the first cluster, and the processing unit determines whether the center of the first cluster has changed from the first center to the second center based on the second center. If the center of the first cluster changes from the first center to the second center, based on the second center, the vector data that does not belong to the first cluster can be identified, and this vector data is migrated to the second cluster to which it belongs. This ensures that the vector data belonging to the first cluster remains in the first cluster, and the first cluster does not include a large amount of vector data that does not belong to it. Therefore, when querying data from the vector database system, the accuracy of the queried data can be improved.

[0299] See Figure 11 This application provides a computing device 1100. For example, the computing device 1100 may be... Figure 1 The vector database system 100 shown includes nodes, or the computing device 1100 can be... Figure 2 Method 200 shown Figure 6 Method 600 shown Figure 7 Method 700 or shown Figure 8 The nodes in the vector database system shown in method 800.

[0300] like Figure 11As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0301] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (e.g., processor 1104, memory 1106, communication interface 1108).

[0302] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0303] The memory 1106 may include volatile memory, such as random access memory (RAM). The memory 1106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0304] See Figure 11 The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the following respectively. Figure 10The functions of the acquisition unit 1001, processing unit 1002, first receiving unit 1003, and second receiving unit 1004 in the illustrated device 1000 are used to implement the method provided in any of the above embodiments. That is, the memory 1106 stores instructions for executing the method provided in any of the above embodiments. Alternatively,

[0305] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0306] This application also provides a cluster for processing vector data. The cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0307] like Figure 12 As shown, the cluster for processing vector data includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the cluster may store the same instructions for performing the methods provided in any of the above embodiments.

[0308] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the cluster processing vector data may also store partial instructions for executing the methods for processing vector data described above. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for performing the methods provided in any of the above embodiments.

[0309] In some possible implementations, one or more computing devices in the cluster processing vector data can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 13 One possible implementation is shown. For example... Figure 13 As shown, the two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0310] In this type of possible implementation, the memory 1106 in the computing device 1100A stores the execution of, for example Figure 10 The instructions for the acquisition unit 1001 and processing unit 1002 in the illustrated embodiment are shown. Meanwhile, the memory 1106 in the computing device 1100B stores instructions for performing operations such as... Figure 10 Instructions for the functions of the first receiving unit 1003 and the second receiving unit 1004 in the illustrated embodiment.

[0311] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0312] This application also provides another cluster for processing vector data. The connection relationships between the computing devices in this vector data processing cluster can be similarly referenced. Figure 13 The connection method of the cluster for processing vector data differs in that the memory 1106 of one or more computing devices 1100 in the cluster for processing vector data can store the same instructions for executing the methods provided in any of the above embodiments.

[0313] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the cluster processing vector data may also store partial instructions for executing the methods provided in any of the above embodiments. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing the methods provided in any of the above embodiments.

[0314] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the methods provided in any of the above embodiments.

[0315] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in any of the above embodiments.

[0316] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0317] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing vector data, characterized in that, The method is applied to a vector database system, which includes a first cluster for storing multiple vector data sets. These multiple vector data sets are stored in the first cluster based on a first center, where the first center is the center of the first cluster. The method includes: Based on the vector data saved by the first cluster, obtain the second center; When the center of the first cluster changes from the first center to the second center based on the second center, at least one vector data is obtained based on the second center. The at least one vector data is data belonging to the second cluster that is stored in the first cluster. The vector database system also includes the second cluster. The at least one vector data is migrated from the first cluster to the second cluster.

2. The method as described in claim 1, characterized in that, For each of the at least one vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center.

3. The method as described in claim 1 or 2, characterized in that, The step of obtaining the second center based on the vector data stored in the first cluster includes: Sample x vector data from the vector data stored in the first cluster, where x is an integer greater than 1 and less than the total number of vector data included in the first cluster; The second center is obtained by clustering based on the x vector data.

4. The method as described in claim 3, characterized in that, The step of determining that the center of the first cluster changes from the first center to the second center based on the second center includes: Based on the second center, y vector data are obtained from the x vector data, where y is a positive integer less than or equal to x. For each of the y vector data, the distance between the vector data and the second center is less than the distance between the vector data and the first center. When the ratio between y and x exceeds a ratio threshold, the center of the first cluster is determined to change from the first center to the second center.

5. The method according to any one of claims 1-4, characterized in that, The vector database system further includes a first index graph of the first cluster, the first index graph includes nodes corresponding to each vector data in the first cluster, and the distance between any two nodes with a connection relationship in the first index graph satisfies a distance condition. After determining that the center of the first cluster changes from the first center to the second center based on the second center, the method further includes: Receive the first vector data to be stored; The first vector data is saved into the first cluster based on the second center; Obtain at least one second node from the first index graph whose distance to the first node corresponding to the first vector data satisfies the distance condition; Add the first node to the first index graph and connect the first node to the at least one second node.

6. The method as described in claim 5, characterized in that, The step of obtaining at least one second node from the first index graph whose distance to the first node corresponding to the first vector data satisfies the distance condition includes: Obtain the multiple nodes with the smallest distance from the first node from the first index graph; At least one second node is selected from the plurality of nodes. For each second node, the distance between the second node and the first node is less than the distance between any two of the at least one second node, or the distance between the second node and the first node is less than the product of the distance between the two second nodes and the target coefficient, wherein the target coefficient is greater than 1.

7. The method as described in claim 5 or 6, characterized in that, The vector database system further includes the correspondence between the identifier of each vector data in the first cluster and the position of the node corresponding to each vector data. The method further includes: Get the identifier of the second vector data to be deleted; Based on the identifier of the second vector data, the second vector data is deleted from the first cluster, and the position of the third node corresponding to the second vector data is obtained from the correspondence. Based on the position of the third node, the third node is deleted from the first index graph to obtain the second index graph.

8. The method as described in claim 7, characterized in that, After deleting the third node from the first index graph based on its position, the method further includes: Obtain the fourth node, which is a node in the first index graph that has a connection relationship with the third node; Obtain at least one fifth node from the second index graph whose distance from the fourth node satisfies the distance condition; Connect the fourth node to the at least one fifth node in the second index graph.

9. The method according to any one of claims 1-8, characterized in that, The first clustering is also used to store metadata for each of the plurality of vector data, and the method further includes: Receive a query request, which includes the third vector data to be queried and filtering conditions; Based on the third vector data and the filtering conditions, at least one cluster is determined in the vector database system. The at least one cluster includes a total number of vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is less than a distance threshold, which is greater than or equal to K, where K is an integer greater than 1. From the at least one cluster, query the K vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is the smallest.

10. The method as described in claim 9, characterized in that, The step of determining at least one cluster included in the vector database system based on the third vector data and the filtering conditions includes: Obtain the proportion corresponding to each cluster included in the vector database system. The proportion corresponding to each cluster is the ratio between the number of vector data whose metadata in the cluster satisfies the filtering conditions and the total number of vector data included in the cluster. Based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster, the first number corresponding to each cluster is obtained. The first number corresponding to each cluster is the number of vector data included in the cluster whose distance to the third vector data is less than the distance threshold. Based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster is obtained. The second number corresponding to each cluster is the number of vector data whose metadata included in the cluster satisfies the filtering conditions and whose distance to the third vector data is less than the distance threshold. If the sum of the second number corresponding to each cluster is greater than or equal to K, then at least one cluster whose second number is greater than or equal to the number threshold is selected.

11. The method as described in claim 10, characterized in that, The method further includes: The vector count prediction model is trained based on multiple training samples, wherein the training samples include a fourth vector data, the distance threshold, the center of each cluster, and the number of vector data in each cluster whose distance to the fourth vector data is less than the distance threshold.

12. An apparatus for processing vector data, characterized in that, The apparatus includes a first cluster for storing multiple vector data, the multiple vector data being stored in the first cluster based on a first center, the first center being the center of the first cluster. The apparatus includes: The acquisition unit is used to acquire the second center based on the vector data stored in the first cluster; The processing unit is configured to, when determining that the center of the first cluster changes from the first center to the second center based on the second center, acquire at least one vector data based on the second center, wherein the at least one vector data is data belonging to the second cluster stored in the first cluster, and the device further includes the second cluster; The processing unit is further configured to migrate the at least one vector data from the first cluster to the second cluster.

13. The apparatus as claimed in claim 12, characterized in that, For each of the at least one vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center.

14. The apparatus as claimed in claim 12 or 13, characterized in that, The acquisition unit is used for: Sample x vector data from the vector data stored in the first cluster, where x is an integer greater than 1 and less than the total number of vector data included in the first cluster; The second center is obtained by clustering based on the x vector data.

15. The apparatus as claimed in claim 14, characterized in that, The processing unit is used for: Based on the second center, y vector data are obtained from the x vector data, where y is a positive integer less than or equal to x. For each of the y vector data, the distance between the vector data and the second center is less than the distance between the vector data and the first center. When the ratio between y and x exceeds a ratio threshold, the center of the first cluster is determined to change from the first center to the second center.

16. The apparatus according to any one of claims 12-15, characterized in that, The device further includes a first index graph of the first cluster, the first index graph including nodes corresponding to each vector data in the first cluster, and for any two nodes in the first index graph that are connected, the distance between the two nodes satisfies a distance condition. The device further includes: a first receiving unit; The first receiving unit is used to receive the first vector data to be stored; The processing unit is further configured to save the first vector data into the first cluster based on the second center; The processing unit is further configured to obtain from the first index graph at least one second node whose distance to the first node corresponding to the first vector data satisfies the distance condition; The processing unit is further configured to add the first node to the first index graph and connect the first node with the at least one second node.

17. The apparatus as claimed in claim 16, characterized in that, The processing unit is used for: Obtain the multiple nodes with the smallest distance from the first node from the first index graph; At least one second node is selected from the plurality of nodes. For each second node, the distance between the second node and the first node is less than the distance between any two of the at least one second node, or the distance between the second node and the first node is less than the product of the distance between the two second nodes and the target coefficient, wherein the target coefficient is greater than 1.

18. The apparatus as claimed in claim 16 or 17, characterized in that, The device further includes a correspondence between the identifier of each vector data in the first cluster and the position of the node corresponding to each vector data. The processing unit is further configured to: Get the identifier of the second vector data to be deleted; Based on the identifier of the second vector data, the second vector data is deleted from the first cluster, and the position of the third node corresponding to the second vector data is obtained from the correspondence. Based on the position of the third node, the third node is deleted from the first index graph to obtain the second index graph.

19. The apparatus as claimed in claim 18, characterized in that, The processing unit is further configured to: Obtain the fourth node, which is a node in the first index graph that has a connection relationship with the third node; Obtain at least one fifth node from the second index graph whose distance from the fourth node satisfies the distance condition; Connect the fourth node to the at least one fifth node in the second index graph.

20. The apparatus according to any one of claims 12-19, characterized in that, The first clustering is also used to store metadata for each of the plurality of vector data, and the device further includes: a second receiving unit; The second receiving unit is used to receive a query request, the query request including the third vector data to be queried and filtering conditions; The processing unit is further configured to determine at least one cluster included in the device based on the third vector data and the filtering conditions, wherein the at least one cluster includes a total number of vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is less than a distance threshold, which is greater than or equal to K, where K is an integer greater than 1. The processing unit is further configured to query from the at least one cluster the K vector data whose metadata satisfies the filtering conditions and whose distance to the third vector data is the smallest.

21. The apparatus as claimed in claim 20, characterized in that, The processing unit is used for: Obtain the proportion corresponding to each cluster included in the device, wherein the proportion corresponding to the cluster is the ratio between the number of vector data whose metadata in the cluster satisfies the filtering conditions and the total number of vector data included in the cluster; Based on the vector count prediction model, the third vector data, the distance threshold, and the center of each cluster, the first number corresponding to each cluster is obtained. The first number corresponding to each cluster is the number of vector data included in the cluster whose distance to the third vector data is less than the distance threshold. Based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster is obtained. The second number corresponding to each cluster is the number of vector data whose metadata included in the cluster satisfies the filtering conditions and whose distance to the third vector data is less than the distance threshold. If the sum of the second number corresponding to each cluster is greater than or equal to K, then at least one cluster whose second number is greater than or equal to the number threshold is selected.

22. The apparatus as claimed in claim 21, characterized in that, The processing unit is further configured to: The vector count prediction model is trained based on multiple training samples, wherein the training samples include a fourth vector data, the distance threshold, the center of each cluster, and the number of vector data in each cluster whose distance to the fourth vector data is less than the distance threshold.

23. A vector database system, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the vector database system to perform the method as described in any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a vector database system, perform the method as described in any one of claims 1-11.

25. A computer program product containing instructions, characterized in that, When the instruction is executed by the vector database system, it causes the vector database system to perform the method as described in any one of claims 1-11.

Images