Method, apparatus and system for processing vector data, and storage medium

Abstract

The present application belongs to the field of storage. Disclosed are a method, apparatus and system for processing vector data, and a storage medium. The method is applied to a vector database system, wherein the vector database system comprises a first cluster, the first cluster is used for storing a plurality of pieces of vector data, the plurality of pieces of vector data are pieces of data that are stored in the first cluster on the basis of a first center, and the first center is the center of the first cluster. The method comprises: acquiring a second center on the basis of vector data stored in a first cluster; when it is determined on the basis of the second center that the center of the first cluster changes from a first center to the second center, acquiring at least one piece of vector data on the basis of the second center, wherein the at least one piece of vector data is data that is stored in the first cluster and belongs to a second cluster, and a vector database system further comprises the second cluster; and migrating the at least one piece of vector data from the first cluster to the second cluster. The present application can improve the accuracy of vector data queries.

Description

Method, device, system and storage medium for processing vector data

[0001] The present application claims priority to the Chinese patent application No. 202411002385.5, filed on July 24, 2024, and entitled "Method, device, system and storage medium for processing vector data", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of storage, and in particular to a method, device, system and storage medium for processing vector data. BACKGROUND

[0003] A vector database is a database for storing vector data, which is usually data obtained by vectorizing text, speech, images or video content. The vector database includes multiple clusters, each of which is used to store different vector data.

[0004] Upon receiving vector data that needs to be stored in the vector database, the distances between the vector data and multiple centers corresponding to the multiple clusters can be calculated. The center with the smallest distance to the vector data is selected from the multiple centers, and the vector data is stored in the cluster corresponding to the center.

[0005] Over time, the distribution of vector data stored in the vector database changes constantly, which causes any cluster in the vector database to include a large amount of vector data that does not belong to the cluster, thus reducing the query accuracy when querying data from the vector database. SUMMARY

[0006] The present application provides a method, device, system and storage medium for processing vector data to improve the accuracy of querying vector data. The technical solution is as follows:

[0007] In a first aspect, the present application provides a method for processing vector data, which is applied to a vector database system. The vector database system includes a first cluster, which is used to store a plurality of vector data. The plurality of vector data is data stored in the first cluster based on a first center. The first center is the center of the first cluster. In the method, a second center is obtained based on the vector data stored in the first cluster. At least one vector data is obtained based on the second center when the center of the first cluster changes from the first center to the second center. The at least one vector data is data stored in the first cluster and belonging to a second cluster. The vector database system further includes the second cluster. The at least one vector data is migrated from the first cluster to the second cluster.

[0008] Based on the vector data saved in the first cluster, a second center is obtained, and the second center is the latest center of the first cluster. When the center of the first cluster is determined to change from the first center to the second center based on the second center, it indicates that there is a large amount of vector data in the first cluster that does not belong to the first cluster. Based on the second center, vector data in the first cluster that belongs to the second cluster is found, and the found vector data is migrated from the first cluster to the second cluster. In this way, the vector data left in the first cluster is data that belongs to the first cluster, thereby improving the accuracy of querying vector data from the vector database system.

[0009] In a possible implementation, for each vector data in the at least one vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center. Therefore, the at least one vector data is closer to the center of the second cluster, and is data that belongs to the second cluster. Migrating the at least one vector data ensures that the vector data left in the first cluster is data that belongs to the first cluster.

[0010] In another possible implementation, x vector data is sampled from the vector data saved in the first cluster, and x is an integer greater than 1 and less than the total number of vector data included in the first cluster. The second center is obtained based on the clustering of the x vector data. Clustering the x vector data can reduce the amount of data for clustering and improve the efficiency of clustering.

[0011] In another possible implementation, y vector data is obtained from the x vector data based on the second center, y is a positive integer less than or equal to x, and for each vector data in the y vector data, the distance between the vector data and the second center is less than the distance between the vector data and the first center. When the ratio between y and x exceeds a ratio threshold, it is determined that the center of the first cluster changes from the first center to the second center.

[0012] When the ratio between y and x exceeds the ratio threshold, it indicates that most of the vector data in the first cluster is closer to the second center, so it is possible to accurately determine that the center of the first cluster changes from the first center to the second center, that is, the actual center of the first cluster is the second center.

[0013] In another possible implementation, the vector database system further includes a first index graph of the first cluster, the first index graph including a node corresponding to each vector data in the first cluster, and for any two nodes having a connection relationship in the first index graph, a distance between the two nodes satisfies a distance condition. The first vector data to be stored is received. The first vector data is saved into the first cluster based on the second center. At least one second node satisfying the distance condition with a first node corresponding to the first vector data is obtained from the first index graph. The first node is added to the first index graph, and the first node is connected with the at least one second node.

[0014] Since the at least one second node satisfying the distance condition with the first node is obtained, and the first node and the at least one second node are connected, the whole index graph does not need to be reconstructed, the operation amount of updating the index graph is reduced, and the efficiency is improved.

[0015] In another possible implementation, a plurality of nodes having a minimum distance with the first node are obtained from the first index graph. At least one second node is selected from the plurality of nodes, and for each second node, a distance between the second node and the first node is smaller than a distance between any two second nodes in the at least one second node, or the distance between the second node and the first node is smaller than a product between a distance between the two second nodes and a target coefficient, and the target coefficient is greater than 1.

[0016] Since the distance between the second node and the first node is smaller than the distance between any two second nodes in the at least one second node, or the distance between the second node and the first node is smaller than the product between the distance between the two second nodes and the target coefficient, the at least one second node is closer to the first node, and the first node is connected with the at least one second node, so that the connectivity of the index graph can be improved.

[0017] In another possible implementation, the vector database system further includes a corresponding relationship between an identifier of each vector data in the first cluster and a position of a node corresponding to the vector data. The identifier of the second vector data to be deleted is obtained. The second vector data is deleted from the first cluster based on the identifier of the second vector data, and a position of a third node corresponding to the second vector data is obtained from the corresponding relationship. The third node is deleted from the first index graph based on the position of the third node, and a second index graph is obtained. In this way, the third node can be quickly found from the first index graph through the corresponding relationship, and the efficiency of deleting the third node is improved.

[0018] In another possible implementation, a fourth node is acquired, the fourth node being a node in the first index graph that has a connection relationship with the third node. At least one fifth node is acquired from the second index graph, the at least one fifth node satisfying a distance condition with the fourth node. The fourth node is connected with the at least one fifth node in the second index graph. Since the third node is deleted, one neighbor node of the fourth node is vacated, and the connectivity of the index graph is reduced. Therefore, the at least one fifth node satisfying the distance condition with the fourth node is acquired, and the fourth node is connected with the at least one fifth node in the second index graph, so as to improve the connectivity of the second index graph.

[0019] In another possible implementation, the first cluster is further configured to store metadata of each vector data in the plurality of vector data. A query request is received, the query request including third vector data to be queried and a filter condition. Based on the third vector data and the filter condition, at least one cluster included in the vector database system is determined, the at least one cluster including a total number of vector data whose metadata satisfies the filter condition and whose distance with the third vector data is less than a distance threshold being greater than or equal to K, K being an integer greater than 1. From the at least one cluster, K vector data whose metadata satisfies the filter condition and whose distance with the third vector data is smallest are queried.

[0020] Since the query range is determined first, that is, at least one cluster is determined, and the required vector data is queried from the at least one cluster, compared with querying from all clusters, the query efficiency is improved.

[0021] In another possible implementation, a proportion corresponding to each cluster of the vector database system is acquired, the proportion corresponding to the cluster being a ratio between a number of vector data whose metadata satisfies a filter condition and included in the cluster and a total number of vector data included in the cluster. Based on a vector number prediction model, third vector data, a distance threshold, and a center of each cluster, a first number corresponding to each cluster is acquired, the first number corresponding to the cluster being a number of vector data included in the cluster and having a distance less than the distance threshold with the third vector data. Based on the first number corresponding to each cluster and the proportion corresponding to each cluster, a second number corresponding to each cluster is obtained, the second number corresponding to the cluster being a number of vector data included in the cluster and having a distance less than the distance threshold with the third vector data and whose metadata satisfies the filter condition. In a case where an accumulated value of the second number corresponding to each cluster is greater than or equal to K, the at least one cluster whose second number is greater than or equal to a number threshold is selected. In this way, the second number corresponding to each cluster can be obtained through the vector number prediction model, and the application range and flexibility are expanded.

[0022] In another possible implementation, the vector number prediction model is trained based on a plurality of training samples, the training samples including fourth vector data, a distance threshold, a center of each cluster, and a number of vector data in each cluster having a distance less than the distance threshold from the fourth vector data.

[0023] In a second aspect, the present application provides an apparatus for processing vector data, configured to perform the method in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus includes units for performing the method in the first aspect or any possible implementation of the first aspect.

[0024] In a third aspect, the present application provides a vector database system, including at least one computing device, each computing device including a processor and a memory;

[0025] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the vector database system performs the method in the first aspect or any possible implementation of the first aspect.

[0026] In one possible implementation, the vector database system is a cluster of computing devices.

[0027] In a fourth aspect, the present application provides a computer program product including instructions, which, when executed by a storage system, cause the storage system to perform the method in the first aspect or any possible implementation of the first aspect.

[0028] In a fifth aspect, the present application provides a computer-readable storage medium including computer program instructions, which, when executed by a storage system, cause the storage system to perform the method in the first aspect or any possible implementation of the first aspect.

[0029] In a sixth aspect, the present application provides a chip including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to call and run the computer instructions from the memory to perform the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0030] FIG. 1 is a structural schematic diagram of a vector database system according to an embodiment of the present application;

[0031] FIG. 2 is a flowchart of a method for saving vector data according to an embodiment of the present application;

[0032] FIG. 3 is a schematic diagram of an index graph according to an embodiment of the present application;

[0033] FIG. 4 is a schematic diagram of another index graph provided by an embodiment of the present application;

[0034] FIG. 5 is a schematic diagram of another index graph provided by an embodiment of the present application;

[0035] FIG. 6 is a flowchart of a method for processing vector data provided by an embodiment of the present application;

[0036] FIG. 7 is a flowchart of a method for deleting vector data provided by an embodiment of the present application;

[0037] FIG. 8 is a flowchart of a method for querying vector data provided by an embodiment of the present application;

[0038] FIG. 9 is a schematic diagram of a histogram provided by an embodiment of the present application;

[0039] FIG. 10 is a schematic diagram of an apparatus structure for processing vector data provided by an embodiment of the present application;

[0040] FIG. 11 is a schematic diagram of a computing device structure provided by an embodiment of the present application;

[0041] FIG. 12 is a schematic diagram of a cluster structure for processing vector data provided by an embodiment of the present application;

[0042] FIG. 13 is a schematic diagram of another cluster structure for processing vector data provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] Referring to FIG. 1, an embodiment of the present application provides a vector database system 100, which includes a coordination node 101 and a plurality of data nodes 102. The coordination node 101 can communicate with each data node 102.

[0044] The coordination node 101 can initialize the number of clusters included in the vector database system 100 as M, where M is an integer greater than 1. That is, the vector database system 100 can include M clusters, and the M clusters are distributed on the plurality of data nodes 102. The number of the plurality of data nodes 102 is less than or equal to M, so each data node 102 includes one or more clusters.

[0045] Initially, the coordination node 101 is configured to, after receiving vector data to be stored, randomly save the vector data to one cluster included in the vector database system 100. In implementation,

[0046] The coordination node 101 is configured to receive vector data to be stored, randomly select one cluster from the M clusters, and send the vector data to the data node 102 where the selected cluster is located.

[0047] The data node 102 is configured to receive the vector data and save the vector data into the selected cluster.

[0048] Optionally, if the data node 102 where the selected cluster is located includes multiple clusters, the coordination node 101 is further configured to send an identification of the selected cluster to the data node 102. The data node 102 is further configured to receive the identification of the selected cluster and save the vector data into the selected cluster based on the identification of the selected cluster.

[0049] The coordination node 101 can continuously receive the vector data to be stored and randomly save the vector data into one of the clusters included in the vector database system 100.

[0050] Optionally, the vector data is a vector including multiple dimension data. For example, the vector data is a vector including three dimension data, and the vector is [1, 2, 3]. For another example, the vector data is a vector including four dimension data, and the vector is [11, 12, 43, 7].

[0051] When the number of the vector data stored in the vector database system 100 exceeds the first threshold, the coordination node 101 is further configured to obtain the centers of the M clusters based on the vector data stored in the M clusters, i.e., to obtain M centers in total.

[0052] Then, when the coordination node 101 receives the vector data to be stored, the coordination node 101 is further configured to obtain the distance between the vector data and the M centers, select the center with the smallest distance to the vector data, determine the cluster corresponding to the selected center, and send the vector data to the data node 102 where the cluster is located. The data node 102 is further configured to receive the vector data and save the vector data into the cluster.

[0053] In the related art, after the coordination node 101 obtains the M centers for the first time, the coordination node 101 will always save the vector data received each time and needed to be stored into one of the clusters in the vector database system 100 based on the M centers.

[0054] However, as time goes by, the distribution of the vector data needed to be stored in the vector database system 100 is constantly changing, which will cause the centers of all or part of the M clusters included in the vector database system 100 to change. The change of the center of a cluster can also be referred to as the drift of the center of the cluster. Thus, for the cluster with the drifted center, the cluster can store a large amount of vector data not belonging to itself, which will reduce the query accuracy when querying the vector data from the vector database system 100.

[0055] Next, the process of storing vector data to the vector database system 100, and the process of finding a cluster whose center changes and migrating out vector data not belonging to the cluster, querying vector data from the vector database 100 and deleting vector data will be described in detail through the following embodiments.

[0056] Referring to FIG. 2, the embodiment of the present application provides a method 200 of saving vector data, which is applied to the vector database system 100 shown in FIG. 1. The method 200 is used to receive target vector data to be stored, and save the vector data to a cluster in the vector database system 100. The method 200 includes the following processes.

[0057] In step 201, the coordination node initializes M clusters included in the vector database system, the M clusters being located on a plurality of data nodes included in the vector database system, M being an integer greater than 1.

[0058] In step 201, the coordination node initializes the number of clusters included in the vector database system to be M, and determines the data node where each of the M clusters is located.

[0059] Optionally, if the number of data nodes included in the vector database system is M, each data node includes one cluster. If the number of data nodes included in the vector database system is less than M, some data nodes can include one cluster, and some data nodes can include multiple clusters.

[0060] In step 202, the coordination node receives target vector data to be stored, randomly selects a cluster, and sends the target vector data to the data node where the cluster is located.

[0061] In some embodiments, if the data node where the cluster is located includes multiple clusters, the coordination node further sends the identification of the cluster to the data node.

[0062] In step 203, the data node receives the target vector data, and saves the target vector data to the cluster.

[0063] In step 203, if the data node includes one cluster, the data node receives the target vector data, and saves the target vector data to the cluster. If the data node includes multiple clusters, the data node further receives the identification of the cluster, and saves the target vector data to the cluster corresponding to the identification of the cluster.

[0064] In some embodiments, the data node can further acquire the metadata of the target vector data, and save the metadata of the target vector data to the cluster.

[0065] Optionally, the metadata of the target vector data can include one or more of an identification of the target vector data, a type of the target vector data, a timestamp of the target vector data, and the like.

[0066] Optionally, the timestamp of the target vector data can be a timestamp of creation of the target vector data, a timestamp of saving of the target vector data, and the like.

[0067] Optionally, the type of the target vector data can be a provenance of the target vector data, a user to which the target vector data belongs, and the like.

[0068] In some embodiments, the cluster can be a data table into which the data node saves the target vector data, and the identification of the target vector data can be a row number of a row in which the target vector data is located. Alternatively, the cluster can be a text file, such as an extensible markup language (XML) file, and the like, into which the data node saves the target vector data, and the identification of the target vector data can be a serial number of the target vector data.

[0069] The coordination node and the data nodes in the vector database system can repeatedly perform the operations of steps 201-203, so as to randomly save the target vector data into one of the clusters in the vector database system again when the target vector data that needs to be saved is received again.

[0070] When the number of the vector data saved in the vector database system exceeds the first number threshold, the following process can be performed.

[0071] Step 204: The coordination node obtains M centers based on the vector data saved in the M clusters, the M centers being centers of the M clusters.

[0072] In some embodiments, for each cluster and for the data node in which the cluster is located, the data node can sample z vector data from the vector data saved in the cluster, z being an integer greater than 1, and send the z vector data to the coordination node. Optionally, the data node can randomly sample z vector data from the vector data saved in the cluster.

[0073] In some embodiments, for each cluster and for the data node in which the cluster is located, the data node can sample z vector data from the vector data saved in the cluster, z being an integer greater than 1, and send the z vector data to the coordination node. Optionally, the data node can randomly sample z vector data from the vector data saved in the cluster.

[0074] The number of vector data saved by the M clusters is much greater than z*M. If the coordination node clusters based on the vector data saved by the M clusters, the data volume of the clustering operation is very large, which reduces the efficiency of the clustering operation. Therefore, z vector data can be randomly sampled from each cluster, and a total of z*M vector data is obtained. The coordination node clusters based on the z*M vector data, which reduces the data volume of the clustering operation and improves the efficiency of the clustering operation.

[0075] Optionally, the coordination node clusters by using a clustering algorithm. For example, the clustering algorithm can be a k-mean algorithm or the like. The center of the clustering is a vector data, which can be a vector data in the cluster or can be different from each vector data in the cluster.

[0076] For each of the M clusters, for the convenience of description, the cluster is referred to as a first cluster, the data node where the first cluster is located is referred to as a first data node, and the center of the first cluster is also referred to as a first center. After receiving the M centers, the first data node can obtain at least one vector data saved by the first cluster and belonging to a second cluster by performing a redistribution process, and then migrate the at least one vector data from the first cluster to the second cluster. The clusters other than the first cluster in the M clusters include the second cluster.

[0077] The first data node performing the redistribution process can include the following steps (11)-(12).

[0078] (11) The first data node receives the M centers, and obtains at least one vector data saved by the first cluster and belonging to a second cluster based on the M centers.

[0079] For each vector data saved by the first cluster, the first data node obtains the distance between the vector data and each center in the M centers. The center with the smallest distance between the vector data is selected from the M centers. If the selected center is the first center, it is determined that the vector data belongs to the first cluster. If the selected center is not the first center, it is determined that the selected center corresponds to the second cluster, and the vector data is the vector data saved by the first cluster and belonging to the second cluster.

[0080] The first data node can obtain at least one vector data saved by the first cluster and belonging to the second cluster by the above process.

[0081] (12) The first data node migrates the at least one vector data from the first cluster to the second cluster.

[0082] In some embodiments, the second cluster is also located on the first data node, and the first data node saves the at least one vector data to the second cluster and deletes the at least one vector data from the first cluster.

[0083] In some embodiments, the second cluster is not located on the first data node, the first data node determines a second data node where the second cluster is located, sends the at least one vector data to the second data node, and deletes the at least one vector data from the first cluster. The second data node receives the at least one vector data and saves the at least one vector data into the second cluster.

[0084] If the second data node includes multiple clusters, the first data node further sends an identification of the second cluster to the second data node. The second data node receives the identification of the second cluster and saves the at least one vector data into the second cluster based on the identification of the second cluster.

[0085] In some embodiments, after the first data node migrates all vector data not belonging to the first cluster saved by the first cluster, the first data node can obtain distances between any two nodes in the first cluster and establish a first index graph of the first cluster based on the distances between the any two nodes.

[0086] The first index graph includes each node corresponding to vector data saved by the first cluster. For any two nodes connected by a relationship in the first index graph, the distance between the two nodes is equal to the distance between two vector data corresponding to the two nodes, and the distance between the two nodes satisfies a distance condition.

[0087] Optionally, in implementation, the first data node can establish the first index graph of the first cluster through the following (21)-(25) processes.

[0088] (21) : Obtain a node set, the node set including each node corresponding to vector data in the first cluster.

[0089] For example, referring to FIG. 3, the obtained node set includes node 1, node 2, node 3, node 4, node 5, node 6, node 7, and node 8.

[0090] (22) : Obtain distances between any two nodes in the node set.

[0091] Optionally, for any two nodes in the node set, obtain two vector data corresponding to the two nodes, calculate the distance between the two vector data, and take the calculated distance as the distance between the two nodes.

[0092] (23) : For any node in the node set, refer to the node as a first target node, and obtain multiple nodes having minimum distances from the first target node from the node set.

[0093] For example, taking node 1 as the first target node, four nodes with the smallest distance from node 1 are obtained from the node set, which are node 2, node 3, node 4 and node 6 respectively.

[0094] (24) selecting at least one second target node from the plurality of nodes, for each second target node, the distance between the second target node and the first target node is smaller than the distance parameter between any two of the at least one second target node.

[0095] Optionally, the distance parameter between two second target nodes comprises one or more of the distance between the two second target nodes, or the product between the distance between the two second target nodes and a target coefficient, the target coefficient being greater than 1. Optionally, the target coefficient is a numerical value of 1.1, 1.2, 1.3, 1.4 or 1.5.

[0096] For example, assuming that the target coefficient is 1.5, as shown in FIG. 3, the distance 12 between node 1 and node 2, the distance 13 between node 1 and node 3 and the distance 14 between node 1 and node 4 are smaller than 1.5 times the distance between node 2 and node 4, smaller than 1.5 times the distance between node 2 and node 3, and smaller than 1.5 times the distance between node 3 and node 4. The distance between node 1 and node 6 is greater than 1.5 times the distance between node 3 and node 6. Therefore, the at least one second target node selected from node 2, node 3, node 4 and node 6 comprises node 2, node 3 and node 4.

[0097] (25) connecting the first target node and the at least one second target node.

[0098] For example, referring to FIG. 3, node 1 is connected with node 2, node 1 is connected with node 4, and node 1 is connected with node 3.

[0099] If there are remaining nodes in the node set that have not been taken as the first target node, a node is selected from the remaining nodes as the first target node, and then the processes of steps (23)-(25) are repeatedly executed. If there are no remaining nodes in the node set that have not been taken as the first target node, the obtained graph is taken as the first index graph of the first cluster.

[0100] In some embodiments, the first data node further obtains, from the first cluster, an identification of each vector data in the first cluster, and obtains, from the first index graph, a position of a node corresponding to each vector data, and stores the identification of each vector data and the position of the node corresponding to each vector data in a correspondence between the identification of the vector data and the position of the node.

[0101] For other data nodes in the vector database system except the first data node, the other data nodes also perform the above redistribution process and establish the index graph of each cluster included by the other data nodes as the first data node does.

[0102] Wherein, after obtaining the M centers, the coordination node receives the vector data to be stored, and can save the vector data based on the M centers according to the following process. Optionally, the above data node can perform the redistribution process asynchronously, that is, the above data node can perform the redistribution process in the process of performing the following process.

[0103] Step 205: The coordination node receives the first vector data to be stored, obtains the distance between the first vector data and each center, selects the center with the smallest distance between the first vector data from the M centers, and assumes that the selected center is the first center.

[0104] Next, the coordination node saves the first vector data into the first cluster corresponding to the first center, and the detailed implementation process is as follows.

[0105] Step 206: The coordination node determines the first cluster corresponding to the first center, and sends the first vector data to the first data node where the first cluster is located.

[0106] In some embodiments, if the first data node where the first cluster is located includes multiple clusters, the coordination node also sends the identifier of the first cluster to the first data node.

[0107] Step 207: The first data node receives the first vector data and saves the first vector data into the first cluster.

[0108] In step 207, if the first data node includes one cluster, after receiving the first vector data, the first data node saves the first vector data into the first cluster. If the first data node includes multiple clusters, the first data node also receives the identifier of the first cluster, and saves the first vector data into the first cluster based on the identifier of the first cluster.

[0109] In some embodiments, the first data node can also obtain the metadata of the first vector data and save the metadata of the first vector data into the first cluster.

[0110] Step 208: The first data node obtains at least one second node from the first index graph of the first cluster, which satisfies the distance condition between the first node corresponding to the first vector data.

[0111] In step 208, the first data node can obtain a plurality of nodes with the smallest distance from the first node by a greedy neighbor searching manner, and select at least one second node from the plurality of nodes. In the greedy neighbor searching manner, the first vector data is taken as the coordinate of the first node in the first index graph and placed in the first index graph, and then the first queue and the second queue are used to find the plurality of nodes.

[0112] Optionally, in implementation, the at least one second node is obtained through the following flow 2081-2085.

[0113] 2081: add the third target node to the second queue, dequeue the third target node from the second queue, add the third target node to the first queue, add at least one neighbor node of the third target node to the second queue, and sort the nodes in the second queue based on the distance between the first node and each node in the second queue.

[0114] In the sorted second queue, the node at the head has the smallest distance from the first node, and the node at the tail has the largest distance from the first node.

[0115] The third target node is any node in the first index graph, and the at least one neighbor node is a node in the first index graph that has a connection relationship with the third target node. That is, first select any node in the first index graph as the third target node, and add the third target node to the second queue. At this time, the third target node is at the head of the second queue, so dequeue the third target node from the second queue and add the third target node to the first queue.

[0116] For example, referring to FIG. 4, assuming that the third target node is node 1, first add node 1 to the second queue, dequeue node 1 from the second queue, and add node 1 to the first queue. Get the neighbor nodes of node 1, which are node 2, node 3 and node 4 respectively. Add node 2, node 3 and node 4 to the second queue. Based on the distance between node 2 and the first node, the distance between node 3 and the first node, and the distance between node 4 and the first node, sort node 2, node 3 and node 4 in the second queue in order of distance from small to large. The distance between node 4 and the first node is the smallest, the distance between node 3 and the first node is smaller than the distance between node 2 and the first node, so in the sorted second queue, node 4 is at the head of the second queue, node 3 is between node 4 and node 2, that is, the second queue saves the node sequence of node 4, node 3 and node 2.

[0117] The first queue is a fixed-length queue, and the length of the first queue is assumed to be Q, Q being an integer greater than 1, and the first queue is used to save Q nodes with the smallest distance from the first node.

[0118] 2082: dequeue a node at the head of the second queue as a fourth target node, add the fourth target node to the first queue, and sort the nodes in the first queue based on the distance between the first node and each node in the first queue.

[0119] Since the length of the first queue is Q, if the number of nodes in the first queue exceeds Q after adding the fourth target node to the first queue, the node with the largest distance from the first node is automatically dequeued from the first queue.

[0120] For example, the head of the second queue is node 4, node 4 is dequeued from the head of the second queue, and node 4 is added to the first queue. The first queue includes node 1 and node 4, and the distance between node 4 and the first node is smaller than the distance between node 1 and the first node, so the sorted first queue saves the node sequence as node 4 and node 1.

[0121] 2083: determine whether there is a node not added to the second queue among the at least one neighbor node of the fourth target node, add the node not added to the second queue to the second queue if there is a node not added to the second queue, sort the nodes in the second queue based on the distance between the first node and each node in the second queue, and return to execute 2082.

[0122] Referring to FIG. 4, in the first index map, the at least one neighbor node of node 4 includes node 2, node 7, and node 8 that have a connection relationship with node 4. Node 7 and node 8 are not added to the second queue, so node 7 and node 8 are added to the second queue, and the second queue includes node 3, node 2, node 7, and node 8. Node 3, node 2, node 7, and node 8 in the second queue are sorted based on the distance between the first node and each of node 3, node 2, node 7, and node 8 in the second queue. The sorted second queue saves the node sequence as node 8, node 7, node 3, and node 2.

[0123] The head of the second queue is node 8, node 8 is dequeued from the head of the second queue, and node 8 is added to the first queue. The first queue includes node 4, node 1, and node 8. Node 4, node 1, and node 8 in the first queue are sorted based on the distance between the first node and each of node 4, node 1, and node 8 in the first queue. The sorted first queue saves the node sequence as node 8, node 4, and node 1.

[0124] Referring to FIG. 4, in the first index map, at least one neighbor node of node 8 includes node 4, node 7 and node 5 which have a connection relationship with node 8. Node 5 has not been added to the second queue, so node 5 is added to the second queue, and the second queue includes node 7, node 3, node 2 and node 5. Node 7, node 3, node 2 and node 5 in the second queue are sorted based on the distances between the first node and node 7, node 3, node 2 and node 5 in the second queue respectively. The node sequence saved by the sorted second queue is node 7, node 5, node 3 and node 2.

[0125] The head of the second queue is node 7, node 7 is dequeued from the head of the second queue, and the second queue includes node 5, node 3 and node 2. Node 7 is added to the first queue, and the first queue includes node 8, node 4, node 1 and node 7. Node 8, node 4, node 1 and node 7 in the first queue are sorted based on the distances between the first node and node 8, node 4, node 1 and node 7 in the first queue respectively. The node sequence saved by the sorted first queue is node 8, node 7, node 4 and node 1.

[0126] 2084: If there is no node which has not been added to the second queue, it is determined whether the second queue is empty, if the second queue is not empty, the execution returns to 2082, and if the second queue is empty, 2085 is executed.

[0127] If the second queue is empty, the plurality of nodes saved in the first queue are the plurality of nodes with the smallest distances from the first node.

[0128] For example, referring to FIG. 4, in the first index map, at least one neighbor node of node 7 includes node 3, node 4 and node 8 which have a connection relationship with node 7. Node 3, node 4 and node 8 have all been added to the second queue, so there is no node which has not been added to the second queue in the at least one neighbor node of node 7.

[0129] The head of the second queue is node 5, node 5 is dequeued from the head of the second queue, and the second queue includes node 3 and node 2. Node 5 is added to the first queue, and the first queue includes node 8, node 7, node 4, node 1 and node 5. Node 8, node 7, node 4, node 1 and node 5 in the first queue are sorted based on the distances between the first node and node 8, node 7, node 4, node 1 and node 5 in the first queue respectively. Assuming that Q=4, after the nodes in the first queue are sorted, node 1 with the largest distance from the first node is dequeued from the first queue, so the node sequence saved by the sorted first queue is node 8, node 7, node 5 and node 4.

[0130] The head of the second queue is node 3, node 3 is dequeued from the head of the second queue, and the second queue includes node 2. Node 3 is added to the first queue, and the first queue includes node 8, node 7, node 5, node 4, and node 3. Node 8, node 7, node 5, node 4, and node 3 in the first queue are sorted based on distances between the first node and node 8, node 7, node 5, node 4, and node 3 in the first queue, respectively. After the nodes in the first queue are sorted, node 3 with the largest distance from the first node is dequeued from the first queue, and thus the first queue after sorting stores a node sequence of node 8, node 7, node 5, and node 4.

[0131] Referring to FIG. 4, in the first index map, at least one neighbor node of node 3 includes node 1, node 7, and node 6 connected to node 3. Node 6 is not added to the second queue, and thus node 6 is added to the second queue, and the second queue includes node 2 and node 6. Node 2 and node 6 in the second queue are sorted based on distances between the first node and node 2 and node 6 in the second queue, respectively. The second queue after sorting stores a node sequence of node 2 and node 6.

[0132] The head of the second queue is node 2, node 2 is dequeued from the head of the second queue, and the second queue includes node 6. Node 2 is added to the first queue, and the first queue includes node 8, node 7, node 5, node 4, and node 2. Node 8, node 7, node 5, node 4, and node 2 in the first queue are sorted based on distances between the first node and node 8, node 7, node 5, node 4, and node 2 in the first queue, respectively. After the nodes in the first queue are sorted, node 2 with the largest distance from the first node is dequeued from the first queue, and thus the first queue after sorting stores a node sequence of node 8, node 7, node 5, and node 4.

[0133] Referring to FIG. 4, in the first index map, at least one neighbor node of node 2 includes node 1, node 5, and node 4 connected to node 2. Node 1, node 5, and node 4 are added to the second queue. The head of the second queue is node 6, node 6 is dequeued from the head of the second queue, and the second queue is empty. Node 6 is added to the first queue, and the first queue includes node 8, node 7, node 5, node 4, and node 6. Node 8, node 7, node 5, node 4, and node 6 in the first queue are sorted based on distances between the first node and node 8, node 7, node 5, node 4, and node 6 in the first queue, respectively. After the nodes in the first queue are sorted, node 6 with the largest distance from the first node is dequeued from the first queue, and thus the first queue after sorting stores a node sequence of node 8, node 7, node 5, and node 4.

[0134] Since the second queue is empty, the node 8, the node 7, the node 5 and the node 4 in the first queue are four nodes with the smallest distance from the first node.

[0135] 2085: selecting at least one second node from the plurality of nodes saved in the first queue, for each second node, the distance between the second node and the first node is smaller than the distance parameter between any two of the at least one second node.

[0136] In some embodiments, the plurality of nodes can be selected as the plurality of second nodes, and then the first node is added to the first index map and connected with the plurality of second nodes.

[0137] In some embodiments, at least one second node can be selected from the plurality of nodes, since the distance between the second node and the first node is smaller than the distance parameter between any two of the at least one second node, the second node is closer to the first node than other nodes, and connecting the first node with the at least one second node can increase the connectivity of the first index map.

[0138] The distance parameter between the two second nodes includes one or more of the distance between the two second nodes, or the product between the distance between the two second nodes and a target coefficient.

[0139] If the distance parameter between the two second nodes is the distance between the two second nodes, the number of second nodes selected from the plurality of nodes is smaller. And the target coefficient is a number greater than 1, so the distance between the two second nodes can be amplified by the target coefficient, and more second nodes can be selected from the plurality of nodes.

[0140] Step 209: the first data node adds the first node to the first index map and connects the first node with the at least one second node.

[0141] For example, referring to FIG. 5, the at least one second node selected by the first data node from the node 8, the node 7, the node 5 and the node 4 includes the node 8, the node 7, the node 5 and the node 4. The first node (the node 9) is added to the first index map, and the node 9 is connected with the node 8, the node 7, the node 5 and the node 4 respectively. Or, the at least one second node selected by the first data node from the node 8, the node 7, the node 5 and the node 4 includes the node 8, the node 7 and the node 5. The first node (the node 9) is added to the first index map, and the node 9 is connected with the node 8, the node 7 and the node 5 respectively.

[0142] In some embodiments, the first data node further obtains an identity of the first vector data and a position of the first node in the first index map, and stores the identity of the first vector data and the position of the first node in a correspondence between identities of vector data and positions of nodes.

[0143] When there is still vector data to be stored in the vector database system, the coordination node and the data nodes in the vector database system repeat the process of steps 205-209 to store the vector data in one cluster in the vector database system.

[0144] If the length of time since the last time the center of the cluster is obtained reaches a specified length of time, or if the number of vector data stored in the vector database system after the last time the center of the cluster is obtained exceeds a second number threshold, the center of some clusters in the vector database system can drift, and the center of these clusters needs to be re-obtained, and the process of redistribution is performed to migrate vector data not belonging to these clusters.

[0145] Optionally, the vector database system can asynchronously obtain the center of the cluster and perform the process of redistribution. For details of the implementation process, refer to the embodiment shown in FIG. 6, which will not be described in detail here.

[0146] In the embodiment of the present application, after the first data node stores the first vector data to be stored in the first cluster, the first data node obtains at least one second node from the first index map of the first cluster, the distance between the first node corresponding to the first vector data and the at least one second node satisfying the distance condition, adds the first node to the first index map, and connects the first node and the at least one second node. In this way, after the first vector data is stored, it is not necessary to re-establish the first index map based on the vector data stored in the first cluster. Compared with re-establishing the first index map, the operation amount of obtaining at least one second node whose distance from the first node satisfies the distance condition and connecting the first node and the at least one second node is smaller, thereby improving the efficiency of obtaining the index map.

[0147] Referring to FIG. 6, the embodiment of the present application provides a method 600 for processing vector data, which is applied to the vector database system shown in FIG. 1. The method 600 is used to determine a first cluster whose center changes (drifts), and migrate vector data belonging to a second cluster from the first cluster to the second cluster. The method 600 includes the following process.

[0148] Step 601: Obtain the second center based on the vector data stored in the first cluster.

[0149] The data node where the first cluster is located is the first data node, and the center of the first cluster is the first center. The vector data saved by the first cluster is vector data saved by the first cluster based on the first center selected by the coordination node.

[0150] When the length of time from the last time the center of the cluster is acquired reaches a specified length of time, or when the number of vector data saved by the vector database system after the last time the center of the cluster is acquired exceeds a second number threshold, each data node in the vector database system starts to acquire the center of the cluster included by the data node. The process of acquiring the center of the cluster by each data node is the same as the process of acquiring the second center by the first data node.

[0151] In some embodiments, the first data node can sample x vector data from the vector data saved by the first cluster, x is an integer greater than 1 and less than the total number of vector data included by the first cluster, and cluster the x vector data to obtain the second center.

[0152] Optionally, the first data node can randomly sample x vector data from the vector data saved by the first cluster, so that the x vector data is evenly distributed in the first cluster. Clustering the x vector data can improve the accuracy of the second center obtained by clustering. In addition, clustering the x vector data reduces the amount of data for clustering and improves the clustering efficiency.

[0153] Optionally, the first data node clusters by a clustering algorithm to obtain the second center. For example, the clustering algorithm can be a k-mean algorithm, etc.

[0154] Step 602: Determine whether the center of the first cluster changes from the first center to the second center based on the second center. If the center of the first cluster changes from the first center to the second center, perform step 603.

[0155] The center of the first cluster changing from the first center to the second center can also be referred to as the center of the first cluster drifting from the first center to the second center.

[0156] In step 602, the first data node determines whether the center of the first cluster changes from the first center to the second center by the operations of 6021-6022 as follows.

[0157] 6021: The first data node acquires y vector data from the x vector data based on the second center, y is a positive integer less than or equal to x, and the distance between each vector data in the y vector data and the second center is less than the distance between each vector data and the first center, respectively.

[0158] In 6021, for each of the x vector data, the first data node obtains the distance between each vector data and the first center, and the distance between each vector data and the second center. The distance between each vector data and the first center is compared with the distance between each vector data and the second center respectively, and y vector data with the distance between each vector data and the second center smaller than the distance between each vector data and the first center are obtained.

[0159] 6022: When the ratio between y and x exceeds a ratio threshold, the first data node determines that the center of the first cluster changes from the first center to the second center; when the ratio between y and x does not exceed the ratio threshold, the first data node determines that the center of the first cluster does not change and remains the first center.

[0160] Step 603: Based on the second center, the first data node obtains at least one vector data saved by the first cluster and belonging to the second cluster.

[0161] For the data nodes other than the first data node in the vector database system, the data nodes determine whether the center of the cluster included by the data nodes changes as the first data node does, and if it is determined that the center of the cluster of the data nodes changes, the data nodes notify other data nodes and the coordination node included in the vector database system of the new center of the cluster. Similarly, the first data node notifies other data nodes and the coordination node in the vector database system of the second center of the first cluster. Therefore, the first data node has the centers of M-1 clusters other than the first cluster in the vector database system.

[0162] For example, the other M-1 clusters include the second cluster, if the center of the second cluster changes, the center of the second cluster on the first data node is the new center. If the center of the second cluster does not change, the center of the second cluster on the first data node is the old center.

[0163] In step 603, for any vector data saved by the first cluster, the first data node obtains the distance between the vector data and each of the M centers, and selects the center with the smallest distance to the vector data. If the selected center is not the second center, it is determined that the vector data does not belong to the first cluster, but belongs to the cluster corresponding to the selected center, which is referred to as the second cluster for ease of description.

[0164] The same operation is performed on each of the other vector data saved by the first cluster, and at least one vector data saved by the first cluster and belonging to the second cluster is obtained. And the distance between each of the at least one vector data and the center of the second cluster is smaller than the distance between each of the at least one vector data and the second center respectively.

[0165] In some embodiments, the coordination node receives a second center of the first cluster of the notification, updates the first center of the first cluster saved locally as the second center. When the coordination node receives vector data to be stored, obtains a distance between the vector data and each of the M centers saved locally. Selects a center with a minimum distance between the vector data and the center, assuming the selected center is the second center. The coordination node determines the first cluster corresponding to the second center, and sends the vector data to the first data node where the first cluster is located. The first data node saves the vector data into the first cluster, and the detailed implementation process refers to steps 205-209 of the method 200 shown in FIG. 2, which will not be described in detail here.

[0166] Step 604: Migrate at least one vector data from the first cluster to the second cluster.

[0167] In some embodiments, the second cluster is also located on the first data node, and the first data node saves the at least one vector data into the second cluster and deletes the at least one vector data from the first cluster.

[0168] In some embodiments, the second cluster is not located on the first data node, and the first data node determines a second data node where the second cluster is located, sends the at least one vector data to the second data node, and deletes the at least one vector data from the first cluster. The second data node receives the at least one vector data and saves the at least one vector data into the second cluster.

[0169] If the second data node includes multiple clusters, the first data node further sends an identification of the second cluster to the second data node. The second data node receives the identification of the second cluster, and based on the identification of the second cluster, saves the at least one vector data into the second cluster.

[0170] After migrating the at least one vector data from the first cluster to the second cluster, it is also necessary to delete at least one node corresponding to the at least one vector data from the first index map of the first cluster.

[0171] In implementation, the first data node obtains a position of at least one node corresponding to the at least one vector data from a correspondence between an identification of the vector data and a position of the node based on the identification of the at least one vector data. Based on the position of the at least one node, deletes the at least one node from the first index map.

[0172] In some embodiments, after a node is deleted from the first index graph, for other nodes having a connection relationship with the node, the other nodes vacate a neighbor node, in order to improve the passability of the first index graph, at least one node having a distance satisfying a distance condition with the other node can be obtained from the first index graph after the deletion, and the other node and the at least one node are connected in the first index graph after the deletion.

[0173] Optionally, detailed implementation process of obtaining the at least one node having a distance satisfying the distance condition with the other node can refer to the flow of 2081-2085 included in step 208 of method 200 shown in FIG. 2, and will not be described in detail here.

[0174] In some embodiments, after the at least one vector data is migrated from the first cluster to the second cluster, at least one node corresponding to the at least one vector data needs to be added to the index graph of the second cluster.

[0175] In implementation, for each vector data in the at least one vector data, and for a node corresponding to the vector data, at least one node having a distance satisfying a distance condition with the node is obtained from the index graph of the second cluster, the node is added to the index graph of the second cluster, and the node and the at least one node are connected.

[0176] Optionally, detailed implementation process of obtaining the at least one node having a distance satisfying the distance condition with the node can refer to the flow of 2081-2085 included in step 208 of method 200 shown in FIG. 2, and will not be described in detail here.

[0177] In the embodiments of the present application, the first data node obtains a second center based on the vector data saved by the first cluster, and determines whether the center of the first cluster changes from a first center to the second center through the second center. If the center of the first cluster changes from the first center to the second center, based on the second center, it can be concluded that the first cluster includes vector data not belonging to the first cluster, and the vector data not belonging to the first cluster is migrated to a second cluster to which the vector data belongs. In this way, the vector data belonging to the first cluster is still retained in the first cluster, and the first cluster does not include a large amount of vector data not belonging to itself, so that the accuracy of querying data can be improved when querying data to the vector database system.

[0178] Referring to FIG. 7, the embodiments of the present application provide a method 700 of deleting vector data, which is applied to the vector database system shown in FIG. 1. The method 700 is used to delete vector data from a first cluster. The method 700 includes the following flow.

[0179] Step 701: Obtain an identifier of second vector data to be deleted.

[0180] In some embodiments, the coordination node receives a first deletion request, the first deletion request comprising an identification of the second vector data, determines a first cluster where the second vector data is located, and sends the first deletion request to a first data node where the first cluster is located. The first data node receives the first deletion request, and obtains the identification of the second vector data from the first deletion request.

[0181] In some embodiments, the coordination node receives a second deletion request, the second deletion request comprising a deletion condition, and sends the second deletion request to each data node in the vector database system. The first data node receives the second deletion request, and obtains an identification of the second vector data satisfying the deletion condition from the vector data saved in the first cluster.

[0182] In some embodiments, other data nodes receive the second deletion request, and perform the same procedure as the first data node.

[0183] In some embodiments, the deletion condition can comprise one or more of an identification range of the vector data to be deleted, a timestamp range of the vector data to be deleted, or a target type of the vector data to be deleted.

[0184] Optionally, the first cluster comprises metadata of each vector data, and the metadata of the vector data can comprise one or more of an identification of the vector data, a type of the vector data, or a timestamp of the vector data.

[0185] In some embodiments, the deletion condition comprises an identification range of the vector data to be deleted, and the second vector data is the vector data in the first cluster whose identification is within the identification range.

[0186] In some embodiments, the deletion condition comprises a timestamp range of the vector data to be deleted, and the second vector data is the vector data in the first cluster whose timestamp is within the timestamp range.

[0187] In some embodiments, the deletion condition comprises a target type of the vector data to be deleted, and the second vector data is the vector data in the first cluster whose type is the target type.

[0188] Step 702: deleting the second vector data from the first cluster based on the identification of the second vector data, and obtaining a position of a third node corresponding to the second vector data from the correspondence between the identification of the vector data and the position of the node.

[0189] The position of the third node is the position of the third node in a first index map, and the first index map is an index map of the first cluster.

[0190] Step 703: deleting the third node from the first index graph based on the position of the third node to obtain a second index graph.

[0191] The position of the third node can be used to quickly locate the third node in the first index graph, thereby improving the efficiency of deleting the third node from the first index graph.

[0192] Step 704: obtaining a fourth node, the fourth node being a node in the first index graph having a connection relationship with the third node.

[0193] Step 705: obtaining at least one fifth node from the second index graph, the at least one fifth node satisfying a distance condition with the fourth node.

[0194] Optionally, the detailed implementation process of obtaining the at least one fifth node satisfying the distance condition with the fourth node can refer to the process of steps 2081-2085 included in step 208 of the method 200 shown in FIG. 2, which will not be described in detail here.

[0195] Step 706: connecting the fourth node and the at least one fifth node in the second index graph.

[0196] In the embodiments of the present application, before the third node is deleted, there is a fourth node in the first index graph having a connection relationship with the third node, and the third node is a neighbor node of the fourth node. After the third node is deleted from the first index graph, the second index graph is obtained, and the fourth node is left with one neighbor node in the second index graph, which affects the connectivity of the second index graph. Therefore, at least one fifth node satisfying a distance condition with the fourth node is obtained from the second index graph, and the fourth node and the at least one fifth node are connected in the second index graph. The at least one fifth node is a neighbor node of the fourth node, thereby enriching the neighbors of the fourth node and improving the connectivity of the second index graph, and thereby improving the query efficiency when querying data.

[0197] Referring to FIG. 8, the embodiments of the present application provide a method 800 for querying vector data, which is applied to the vector database system shown in FIG. 1. The method 800 is used for querying vector data from a first cluster. The method 800 includes the following processes.

[0198] Step 801: receiving a query request, the query request including third vector data to be queried and a filter condition.

[0199] In some embodiments, the coordination node receives the query request, the query request including third vector data to be queried and a filter condition.

[0200] In some embodiments, the filtering condition can include one or more of a target identification range of the vector data to be queried, a target timestamp range of the vector data to be queried, or a target type of the vector data to be queried.

[0201] For example, the filtering condition can be to query the vector data with an identification less than 1000, or the filtering condition can be to query the vector data with a timestamp between t0 and t1, or the filtering condition can be to query the vector data with a type of origin of the vector data, and the filtering condition can be to query the vector data with an origin of a certain city.

[0202] Step 802: Based on the third vector data and the filtering condition, determining at least one cluster included in the vector database system, the at least one cluster including a total number of vector data satisfying the filtering condition and having a distance less than a distance threshold from the third vector data being greater than or equal to K, K being an integer greater than 1.

[0203] In step 802, the coordination node can obtain the at least one cluster through the operations of 8021-8026 as follows.

[0204] 8021: Obtaining a proportion corresponding to each cluster included in the vector database system, the proportion corresponding to the cluster being a ratio between a number of vector data satisfying the filtering condition and a total number of vector data included in the cluster.

[0205] The coordination node includes statistical information of each cluster, and the statistical information of the cluster can include a number of vector data in different identification ranges in the cluster, a number of vector data in different timestamp ranges in the cluster, and / or a number of different types of vector data in the cluster.

[0206] For example, referring to FIG. 9, a histogram of a cluster included in the coordination node, i.e., the statistical information of the cluster, is in the form of a histogram. The horizontal axis of the histogram includes different identification ranges (or different timestamp ranges or different types), and the height of the rectangular graph corresponding to any identification range represents the number of vector data in the identification range. In FIG. 9, ID: 0-100 represents an identification range greater than or equal to 0 and less than or equal to 100, ID: 101-200 represents an identification range greater than or equal to 101 and less than or equal to 200, and ID: 201-300 represents an identification range greater than or equal to 201 and less than or equal to 300.

[0207] In 8021, for each cluster in the vector database system, the coordination node obtains, based on the statistical information of the cluster, a number of vector data satisfying the filtering condition included in the cluster, and based on the number of vector data satisfying the filtering condition included in the cluster and a total number of vector data included in the cluster, obtains a proportion corresponding to the cluster.

[0208] In some embodiments, the filter condition comprises a target identity range, the number of vector data in the target identity range is obtained by acquiring at least one identity range intersecting with the target identity range and based on the number of vector data in the at least one identity range. The number of vector data in the target identity range is the number of vector data included in the cluster and satisfying the filter condition.

[0209] For example, assuming that the target identity range is ID less than 150, the histogram of a cluster in the vector database system is as shown in FIG. 9. The identity ranges intersecting with the target identity range include ID: 0-100 and ID: 101-200. The number of vector data in the identity range ID: 0-100 is 1000 and the number of vector data in the identity range ID: 101-200 is 1200, which are obtained from the histogram of the cluster as shown in FIG. 9.

[0210] The ratio between the interval length of the intersection between the identity range ID: 101-200 and the target identity range and the interval length of the identity range ID: 101-200 is 0.5, so the number of vector data in the target identity range is equal to 1000+0.5*1200=1600.

[0211] In the case where the filter condition comprises a target timestamp range, the process of obtaining the number of vector data in the target timestamp range can refer to the process of obtaining the number of vector data in the target identity range described above, which will not be described in detail here.

[0212] In some embodiments, the filter condition comprises a target type, the target type comprises one or more types, and the number of vector data corresponding to each type can be obtained from the histogram of the cluster. The number of vector data corresponding to each type is accumulated to obtain the number of vector data included in the cluster and satisfying the filter condition.

[0213] 8022: Based on the vector number prediction model, the third vector data, the distance threshold, and the center of each cluster, the number corresponding to each cluster is obtained. The number corresponding to each cluster is the number of vector data included in the cluster and having a distance less than the distance threshold from the third vector data.

[0214] The initial value of the distance threshold can be a specified threshold. For example, assuming that the specified threshold is 0.2, the initial value of the distance threshold is 0.2.

[0215] In 8022, the coordination node inputs the third vector data, the distance threshold, and the center of each cluster in the vector database system into the vector number prediction model. The vector number prediction model reasons the third vector data, the distance threshold, and the center of each cluster to obtain the number of vector data included in each cluster and having a distance less than the distance threshold from the third vector data. The coordination node obtains the number of vector data included in each cluster and having a distance less than the distance threshold from the third vector data output by the vector number prediction model.

[0216] 8023: Obtain, based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster, the second number corresponding to each cluster being the number of vector data included in the cluster and having a distance less than the distance threshold from the third vector data and satisfying the filtering condition.

[0217] In 8023, the coordination node multiplies the first number corresponding to the cluster and the proportion corresponding to the cluster to obtain the second number corresponding to the cluster.

[0218] 8024: Accumulate the second number corresponding to each cluster to obtain the total number of vector data included in the vector database system and having a distance less than the distance threshold from the third vector data and satisfying the filtering condition, when the total number is less than K, perform 8025, and when the total number is greater than or equal to K, perform 8026.

[0219] K is the number of vector data included in the vector database system and having a distance less than the distance threshold from the third vector data and satisfying the filtering condition, K is an integer greater than 1.

[0220] 8025: Accumulate the distance threshold and the offset value, update the distance threshold to the accumulated value, and return to perform 8022.

[0221] The total number of vector data included in the vector database system and having a distance less than the distance threshold from the third vector data and satisfying the filtering condition being less than K indicates that the distance threshold is small, resulting in fewer vector data found to have a distance less than the distance threshold from the third vector data, and thus the distance threshold needs to be enlarged.

[0222] For example, assuming that the distance threshold is 0.2 and the offset value is 0.01, the distance threshold 0.2 and the offset value 0.01 are accumulated, the distance threshold 0.2 is updated to the accumulated value 0.21, that is, the distance threshold becomes 0.21, and then it is returned to perform 8022.

[0223] 8026: Select at least one cluster whose second number is greater than or equal to the number threshold.

[0224] For example, the number threshold is 1, and the coordination node can select at least one cluster whose second number is greater than or equal to 1.

[0225] For the above vector number prediction model, a plurality of training samples can be obtained, each training sample including fourth vector data, a distance threshold, a center of each cluster, and a number of vector data in each cluster that has a distance less than the distance threshold from the fourth vector data. Then, the vector number prediction model is trained based on the plurality of training samples.

[0226] Step 803: Querying, from the at least one cluster, K vector data whose metadata satisfies the filtering condition and has a minimum distance from the third vector data.

[0227] In step 803, the coordination node determines a data node where the at least one cluster is located, and sends a query request to the determined data node.

[0228] The determined data node includes the first data node, and the at least one cluster includes the first cluster. The first data node includes the first cluster and the first index map of the first cluster. The first data node receives the query request, queries, based on the first index map, at least one vector data whose metadata satisfies the filtering condition and has a minimum distance from the third vector data from the first cluster, and sends the queried at least one vector data to the coordination node. The coordination node can receive vector data sent by different data nodes, the number of received vector data is greater than or equal to K, select K vector data whose distance from the third vector data is the smallest from the received vector data, and send the K vector data.

[0229] The at least one cluster found by the coordination node includes vector data whose metadata satisfies the filtering condition, and the vector data whose metadata satisfies the filtering condition includes one or more vector data closest to the third vector data. Therefore, the query from the at least one cluster narrows down the query range, not only improves the query efficiency, but also improves the query accuracy.

[0230] In some embodiments, the process in which the first data node queries, based on the first index map, at least one vector data whose metadata satisfies the filtering condition and has a minimum distance from the third vector data from the first cluster, can be:

[0231] For the sixth node corresponding to the third vector data, the first data node places the third vector data as the coordinates of the sixth node in the first index map in the first index map, and then finds a plurality of nodes in a greedy neighbor-finding manner, the plurality of nodes being a plurality of nodes having a minimum distance from the sixth node. The plurality of vector data corresponding to the plurality of nodes are a plurality of vector data in the first cluster having a minimum distance from the third vector data. At least one vector data whose metadata satisfies the filtering condition is obtained from the plurality of vector data.

[0232] The first data node finds the detailed implementation process of the plurality of nodes by a greedy neighbor searching manner. Referring to the flow of steps 2081-2084 included in step 208 of the embodiment shown in FIG. 2, detailed description is not given herein.

[0233] In the embodiment of the present application, when the query request is received, the query request includes third vector data to be queried and a filter condition. Based on the third vector data and the filter condition, at least one cluster included in the vector database system is determined, and the total number of vector data in the at least one cluster whose metadata satisfies the filter condition and whose distance from the third vector data is less than a distance threshold is greater than or equal to K. In this way, the query range is narrowed, and since the vector data in the at least one cluster whose metadata satisfies the filter condition has one or more vector data closest to the third vector data, the K vector data whose metadata satisfies the filter condition and whose distance from the third vector data is the smallest is queried from the at least one cluster, which not only improves the query efficiency, but also improves the query accuracy.

[0234] Referring to FIG. 10, the embodiment of the present application provides a device 1000 for processing vector data, and the device 1000 can be applied to the vector database system 100 shown in FIG. 1. The device 1000 includes a first cluster, and the first cluster is used to save a plurality of vector data, and the plurality of vector data is data saved to the first cluster based on a first center, and the first center is a center of the first cluster. The device 1000 includes:

[0235] An acquisition unit 1001 is configured to acquire a second center based on vector data saved by the first cluster;

[0236] A processing unit 1002 is configured to acquire at least one vector data based on the second center when the center of the first cluster changes from the first center to the second center based on the second center, and the at least one vector data is data saved by the first cluster and belonging to the second cluster. The device 1000 further includes a second cluster.

[0237] The processing unit 1002 is further configured to migrate the at least one vector data from the first cluster to the second cluster.

[0238] Optionally, the detailed implementation process of the acquisition unit 1001 for acquiring the second center based on the vector data saved by the first cluster can refer to the related content of step 601 of the method 600 shown in FIG. 6, and detailed description is not given herein.

[0239] Optionally, the detailed implementation process of the processing unit 1002 for acquiring the at least one vector data based on the second center when the center of the first cluster changes from the first center to the second center based on the second center can refer to the related content of steps 602-603 of the method 600 shown in FIG. 6, and detailed description is not given herein.

[0240] Optionally, the detailed implementation process of the processing unit 1002 migrating the at least one vector data from the first cluster to the second cluster can refer to the related content of step 604 of method 600 shown in FIG. 6, which will not be described in detail here.

[0241] Optionally, for each of the at least one vector data, the distance between the vector data and the center of the second cluster is less than the distance between the vector data and the second center.

[0242] Optionally, the obtaining unit 1001 is configured to:

[0243] sample x vector data from the vector data saved by the first cluster, x being an integer greater than 1 and less than the total number of vector data included in the first cluster;

[0244] perform clustering based on the x vector data to obtain the second center.

[0245] Optionally, the detailed implementation process of the processing unit 1002 sampling x vector data from the vector data saved by the first cluster can refer to the related content of step 601 of method 600 shown in FIG. 6, which will not be described in detail here.

[0246] Optionally, the detailed implementation process of the processing unit 1002 performing clustering based on the x vector data to obtain the second center can refer to the related content of step 601 of method 600 shown in FIG. 6, which will not be described in detail here.

[0247] Optionally, the processing unit 1002 is configured to:

[0248] based on the second center, obtain y vector data from the x vector data, y being a positive integer less than or equal to x, and for each of the y vector data, the distance between the vector data and the second center is less than the distance between the vector data and the first center;

[0249] when the ratio between y and x exceeds a ratio threshold, determine that the center of the first cluster changes from the first center to the second center.

[0250] Optionally, the detailed implementation process of the processing unit 1002 obtaining y vector data from the x vector data based on the second center can refer to the related content of step 602 of method 600 shown in FIG. 6, which will not be described in detail here.

[0251] Optionally, the detailed implementation process of the processing unit 1002 determining that the center of the first cluster changes from the first center to the second center can refer to the related content of step 602 of method 600 shown in FIG. 6, which will not be described in detail here.

[0252] Optionally, the apparatus 1000 further comprises a first index graph of the first cluster, the first index graph comprising a node corresponding to each vector data in the first cluster, and for any two nodes having a connection relationship in the first index graph, a distance between the two nodes satisfies a distance condition;

[0253] The apparatus 1000 further comprises a first receiving unit 1003;

[0254] The first receiving unit 1003 is configured to receive first vector data to be stored.

[0255] The processing unit 1002 is further configured to save the first vector data into the first cluster based on the second center.

[0256] The processing unit 1002 is further configured to acquire at least one second node from the first index graph, the distance between the first node corresponding to the first vector data and the at least one second node satisfying the distance condition.

[0257] The processing unit 1002 is further configured to add the first node into the first index graph and connect the first node with the at least one second node.

[0258] Optionally, the first receiving unit 1003 receives the first vector data to be stored, and the detailed implementation process can refer to the related content of step 205 of method 200 shown in FIG. 2, which will not be described in detail here.

[0259] Optionally, the processing unit 1002 saves the first vector data into the first cluster based on the second center, and the detailed implementation process can refer to the related content of step 207 of method 200 shown in FIG. 2, which will not be described in detail here.

[0260] Optionally, the processing unit 1002 acquires at least one second node from the first index graph, the distance between the first node corresponding to the first vector data and the at least one second node satisfying the distance condition, and the detailed implementation process can refer to the related content of step 208 of method 200 shown in FIG. 2, which will not be described in detail here.

[0261] Optionally, the processing unit 1002 adds the first node into the first index graph and connects the first node with the at least one second node, and the detailed implementation process can refer to the related content of step 209 of method 200 shown in FIG. 2, which will not be described in detail here.

[0262] Optionally, the processing unit 1002 is configured to:

[0263] acquire a plurality of nodes from the first index graph, the distance between the first node and the plurality of nodes being the smallest;

[0264] selecting at least one second node from the plurality of nodes, for each second node, a distance between the second node and the first node is less than a distance between any two second nodes of the at least one second node, or the distance between the second node and the first node is less than a product of the distance between the two second nodes and a target coefficient, the target coefficient being greater than 1.

[0265] Optionally, the processing unit 1002 obtains the plurality of nodes with the smallest distance from the first node from the first index map, and the detailed implementation process can refer to the related content of step 208 of method 200 shown in FIG. 2, which will not be described in detail here.

[0266] Optionally, the processing unit 1002 selects at least one second node from the plurality of nodes, and the detailed implementation process can refer to the related content of step 208 of method 200 shown in FIG. 2, which will not be described in detail here.

[0267] Optionally, the apparatus 1000 further includes a correspondence between an identifier of each vector data in the first cluster and a position of a node corresponding to the vector data, and the processing unit 1002 is further configured to:

[0268] obtain an identifier of a second vector data to be deleted;

[0269] delete the second vector data from the first cluster based on the identifier of the second vector data, and obtain a position of a third node corresponding to the second vector data from the correspondence;

[0270] delete the third node from the first index map based on the position of the third node, to obtain a second index map.

[0271] Optionally, the processing unit 1002 obtains the identifier of the second vector data to be deleted, and the detailed implementation process can refer to the related content of step 701 of method 700 shown in FIG. 7, which will not be described in detail here.

[0272] Optionally, the processing unit 1002 deletes the second vector data from the first cluster based on the identifier of the second vector data, and obtains the position of the third node corresponding to the second vector data from the correspondence, and the detailed implementation process can refer to the related content of step 702 of method 700 shown in FIG. 7, which will not be described in detail here.

[0273] Optionally, the processing unit 1002 deletes the third node from the first index map based on the position of the third node, to obtain a second index map, and the detailed implementation process can refer to the related content of step 703 of method 700 shown in FIG. 7, which will not be described in detail here.

[0274] Optionally, the processing unit 1002 is further configured to:

[0275] obtain a fourth node, the fourth node being a node in the first index graph having a connection relationship with the third node;

[0276] obtain at least one fifth node from the second index graph, the at least one fifth node satisfying a distance condition with the fourth node;

[0277] connect the fourth node and the at least one fifth node in the second index graph.

[0278] Optionally, the detailed implementation process of the processing unit 1002 obtaining the fourth node can refer to the related content of step 704 of the method 700 shown in FIG. 7, and will not be described in detail here.

[0279] Optionally, the detailed implementation process of the processing unit 1002 obtaining at least one fifth node from the second index graph, the at least one fifth node satisfying a distance condition with the fourth node can refer to the related content of step 705 of the method 700 shown in FIG. 7, and will not be described in detail here.

[0280] Optionally, the detailed implementation process of the processing unit 1002 connecting the fourth node and the at least one fifth node in the second index graph can refer to the related content of step 706 of the method 700 shown in FIG. 7, and will not be described in detail here.

[0281] Optionally, the first cluster is further used to save metadata of each vector data in the plurality of vector data, and the apparatus 1000 further includes a second receiving unit 1004.

[0282] The second receiving unit 1004 is configured to receive a query request, the query request including third vector data to be queried and a filter condition.

[0283] The processing unit 1002 is further configured to determine, based on the third vector data and the filter condition, at least one cluster included in the apparatus 1000, the at least one cluster including a total number of vector data whose metadata satisfies the filter condition and whose distance from the third vector data is less than a distance threshold being greater than or equal to K, K being an integer greater than 1.

[0284] The processing unit 1002 is further configured to query, from the at least one cluster, K vector data whose metadata satisfies the filter condition and whose distance from the third vector data is the smallest.

[0285] Optionally, the detailed implementation process of the second receiving unit 1004 receiving the query request can refer to the related content of step 801 of the method 800 shown in FIG. 8, and will not be described in detail here.

[0286] Optionally, the processing unit 1002 determines at least one cluster included in the device 1000 based on the third vector data and the filtering condition. For details, refer to the related content of step 802 of method 800 shown in FIG. 8, which will not be described in detail here.

[0287] Optionally, the processing unit 1002 queries, from the at least one cluster, K vector data whose metadata satisfy the filtering condition and whose distance to the third vector data is the smallest. For details, refer to the related content of step 803 of method 800 shown in FIG. 8, which will not be described in detail here.

[0288] Optionally, the processing unit 1002 is configured to:

[0289] obtain a proportion corresponding to each cluster included in the device 1000, the proportion corresponding to the cluster being a ratio between a number of vector data included in the cluster and satisfying the filtering condition and a total number of vector data included in the cluster;

[0290] obtain a first number corresponding to each cluster based on the vector number prediction model, the third vector data, the distance threshold, and a center of each cluster, the first number corresponding to the cluster being a number of vector data included in the cluster and having a distance less than the distance threshold to the third vector data;

[0291] obtain a second number corresponding to each cluster based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to the cluster being a number of vector data included in the cluster and satisfying the filtering condition and having a distance less than the distance threshold to the third vector data;

[0292] select at least one cluster whose second number is greater than or equal to the number threshold, in a case where an accumulated value of the second number corresponding to each cluster is greater than or equal to K.

[0293] Optionally, the processing unit 1002 obtains the proportion corresponding to each cluster included in the device 1000. For details, refer to the related content of step 8021 of method 800 shown in FIG. 8, which will not be described in detail here.

[0294] Optionally, the processing unit 1002 obtains the first number corresponding to each cluster based on the vector number prediction model, the third vector data, the distance threshold, and the center of each cluster. For details, refer to the related content of step 8022 of method 800 shown in FIG. 8, which will not be described in detail here.

[0295] Optionally, the processing unit 1002 obtains the second number corresponding to each cluster based on the first number corresponding to each cluster and the proportion corresponding to each cluster. For details, refer to the related content of step 8023 of method 800 shown in FIG. 8, which will not be described in detail here.

[0296] Optionally, the processing unit 1002 selects the second number of at least one cluster greater than or equal to the number threshold. For details, refer to the related content of step 8026 of method 800 shown in FIG. 8, which will not be described in detail here.

[0297] Optionally, the processing unit 1002 is further configured to:

[0298] The vector number prediction model is trained based on a plurality of training samples, and the training samples include fourth vector data, a distance threshold, a center of each cluster, and a number of vector data in each cluster having a distance less than the distance threshold from the fourth vector data.

[0299] In the embodiment of the present application, the obtaining unit obtains the second center based on the vector data saved by the first cluster, and the processing unit determines whether the center of the first cluster changes from the first center to the second center through the second center. If the center of the first cluster changes from the first center to the second center, based on the second center, it can be concluded that the first cluster includes vector data not belonging to the first cluster, and the vector data not belonging to the first cluster is migrated to the second cluster to which the vector data belongs. In this way, the vector data belonging to the first cluster is still retained in the first cluster, and the first cluster does not include a large amount of vector data not belonging to itself, so that the accuracy of querying data in the vector database system can be improved.

[0300] Referring to FIG. 11, an embodiment of the present application provides a computing device 1100. For example, the computing device 1100 can be a node included in the vector database system 100 shown in FIG. 1, or the computing device 1100 can be a node in the vector database system in the method 200 shown in FIG. 2, the method 600 shown in FIG. 6, the method 700 shown in FIG. 7, or the method 800 shown in FIG. 8, etc.

[0301] As shown in FIG. 11, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate through the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1100 is not limited by the present application.

[0302] The bus 1102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is represented in FIG. 11, but it does not mean that there is only one bus or only one type of bus. The bus 1102 can include a path for transmitting information between various components (e.g., the processor 1104, the memory 1106, the communication interface 1108) of the computing device 1100.

[0303] The processor 1104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0304] The memory 1106 can include a volatile memory (e.g., a random access memory (RAM)), and can also include a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0305] Referring to FIG. 11, the memory 1106 stores executable program codes, and the processor 1104 executes the executable program codes to respectively implement the functions of the obtaining unit 1001, the processing unit 1002, the first receiving unit 1003, and the second receiving unit 1004 in the apparatus 1000 shown in FIG. 10, thereby implementing the method provided in any of the above embodiments. That is, the memory 1106 has instructions for executing the method provided in any of the above embodiments. Alternatively,

[0306] The communication interface 1108 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, etc., to implement the communication between the computing device 1100 and other devices or communication networks.

[0307] The embodiments of the present application also provide a cluster for processing vector data. The cluster for processing vector data comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0308] As shown in FIG. 12, the cluster for processing vector data comprises at least one computing device 1100. The same instructions for performing the method provided in any of the above embodiments can be stored in the memory 1106 of one or more computing devices 1100 in the cluster for processing vector data.

[0309] In some possible implementation manners, partial instructions for performing the method for processing vector data can also be stored in the memory 1106 of one or more computing devices 1100 in the cluster for processing vector data, respectively. In other words, the combination of one or more computing devices 1100 can collectively perform the instructions for performing the method provided in any of the above embodiments.

[0310] In some possible implementation manners, one or more computing devices in the cluster for processing vector data can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 13 shows a possible implementation manner. As shown in FIG. 13, two computing devices 1100A and 1100B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices.

[0311] In this type of possible implementation manner, the memory 1106 in the computing device 1100A stores instructions for performing the functions of the obtaining unit 1001 and the processing unit 1002 in the embodiment shown in FIG. 10. Meanwhile, the memory 1106 in the computing device 1100B stores instructions for performing the functions of the first receiving unit 1003 and the second receiving unit 1004 in the embodiment shown in FIG. 10.

[0312] It should be understood that the functions of the computing device 1100A shown in FIG. 13 can also be completed by multiple computing devices 1100. Similarly, the functions of the computing device 1100B can also be completed by multiple computing devices 1100.

[0313] The embodiments of the present application also provide another cluster for processing vector data. The connection relationship between the computing devices in the cluster for processing vector data can be similar to the connection manner of the cluster for processing vector data described with reference to FIG. 13. The difference is that the same instructions for performing the method provided in any of the above embodiments can be stored in the memory 1106 of one or more computing devices 1100 in the cluster for processing vector data.

[0314] In some possible implementation manners, the memory 1106 of the one or more computing devices 1100 in the cluster that processes the vector data can also respectively store partial instructions for performing the method provided by any of the above embodiments. In other words, the combination of the one or more computing devices 1100 can collectively execute the instructions for performing the method provided by any of the above embodiments.

[0315] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the method provided by any of the above embodiments.

[0316] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the method provided by any of the above embodiments.

[0317] Those of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0318] The above description is only optional embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of processing vector data, characterized by, The method is applied to a vector database system, the vector database system comprising a first cluster for saving a plurality of vector data, the plurality of vector data being data saved to the first cluster based on a first center, the first center being a center of the first cluster, the method comprising: obtaining a second center based on the vector data saved by the first cluster; determining that the center of the first cluster changes from the first center to the second center based on the second center; obtaining at least one vector data based on the second center, the at least one vector data being data saved by the first cluster and belonging to a second cluster, the vector database system further comprising the second cluster; 2. The method of claim 1, wherein, migrating the at least one vector data from the first cluster to the second cluster.

3. The method of claim 1 or 2, wherein, For each of the at least one vector data, a distance between the vector data and a center of the second cluster is less than a distance between the vector data and the second center. The obtaining of the second center based on the vector data saved by the first cluster comprises: sampling x vector data from the vector data saved by the first cluster, x being an integer greater than 1 and less than a total number of vector data included by the first cluster; 4. The method of claim 3, wherein, performing clustering based on the x vector data to obtain the second center. The determining that the center of the first cluster changes from the first center to the second center based on the second center comprises: obtaining y vector data from the x vector data based on the second center, y being a positive integer less than or equal to x, for each of the y vector data, a distance between the vector data and the second center is less than a distance between the vector data and the first center; 5. The method according to any one of claims 1 to 4, wherein determining that the center of the first cluster changes from the first center to the second center when a ratio between the y and the x exceeds a ratio threshold. The vector database system further comprises a first index graph of the first cluster, the first index graph comprising a node corresponding to each vector data in the first cluster, a distance between any two nodes having a connection relationship in the first index graph satisfying a distance condition; The method further comprises, after the determining that the center of the first cluster changes from the first center to the second center based on the second center: receiving a first vector data to be stored; saving the first vector data to the first cluster based on the second center; obtaining at least one second node from the first index graph, the at least one second node satisfying the distance condition with a distance between the first node corresponding to the first vector data; 6. The method of claim 5, wherein, adding the first node to the first index graph and connecting the first node with the at least one second node. The obtaining of the at least one second node from the first index graph, the at least one second node satisfying the distance condition with a distance between the first node corresponding to the first vector data, comprises: obtaining a plurality of nodes from the first index graph, the plurality of nodes having a minimum distance with the first node; The at least one second node is selected from the plurality of nodes, and for each second node, a distance between the second node and the first node is less than a distance between any two of the at least one second node, or the distance between the second node and the first node is less than a product of the distance between the two second nodes and a target coefficient, and the target coefficient is greater than 1.

7. The method of claim 5 or 6, wherein, The vector database system further comprises a correspondence between an identifier of each vector data in the first cluster and a location of a node corresponding to the vector data, and the method further comprises: obtaining an identifier of a second vector data to be deleted; based on the identifier of the second vector data, deleting the second vector data from the first cluster, and obtaining a location of a third node corresponding to the second vector data from the correspondence; based on the location of the third node, deleting the third node from the first index graph to obtain a second index graph.

8. The method of claim 7, wherein, After the third node is deleted from the first index graph based on the location of the third node, the method further comprises: obtaining a fourth node, the fourth node being a node in the first index graph having a connection relationship with the third node; obtaining at least one fifth node from the second index graph, the at least one fifth node satisfying a distance condition with the fourth node; connecting the fourth node and the at least one fifth node in the second index graph.

9. The method according to any one of claims 1 to 8, wherein, The first cluster is further configured to store metadata of each vector data in the plurality of vector data, and the method further comprises: receiving a query request, the query request comprising a third vector data to be queried and a filter condition; based on the third vector data and the filter condition, determining at least one cluster included in the vector database system, the at least one cluster comprising a total number of vector data whose metadata satisfy the filter condition and whose distance with the third vector data is less than a distance threshold being greater than or equal to K, K being an integer greater than 1; querying, from the at least one cluster, K vector data whose metadata satisfy the filter condition and whose distance with the third vector data is the smallest.

10. The method of claim 9, wherein, The determining, based on the third vector data and the filter condition, of the at least one cluster included in the vector database system comprises: obtaining a proportion corresponding to each cluster included in the vector database system, the proportion corresponding to the cluster being a ratio between a number of vector data whose metadata satisfy the filter condition and a total number of vector data included in the cluster; based on a vector number prediction model, the third vector data, the distance threshold, and a center of each cluster, obtaining a first number corresponding to the cluster, the first number corresponding to the cluster being a number of vector data included in the cluster and whose distance with the third vector data is less than the distance threshold; and obtaining a second number corresponding to each cluster based on the first number corresponding to each cluster and the proportion corresponding to each cluster, the second number corresponding to each cluster being a number of vector data included in the cluster and satisfying the filtering condition and having a distance less than the distance threshold to the third vector data; in a case where an accumulated value of the second number corresponding to each cluster is greater than or equal to K, selecting the at least one cluster having a second number greater than or equal to a number threshold.

11. The method of claim 10, wherein, The method further includes: training the vector number prediction model based on a plurality of training samples, the training samples including fourth vector data, the distance threshold, the center of each cluster, and a number of vector data in each cluster and having a distance less than the distance threshold to the fourth vector data.

12. An apparatus for processing vector data, characterized by The device includes a first cluster for storing a plurality of vector data, the plurality of vector data being data stored into the first cluster based on a first center, the first center being a center of the first cluster, and the device includes: an obtaining unit configured to obtain a second center based on vector data stored by the first cluster; a processing unit configured to, based on the center of the first cluster being changed from the first center to the second center based on the second center, obtain at least one vector data based on the second center, the at least one vector data being data stored by the first cluster and belonging to a second cluster, and the device further includes the second cluster; the processing unit is further configured to migrate the at least one vector data from the first cluster to the second cluster.

13. The apparatus of claim 12, wherein, For each of the at least one vector data, a distance between the vector data and a center of the second cluster is less than a distance between the vector data and the second center.

14. The apparatus of claim 12 or 13, wherein, The obtaining unit is configured to: sample x vector data from the vector data stored by the first cluster, x being an integer greater than 1 and less than a total number of vector data included in the first cluster; perform clustering based on the x vector data to obtain the second center.

15. The apparatus of claim 14, wherein, The processing unit is configured to: based on the second center, obtain y vector data from the x vector data, y being a positive integer less than or equal to the x, for each of the y vector data, a distance between the vector data and the second center is less than a distance between the vector data and the first center; determine that the center of the first cluster is changed from the first center to the second center when a ratio between the y and the x exceeds a ratio threshold.

16. The apparatus of any one of claims 12-15, wherein, The device further includes a first index map of the first cluster, the first index map including a node corresponding to each vector data in the first cluster, for any two nodes having a connection relationship in the first index map, a distance between the two nodes satisfies a distance condition; The device further includes a first receiving unit; the first receiving unit is configured to receive first vector data to be stored; the processing unit is further configured to store the first vector data into the first cluster based on the second center; The processing unit is further configured to obtain, from the first index graph, at least one second node that satisfies the distance condition with the first node corresponding to the first vector data; The processing unit is further configured to add the first node to the first index graph, and connect the first node with the at least one second node.

17. The apparatus of claim 16, wherein, The processing unit is configured to: obtain, from the first index graph, a plurality of nodes that have the smallest distance with the first node; select, from the plurality of nodes, the at least one second node, for each second node, the distance between the second node and the first node is smaller than the distance between any two second nodes in the at least one second node, or the distance between the second node and the first node is smaller than the product of the distance between the two second nodes and a target coefficient, and the target coefficient is greater than 1.

18. The apparatus of claim 16 or 17, wherein, The device further comprises a correspondence relationship between the identity of each vector data in the first cluster and the position of the node corresponding to the each vector data, and the processing unit is further configured to: obtain the identity of a second vector data to be deleted; delete, from the first cluster, the second vector data based on the identity of the second vector data, and obtain, from the correspondence relationship, the position of a third node corresponding to the second vector data; delete, from the first index graph, the third node based on the position of the third node, to obtain a second index graph.

19. The apparatus of claim 18, wherein, The processing unit is further configured to: obtain a fourth node, the fourth node being a node in the first index graph that has a connection relationship with the third node; obtain, from the second index graph, at least one fifth node that satisfies the distance condition with the fourth node; connect the fourth node with the at least one fifth node in the second index graph.

20. The apparatus of any one of claims 12-19, wherein, The first cluster is further configured to save the metadata of each vector data in the plurality of vector data, and the device further comprises a second receiving unit; The second receiving unit is configured to receive a query request, the query request comprising third vector data to be queried and a filter condition; The processing unit is further configured to determine, based on the third vector data and the filter condition, at least one cluster included in the device, the at least one cluster comprising a total number of vector data whose metadata satisfy the filter condition and whose distance with the third vector data is smaller than a distance threshold, the total number being greater than or equal to K, K being an integer greater than 1; The processing unit is further configured to query, from the at least one cluster, K vector data whose metadata satisfy the filter condition and whose distance with the third vector data is the smallest.

21. The apparatus of claim 20, wherein, The processing unit is configured to: obtain the proportion corresponding to each cluster included in the device, the proportion corresponding to the cluster being the ratio between the number of vector data whose metadata satisfy the filter condition and the total number of vector data included in the cluster; obtain a first number of each cluster based on the vector number prediction model, the third vector data, the distance threshold, and the center of each cluster, the first number of each cluster being a number of vector data included in the cluster and having a distance less than the distance threshold from the third vector data; obtain a second number of each cluster based on the first number of each cluster and the proportion of each cluster, the second number of each cluster being a number of vector data included in the cluster and having a distance less than the distance threshold from the third vector data and satisfying the filtering condition; in a case where an accumulated value of the second number of each cluster is greater than or equal to K, select the at least one cluster having a second number greater than or equal to the number threshold.

22. The apparatus of claim 21, wherein, The processing unit is further configured to: train the vector number prediction model based on a plurality of training samples, the training samples including fourth vector data, the distance threshold, the center of each cluster, and a number of vector data in each cluster and having a distance less than the distance threshold from the fourth vector data.

23. A vector database system, characterized by The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the vector database system to perform the method of any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The instructions, when executed by the vector database system, cause the vector database system to perform the method of any one of claims 1-11.

25. A computer program product comprising instructions, wherein: The instructions, when executed by the vector database system, cause the vector database system to perform the method of any one of claims 1-11.

Images