Vector database index recommendation method, device, equipment and medium

Abstract

The present invention relates to the technical field of vector database retrieval, and discloses a vector database index recommendation method, apparatus, device, and medium. The method comprises: performing cluster analysis on a vector data set, extracting data distribution characteristics based on the clustering results, and obtaining a vector data portrait; the data distribution characteristics include the center distance between clusters and whether a head-tail structure exists; determining a recommended index based on the vector data portrait according to a preset index recommendation strategy: if the center distance between clusters exceeds the upper limit of a set threshold interval, the IVF index is used as a whole; if the center distance between clusters is within the set threshold interval and there is no head-tail structure, the IVF index is used as a whole; if there is a head-tail structure, the IVF index is used for the backbone data portion, and the HNSW index is used for the sparse data portion; if the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW. The present invention can adaptively recommend a suitable index type based on the data distribution characteristics within the vector data set.

Description

Technical Field

[0001] The present invention belongs to the technical field of vector database retrieval, and in particular relates to a vector database index recommendation method, device, equipment and medium. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the widespread application of vector databases in various fields, the engineering importance of their storage capacity has become increasingly significant. Storage performance, efficiency, and stability have become key considerations in software system design. Furthermore, with the widespread use of large language models (LLMs) and retrieval-augmented generation (RAG) technologies, high-dimensional vectors (such as 768 or 1024 dimensions) encoded by models like BERT have become the most common storage targets. To improve the efficiency and accuracy of retrieval of similar vectors, vector storage indexes have become a crucial component of vector database design.

[0004] The selection of index types in existing vector databases mostly relies on manual experience and static configuration. However, retrieval efficiency and accuracy are not only related to obvious characteristics such as data scale and data dimension, but also affected by characteristics such as the intrinsic structure of the data and query patterns. Therefore, it is difficult to cover all possible scenarios based on manual experience and static configuration. Summary of the Invention

[0005] In view of this, the present invention provides a vector database index recommendation method, apparatus, device and medium for adaptively recommending appropriate index types based on the internal data distribution characteristics of a vector dataset.

[0006] A first aspect of the present invention provides a vector database index recommendation method, comprising the following steps:

[0007] Perform cluster analysis on the vector data set, extract data distribution features based on the clustering results, and obtain a vector data portrait; the data distribution features include the center distance between clusters and whether there is a head-tail structure;

[0008] According to the preset index recommendation strategy, the recommended index is determined based on the vector data profile; the index recommendation strategy includes the strategy for selecting the index type under different data distribution characteristics:

[0009] If the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole;

[0010] If the center distance between clusters is within the set threshold range, if there is no head-tail structure, the IVF index is used as the whole; if there is a head-tail structure, the IVF index is used for the backbone data part and the HNSW index is used for the sparse data part;

[0011] If the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW.

[0012] In some embodiments, after performing cluster analysis on a vector dataset, the number of clusters in the clustering result is optimized based on the data distribution within each cluster:

[0013] The number of clusters is used as a variable, and its variation interval is set. Clustering operations are performed sequentially for multiple numbers of clusters within the variation interval, and the data distribution parameters within the cluster are calculated. The corrected value of the number of clusters is determined by the elbow method.

[0014] According to the correction value, the clustering operation is re-executed, the clustering result is tested for cluster distribution rationality, the number of data samples in each cluster is obtained, and the number of clusters is further adjusted according to the distribution of the number of samples to obtain the optimal number of clusters.

[0015] In some embodiments, the cluster distribution rationality test is used to test the balance of the number of data samples in each cluster, including the following test contents: whether there are too many low-density clusters, whether there are too many fragmented clusters, whether the overall distribution of data samples is unbalanced, and the separation between classes.

[0016] In some embodiments, when IVF indexing is recommended for the entire or partial vector dataset, if the amount of data recommended for IVF indexing exceeds a set threshold, the intra-cluster index is recommended to be FLAT index; otherwise, the intra-cluster index is recommended to be PQ index.

[0017] In some embodiments, the data distribution characteristics further include a covariance principal component ratio. If the covariance principal component ratio exceeds a set threshold, the vector data set is dimensionality reduced and then index recommendation is performed.

[0018] In some embodiments, the data distribution characteristics also include the proportion of noise points. If the proportion of noise points exceeds the upper limit of a set threshold interval, it is prompted that the vector data set needs to be cleaned; if the proportion of noise points is within the set threshold interval, it is recommended to use IVF indexing for the backbone data part and HNSW indexing for the sparse data part; if the proportion of noise points is lower than the lower limit of the set threshold, it is prompted to remove the noise vector before making index recommendations.

[0019] In some embodiments, the method further includes: performing index construction on the vector dataset based on the recommended index, and evaluating the current index type according to an index construction cost indicator and a query performance indicator.

[0020] A second aspect of the present invention provides a vector database index recommendation device, comprising:

[0021] The feature extraction module is configured to: perform cluster analysis on the vector data set, extract data distribution features based on the clustering results, and obtain a vector data portrait; the data distribution features include the center distance between clusters and whether there is a head-tail structure;

[0022] The index recommendation module is configured to determine a recommended index based on the vector data portrait according to a preset index recommendation strategy; the index recommendation strategy includes:

[0023] If the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole;

[0024] If the center distance between clusters is within the set threshold range, if there is no head-tail structure, the IVF index is used as the whole; if there is a head-tail structure, the IVF index is used for the backbone data part and the HNSW index is used for the sparse data part;

[0025] If the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW.

[0026] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method described.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the program is executed by a processor.

[0028] In one or more of the above technical solutions, by constructing an index recommendation strategy, it is possible to adaptively recommend a suitable index type based on the data distribution characteristics within the vector dataset.

[0029] In addition, the recommended index type can also be evaluated based on the index creation cost and query performance, thereby guiding the optimization of the index type and its parameters during the construction process, balancing the query load and system resource constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 A flow chart of a vector database index recommendation method provided by an embodiment of the present invention is shown;

[0032] Figure 2 A flowchart of constructing a vector data portrait provided by an embodiment of the present invention is shown;

[0033] Figure 3It shows a program module diagram of a vector database index recommendation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0035] In the description of the embodiments of the present application, the term “including” and similar terms should be understood as open inclusion, that is, “including but not limited to.” The term “based on” should be understood as “at least partially based on.”

[0036] The data distribution features involved in one or more of the following embodiments include the number of clusters, the average distance within a cluster, the center distance between clusters, the proportion of noise points, and the proportion of covariance principal components. The definition of each data distribution feature and its use in the specific implementation are shown in Table 1.

[0037] Table 1 Description of data distribution characteristics

[0038]

[0039] (1) Average distance within a cluster

[0040] After clustering a vector dataset, the data distribution is evaluated primarily through dimensions such as intra-cluster compactness and the Silhouette Score. Intra-cluster compactness is used to assess the distribution of data samples within a cluster, specifically to measure the sparseness or density of data within each cluster, and whether a cluster center has too many or too few data samples. Intra-cluster average distance is a linear indicator of intra-cluster compactness and is strongly correlated with the clustering cost function (Inertia). Therefore, the clustering cost function is used to measure the intra-cluster average distance.

[0041] The clustering cost function is a measure of clustering compactness and is defined as the sum of the squared distances of all sample points to the center of their cluster:

[0042]

[0043] Among them, k represents the number of clusters (i.e., candidate nlist), represents the i-th cluster, represents the center of the i-th cluster, Represents the sample vector belonging to this cluster.

[0044] (2) Inter-cluster center distance

[0045] The silhouette coefficient is used to measure the degree of intra-class aggregation and inter-class separation of each sample. The nearest neighbor cluster distance term in the silhouette coefficient is closely related to the inter-cluster center distance, so the silhouette coefficient is used to measure the inter-cluster center distance. The calculation formula is:

[0046]

[0047] in, represents the average distance between sample i and other samples in the same cluster, represents the average distance between sample i and the samples in the nearest neighbor cluster, It represents the silhouette coefficient of the ith sample, and the center distance between clusters is represented by the average silhouette coefficient of all samples.

[0048] (3) Noise point ratio

[0049] Noise points come from various sources, such as blank rows, pure numeric columns, punctuation rows in table segmentation, UUIDs and timestamps in business logs, etc. They do not have direct semantic information.

[0050] DBSCAN is used to detect noise points. Based on the density accessibility theory, the neighborhood radius is set. With minimum points This method identifies high-density areas and noise samples in the semantic space. Based on the proportion of samples marked as noise points in the clustering results, the noise point ratio (noise_rate) is calculated, and recommendations such as noise removal and fallback are determined accordingly. This effectively prevents formatting anomalies from interfering with the index structure, improving the quality of subsequent clustering structures and the stability of index compression.

[0051] For example, the neighborhood radius Set to 0.3, the minimum number of points Set to 5.

[0052] (4) Covariance principal component ratio

[0053] The GMM model is used to fit the vector set distribution, the eigenvalue decomposition covariance matrix is ​​used to extract the explanatory power of the principal components of each dimension, and the cumulative variance ratio is calculated to determine whether there is a compressible main direction.

[0054] The vector set is , GMM models it as A mixture of Gaussian distributions:

[0055]

[0056] in, Indicates the The weights of the Gaussian components satisfy ; Indicates the The mean vector of the components; Indicates the The covariance matrix of the components; Represents a multidimensional Gaussian probability density function.

[0057] For each covariance moment The matrix performs eigenvalue decomposition, and the system constructs the global cumulative explained variance ratio based on the principal component eigenvalues ​​of all components:

[0058]

[0059] at last, Indicates that the principal component retains dimension; Represents the original vector dimension; Indicates the The numerator represents the dimension of the i-th principal component in the Gaussian components. The denominator represents the total variance.

[0060] For example, Set to 20, the principal component retains the dimension is 256.

[0061] The index types involved in one or more of the following embodiments include: IVF, HNSW, PQ, Flat, IVF_FLAT, IVF_PQ, IVF_FLOAT, etc. Table 2 shows the meanings of these index types.

[0062] Table 2 Index types and their meanings

[0063]

[0064] As mentioned in the background technology, the selection of index types in existing vector databases mostly relies on manual experience and static configuration. Although manual experience has certain reference value, it is often difficult to cover all possible scenarios. If the selected index type is not suitable for the current data characteristics or query mode, it may lead to a significant decrease in retrieval efficiency.

[0065] Figure 1 A flow chart of a vector database index recommendation method provided by an embodiment of the present invention is shown, and the method includes the following steps:

[0066] S110: Perform cluster analysis on the vector dataset, extract data distribution characteristics based on the clustering results, and obtain a vector data profile. The goal of cluster analysis is to divide the data into multiple clusters, minimizing the distance within the clusters to facilitate the extraction of data distribution structure characteristics. Specifically, commonly used cluster analysis methods include K-Means clustering, hierarchical clustering, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), and Gaussian Mixture Model (GMM). You can select one or more of these methods based on the characteristics of the vector dataset.

[0067] S120: According to the preset index recommendation strategy, the recommended index is determined according to the vector data portrait. The preset index recommendation strategy has two implementation methods: rule engine (Rule-Based) and algorithm model (ML-Based). Among them, the rule engine maps the input features based on a set of manually set empirical rules. The algorithm model uses a lightweight classifier (such as XGBoost) to learn the relationship between features and index strategies. As an example of a rule engine, the index recommendation strategy includes a strategy for selecting index types under different data distribution characteristics: if the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole; if the center distance between clusters is within the set threshold interval, if there is no head-tail structure, the IVF index is used as a whole; if there is a head-tail structure, the IVF index is used for the backbone data part and the HNSW index is used for the sparse data part; if the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW.

[0068] Based on the above steps, the clustering effect of the data can be measured by the center distance between clusters, which can then be used as a measure of the distribution balance of the overall data in the vector database, and indexes can be adaptively recommended to ensure retrieval efficiency.

[0069] S130: Build an index for the vector dataset based on the recommended index, and evaluate the current index type based on index build cost metrics and query performance metrics. Index cost metrics include index build time and index size, while query performance metrics include recall rate and query latency. Based on the evaluation results, it is determined whether further optimization of the index type is necessary.

[0070] Based on Figure 1 In an optional embodiment, as Figure 2 As shown, the above step S110 can be implemented as steps S111 to S113.

[0071] S111: Perform cluster analysis on the vector dataset based on unsupervised cluster analysis. For example, the unsupervised cluster analysis uses the Kmean algorithm. The purpose of cluster analysis is to divide the index (IVF) partitions. The key is to divide the appropriate clusters according to the data distribution characteristics so that the content can be quickly located when performing vector retrieval. Generally speaking, the richer the data sample volume, the more partitions should be. The relationship between the total data sample volume and the number of initial clusters can be constructed, for example, defined as , where N represents the total data sample size. Of course, the default initial cluster number can also be specified for different levels of sample size. Table 3 shows the default configuration of the number of clusters when the Kmean algorithm clusters with different levels of sample size.

[0072] Table 3 Default configuration table of the number of clusters

[0073]

[0074] S112: Optimize the number of clusters in the clustering results based on the data distribution within each cluster. The above steps determine the initial number of clusters based on the principle that the richer the data sample size, the more partitions should be. However, in some scenarios, there are situations where the sample size is large but the distribution characteristics are not obvious. Therefore, the initial number of clusters is not necessarily the optimal number of clusters. The number of clusters needs to be optimized based on the distribution of data within the cluster and the distribution of the number of data samples within each cluster.

[0075] As a specific implementation method, first, the number of clusters is used as a variable and its variation interval is set. Clustering operations are performed in sequence for multiple numbers of clusters within the variation interval, and the data distribution parameters within the cluster are calculated. The corrected value of the number of clusters is determined by the elbow method. Then, based on the corrected value, the clustering operation is re-executed, and the rationality of the cluster distribution is tested on the clustering results. The number of data samples in each cluster is obtained. According to the distribution of the sample number, the number of clusters is further adjusted to obtain the optimal number of clusters.

[0076] Specifically, the distribution of data within a cluster is measured using the clustering cost function (Inertia), which represents the sum of the squared distances from all data points to their cluster centers. A smaller Inertia indicates that the data points are more closely clustered at the cluster center, and the better the clustering effect. The elbow method works as follows: as k increases, Inertia gradually decreases. However, when k exceeds a certain value, the rate of decline in Inertia decreases significantly. This point is the corrected value for the number of clusters. For example, a range of cluster number variations is set, and the range is traversed in 12% increments. The Inertia after clustering each cluster number is recorded, and its downward trend is analyzed using the elbow method. If the Inertia for the current number of clusters decreases by less than 12% compared to the initial value or the limit value, convergence is considered, and the current number of clusters, k, can be used as the corrected value for the number of clusters. It is understood that the number of clusters should be an integer value, and rounding is performed after calculating the number of clusters using the set increments.

[0077] The cluster distribution rationality test primarily examines the balance of data sample numbers within each cluster. Imbalanced data sample numbers within clusters can be manifested by: an excessive number of low-density clusters, an excessive number of fragmented clusters, an overall imbalance in data sample distribution, and poor separation between clusters. If at least one of these issues exists, the current number of clusters is considered too large and should be reduced. The number of clusters is reduced by a set increment (e.g., 12%), and the clustering operation is repeated until the cluster distribution meets the requirements for rationality.

[0078] For example, if more than 5% of the clusters in the total samples have less than 10 data samples, it is considered that there are too many low-density clusters; if the number of samples in the smallest cluster is less than 10% of the average number of samples in each cluster, it is considered that there are fragmented clusters. Specifically, if the number of samples in a cluster is 5 and the overall average number of samples per cluster is 90, then the cluster is a fragmented cluster.

[0079] The overall imbalance in the distribution of data samples is measured by the ratio of the number of data samples in the largest cluster to the number of data samples in the smallest cluster. If the ratio of the largest cluster to the smallest cluster samples exceeds 10 times, it is determined that the sample distribution is imbalanced.

[0080] Structural separability is assessed using the distance between cluster centers. Specifically, the distance between cluster centers is measured based on the silhouette coefficient, which ranges from -1 to 1. Values ​​closer to 1 indicate higher separation between clusters and better clustering results, while values ​​closer to -1 indicate lower separation between clusters and poorer clustering results. Therefore, when the silhouette coefficient falls below a set threshold, structural separability is considered unsatisfactory.

[0081] Based on this, the optimal number of clusters, that is, the size of nlist in the IVF index, is obtained, which meets the conditions for constructing the IVF series index.

[0082] S113: Perform cluster analysis based on the optimized number of clusters, extract data distribution features based on the clustering results, and obtain a vector data profile. For example, the data distribution features include the inter-cluster center distance (inter_dist), the noise point ratio (noise_rate), the covariance principal component ratio (var_ratio), and whether there is a head-tail structure. Based on the above multiple data distribution features VDataProfile.

[0083] Head-tail structure identification is used to determine whether data samples have structural skew. The number of data samples in each cluster is obtained by counting and sorting, and the "head-tail structure" in the vector set is identified: if the top 10% of clusters have covered 50% of the total samples, it means that the head clustering is strong; if the number of samples in more than 20% of clusters is less than 20, or the proportion of single sample clusters exceeds 10%, it indicates that there is a significant long-tail structure. In both cases, the above two situations are considered to have a head-tail structure.

[0084] In step S120, by summarizing the impact of different data features on indexing, it can be found that: when clustering is effective, IVF indexing is superior to HNSW indexing; when vector distribution is dense, that is, the average intra-cluster distance is small, this situation itself will affect retrieval accuracy. If PQ indexing is used, it will cause information loss, which will further aggravate the impact on retrieval accuracy; if data is evenly distributed within each cluster and the differences between clusters are small, HNSW indexing can more effectively utilize its hierarchical graph structure for navigation; when the proportion of principal components of intra-cluster covariance is high, it indicates that the data has obvious correlations or trends in certain directions. In this case, using L2 distance (Euclidean distance) as a metric may be more appropriate than using cosine similarity. This is because L2 distance can better capture data differences in these directions; when the curse of dimensionality exists, reducing the dimension through principal component analysis (PCA) before building the index can improve retrieval efficiency.

[0085] Based on this, we can draw the following recommended strategies: When data exhibits a clear clustering structure in vector space, a center-partitioning index (such as IVF_FLAT or IVF_PQ) is suitable. For evenly distributed data, the small-world graph (HNSW) structure is more effective due to its jump-ability. If the data has a long tail but good local clustering, IVF_FLAT / PQ+HNSW can achieve a good balance. When the dimensionality is very high (e.g., d > 2048), PCA dimensionality reduction followed by IVF_Flat indexing can avoid the degradation of the graph caused by high-dimensional sparsity while maintaining accuracy.

[0086] The above step S120 can be implemented as steps S121 to S124.

[0087] S121: for the vector data set, respectively calculating the inter-cluster center distance, the covariance principal component ratio, and the noise point ratio, and determining whether there is a head-tail structure;

[0088] S122: Determine whether the proportion of the covariance principal component exceeds a set threshold. If so, perform dimensionality reduction on the vector data set. If not, directly execute step S123;

[0089] S123: Recommend index type based on inter-cluster center distance:

[0090] If the center distance between clusters exceeds the upper limit of the set threshold interval, the clustering effect is considered good and the IVF index is recommended as the overall index. Furthermore, based on the data scale, it is recommended whether the intra-cluster index should be a FLAT index or a PQ index: if the number of vectors in the vector dataset exceeds the set threshold, the FLAT index is recommended for the intra-cluster index; otherwise, the PQ index is recommended for the intra-cluster index.

[0091] If the center distance between clusters is within the set threshold range, the clustering effect is considered average. If there is a head-tail structure, the IVF index is recommended for the backbone data part, and the HNSW index is recommended for the sparse data part. In addition, based on the data scale of the backbone data part, the FLAT index or PQ index is recommended for the cluster index. The recommendation idea is the same as above. If there is no head-tail structure, the IVF index is recommended for the cluster. Further, based on the data scale, the FLAT index or PQ index is recommended for the cluster index. The recommendation idea is the same as above.

[0092] If the center distance between clusters is less than the lower limit of the set threshold interval, the clustering effect is considered poor and the recommended index is HNSW.

[0093] S124: If the proportion of noise points exceeds the upper limit of the set threshold range, it is prompted that the vector data set needs to be cleaned; if the proportion of noise points is within the set threshold range, it is recommended to use IVF indexing for the backbone data part and HNSW indexing for the sparse data part; if the proportion of noise points is lower than the lower limit of the set threshold, it is prompted to remove the noise vector before indexing.

[0094] Exemplarily, the above step S120 specifically includes:

[0095] (1) Determine whether the average silhouette coefficient S satisfies S ≥0.45, if yes, go to step (2), if no, go to step (3);

[0096] (2) Use IVF index as a whole; determine whether the proportion of the principal component of the covariance satisfies var_ratio ≥ 0.9. If so, use PCA to reduce the dimension to 256 or 128; if var_ratio < 0.90, do not reduce the dimension; further select the IVF type based on the data scale and compression requirements: if the number of vectors is < 1 million, IVF_FLAT is recommended; if the number of vectors is ≥ 1 million, IVF_PQ is recommended.

[0097] (3) If 0.30≤ S <0.45 indicates a moderate clustering structure. In this case, if there is a head-tail structure, IVF_FLAT or IVF_PQ is used for the backbone data, and HNSW index is used for the sparse data. If there is no head-tail structure, IVF_FLAT or IVF_PQ is recommended based on the data size.

[0098] (4) If S <0.30, indicating that the clusters are inseparable or overlap, and the recommended index is HNSW, using the default configuration;

[0099] (5) If noise_rate ≥ 0.20, it is considered that this data sample needs to be cleaned or manually intervened before it can be used; if noise_rate ∈ [0.10, 0.20), the backbone data part uses IVF indexing and the sparse data part uses HNSW indexing; if noise_rate < 0.10, the noise vector is removed before indexing.

[0100] 100,000, 1,000,000, and 10,000,000 samples were selected for the experiment. The sample data included documents, images, videos, and other modalities, with feature dimensions typically ranging from 768 to 1024, which is different from common recommendation systems (which have dimensions ranging from 32 to 256). For example, for a sample of 1,000,000 text vectors, the text vector data dimension was 768 (this dimension is typically used as the encoding dimension for the Chinese text model Bge), and the initial number of clusters k = 1024. Based on the processing flow of step S110, a Kmeans clustering analysis was performed on the vector dataset based on the initial number of clusters, extracting data distribution features to optimize the number of clusters.

[0101] After optimizing the number of clusters, the data distribution characteristics were calculated again. The recommended index was determined based on the preset index recommendation strategy. The recommendation process is as follows: (1) Calculate the data distribution characteristics: S = 0.47, indicating a clear clustering structure; overall balance, but with a significant tail; (2) var_ratio = 91.3, first use PCA compression to reduce the dimension to 256; (3) The proportion of noise points is low: noise_rate = 3%, directly remove the noise vector. The final recommended index type is IVF_PQ index for the backbone data part and HNSW index for the sparse data part.

[0102] The recommendation process uses the distance between cluster centers as a criterion, employing different judgment logics for different clustering effects. For example, when clustering is effective, IVF is used overall, and dimensionality reduction is selected based on the principal component distribution. Different construction strategies, such as FLAT and PQ, are recommended for different data sizes. When clustering is average, index recommendations are made based on the presence of a head-tail structure, while the default index is used when clustering is poor. This enables personalized recommendations based on the data size, dimensionality, and internal structure distribution characteristics of the vector database. The judgment process also involves dimensionality reduction and noise point ratio analysis. Based on this, the vector database storage is optimized, improving sample quality.

[0103] In step S130, index construction time and index size are used as index cost indicators, while recall rate and average query latency are used as query performance indicators. Index construction time measures the efficiency of constructing the vector database under different configurations, index size determines the cost of index deployment, recall rate measures query accuracy, and if dimensionality reduction was performed on the vector database in step S120, recall rate can also be used to measure over-compression. Average query latency measures query efficiency.

[0104] When the index construction time exceeds the set threshold, one or more of the following optimization strategies can be selected based on the size of the data: (1) reduce the number of clusters; (2) if IVF and other index types are currently used, adjust the index type within the cluster; (3) reduce the dimensionality of the vector dataset when the proportion of the covariance principal component meets the set standard.

[0105] When the index size exceeds the set threshold, one or more of the following optimization strategies can be selected based on the data size: (1) enable the PQ index type; (2) reduce the dimensionality of the vector dataset when the proportion of the covariance principal component meets the set standard.

[0106] When the recall rate is lower than the set threshold, one or more of the following optimization strategies can be selected based on the data size: (1) increase the number of clusters; (2) if IVF and other index types are currently used, adjust the index type within the cluster, giving priority to the Float index type; (3) cancel dimensionality reduction; (4) disable the PQ index type.

[0107] When the average query latency exceeds the set threshold, one or more of the following optimization strategies can be selected based on the data size: (1) If IVF and other index types are currently used, replace the intra-cluster index type with FLAT; (2) Reduce the dimension of the vector dataset when the covariance principal component ratio meets the set standard; (3) Reduce the number of clusters and limit the number of concurrent threads during query; (4) Use HNSW.

[0108] For different data scales, when different index cost indicators and query performance indicators exceed the upper or lower limits, the specific index callback optimization strategies are shown in Table 4.

[0109] Table 4 Index callback optimization strategies under different data scales

[0110]

[0111] One or more of the above embodiments provide methods for performing index recommendations based on the data distribution characteristics in the vector database and for reverse verification through performance indicators. However, the above methods still build indexes for the current data set stored in the vector database and cannot be dynamically adjusted according to changes in data or changes in query patterns. Based on this, the vector database is partitioned, and after the vector database is indexed based on the recommended index type, incremental data is received and stored on separate shards. When the scale of incremental data exceeds the set standard, the above index recommendation method is re-executed for the vector database, and the vector database is re-indexed. The setting standard for the scale of incremental data can be the proportion of the total amount (for example, the incremental amount exceeds 15% of the total amount), or the number or size of incremental data samples, etc., which are not specifically limited here.

[0112] Based on the above method, one or more embodiments of the present invention further provide a vector database index recommendation device, such as Figure 3 As shown, it includes: a feature extraction module 401, which is configured to: perform cluster analysis on the vector data set, extract data distribution features based on the clustering results, and obtain a vector data portrait; the data distribution features include the center distance between clusters and whether there is a head-tail structure; an index recommendation module 402, which is configured to: determine the recommended index according to the vector data portrait based on a preset index recommendation strategy; the index recommendation strategy includes: if the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole; if the center distance between clusters is within the set threshold interval, if there is no head-tail structure, the IVF index is used as a whole; if there is a head-tail structure, the IVF index is used for the backbone data part, and the HNSW index is used for the sparse data part; if the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW.

[0113] In order to verify and optimize the recommended index type, the device also includes an index evaluation module 403, which is configured to: perform index construction on the vector dataset based on the recommended index, and evaluate the current index type according to the index construction cost index and the query performance index.

[0114] One or more embodiments of the present invention further provide an electronic device that can be used to implement the method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0115] One or more embodiments of the present invention further provide an electronic device that can be used to implement the method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0116] The memory in the embodiment of the present invention is used to store various types of data to support Figure 1 Implementation of the method shown in .

[0117] It is understood that the memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The memory in the embodiment of the present invention can store the following: Figure 1 The computer programs corresponding to the steps in the method shown in . The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and handle hardware-based tasks. The application program can include various application programs.

[0118] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0119] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion and / or installed from a removable medium. When the computer program is executed by the central processing unit, the various functions defined in the apparatus of the present application are performed.

[0120] in, Figure 1 The computer program instructions corresponding to the method shown can also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A vector database index recommendation method, characterized in that: The following steps are involved: Perform cluster analysis on vector data sets, extract data distribution characteristics based on clustering results, and obtain vector data portraits; The data distribution characteristics include the center distance between clusters and whether there is a head-tail structure; According to the preset index recommendation strategy, the recommended index is determined based on the vector data portrait; The index recommendation strategy includes: If the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole; If the center distance between clusters is within the set threshold range, if there is no head-tail structure, the IVF index is used as the whole; if there is a head-tail structure, the IVF index is used for the backbone data part and the HNSW index is used for the sparse data part; If the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW; When IVF index is recommended for the entire or part of a vector dataset, if the amount of data recommended for IVF index exceeds the set threshold, FLAT index is recommended for the intra-cluster index; otherwise, PQ index is recommended for the intra-cluster index. Head-tail structure identification is used to determine whether data samples have structural skew. The number of data samples in each cluster is counted and then sorted to identify the "head-tail structure" in the vector set: if the top 10% of clusters cover 50% of the total samples, it indicates strong head clustering; if the number of samples in more than 20% of clusters is less than 20, or the proportion of single sample clusters exceeds 10%, it indicates the presence of a significant long-tail structure. In both cases, head-tail structure is considered to exist.

2. The vector database index recommendation method according to claim 1, wherein: After performing cluster analysis on the vector dataset, the number of clusters in the clustering results is optimized based on the data distribution within each cluster: The number of clusters is used as a variable, and its variation interval is set. Clustering operations are performed sequentially for multiple numbers of clusters within the variation interval, and the data distribution parameters within the cluster are calculated. The corrected value of the number of clusters is determined by the elbow method. According to the correction value, the clustering operation is re-executed, the clustering result is tested for cluster distribution rationality, the number of data samples in each cluster is obtained, and the number of clusters is further adjusted according to the distribution of the number of samples to obtain the optimal number of clusters.

3. The vector database index recommendation method according to claim 2, wherein: The cluster distribution rationality test is used to test the balance of the number of data samples in each cluster, including the following test contents: whether there are too many low-density clusters, whether there are too many fragmented clusters, whether the overall distribution of data samples is unbalanced, and the separation between classes.

4. The vector database index recommendation method according to claim 1, wherein: The data distribution characteristics also include the proportion of covariance principal components. If the proportion of covariance principal components exceeds a set threshold, the vector data set is reduced in dimension and then index recommendation is performed.

5. The vector database index recommendation method according to claim 1, wherein: The data distribution characteristics also include the proportion of noise points. If the proportion of noise points exceeds the upper limit of the set threshold range, it is prompted that the vector data set needs to be cleaned; if the proportion of noise points is within the set threshold range, it is recommended to use IVF indexing for the backbone data part and HNSW indexing for the sparse data part; if the proportion of noise points is lower than the lower limit of the set threshold, it is prompted to remove the noise vector before making index recommendations.

6. The vector database index recommendation method according to claim 1, wherein: The method further includes: executing index construction on the vector dataset based on the recommended index, and evaluating the current index type according to an index construction cost index and a query performance index.

7. A vector database index recommendation device, characterized in that: include: The feature extraction module is configured to: perform cluster analysis on the vector data set, extract data distribution features based on the clustering results, and obtain a vector data portrait; The data distribution characteristics include the center distance between clusters and whether there is a head-tail structure; The index recommendation module is configured to determine a recommended index based on the vector data portrait according to a preset index recommendation strategy; the index recommendation strategy includes: If the center distance between clusters exceeds the upper limit of the set threshold interval, the IVF index is used as a whole; If the center distance between clusters is within the set threshold range, if there is no head-tail structure, the IVF index is used as the whole; if there is a head-tail structure, the IVF index is used for the backbone data part and the HNSW index is used for the sparse data part; If the center distance between clusters is less than the lower limit of the set threshold interval, the recommended index is HNSW; When IVF index is recommended for the entire or part of a vector dataset, if the amount of data recommended for IVF index exceeds the set threshold, FLAT index is recommended for the intra-cluster index; otherwise, PQ index is recommended for the intra-cluster index. Head-tail structure identification is used to determine whether data samples have structural skew. The number of data samples in each cluster is counted and then sorted to identify the "head-tail structure" in the vector set: if the top 10% of clusters cover 50% of the total samples, it indicates strong head clustering; if the number of samples in more than 20% of clusters is less than 20, or the proportion of single sample clusters exceeds 10%, it indicates the presence of a significant long-tail structure. In both cases, head-tail structure is considered to exist.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein computer instructions are stored in the memory. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images