Vector data similarity query method, device, equipment and medium

Abstract

The application provides a vector data similarity query method, device, equipment and medium, the method comprises the steps of: dividing a plurality of vector data to be queried into a plurality of data buckets, and storing the vector data included in each data bucket into the corresponding storage area respectively; determining the neighbor data bucket having an association relationship with each data bucket, constructing a directed edge between each data bucket and its corresponding neighbor data bucket, and generating a data bucket dependency graph; determining the processing order when writing the plurality of data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache; sequentially writing each data bucket and its corresponding neighbor data bucket from the storage area into the target cache according to the processing order, and determining the distance between two vector data respectively belonging to each data bucket and its corresponding neighbor data bucket, and screening out the vector pair with a distance less than or equal to a preset threshold from the plurality of vector data. In this way, I / O waste is effectively reduced, and operation and maintenance and hardware investment are reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and medium for vector data similarity query. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, processing massive vector data has become a core processing method. Similarity Self-Join (SSJ) is a fundamental and crucial operation. SSJ can identify all vector pairs in a massive dataset whose distance is less than or equal to a preset threshold. This operation plays a vital role in tasks such as large-scale semantic deduplication and outlier detection. For example, when training large language models, performing semantic deduplication on the original dataset containing billions of samples can effectively improve model training efficiency and performance.

[0003] When performing SSJ, data read and write operations are often required, resulting in a large amount of wasted input / output (I / O) and high investment in maintenance and hardware. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device and medium for vector data similarity query, which performs batch processing of vector data in the form of data buckets and performs targeted sorting, effectively reducing I / O waste and reducing operation and maintenance and hardware investment.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of this application, a vector data similarity query method is provided, the method comprising:

[0007] The multiple vector data to be queried are divided into multiple data buckets, and the vector data included in each data bucket is stored in the corresponding storage area. Each data bucket includes at least one of the vector data.

[0008] Identify the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each data bucket and its corresponding neighboring data buckets, and generate a data bucket dependency graph;

[0009] Based on the data bucket dependency graph and the capacity of the target cache, determine the processing order when writing the multiple data buckets into the target cache;

[0010] According to the processing order, each data bucket and its corresponding neighbor data bucket are written from the storage area to the target cache in sequence, and the distance between two vector data belonging to each data bucket and its corresponding neighbor data bucket is determined. Vector pairs with a distance less than or equal to a preset threshold are selected from the multiple vector data.

[0011] In one optional implementation, dividing the multiple vector data to be queried into multiple data buckets includes:

[0012] Multiple target vector data are determined from multiple vector data to be queried according to a preset sampling ratio, and each target vector data serves as the center of a data bucket;

[0013] For each vector data other than the target vector data among the plurality of vector data, the vector data is assigned to the data bucket to which the target vector data that is closest to the vector data belongs.

[0014] In one optional implementation, storing the vector data included in each of the data buckets into corresponding storage areas includes:

[0015] Set a corresponding buffer for each of the data buckets;

[0016] For each data bucket, the vector data allocated to that data bucket is loaded into the corresponding buffer. When the buffer is saturated, the vector data in the buffer is written into the storage area corresponding to that data bucket.

[0017] In one optional implementation, determining the neighboring data buckets associated with each of the data buckets includes:

[0018] For each data bucket, a preset number of data buckets that are closest to the center of the data bucket are determined from other data buckets to form an initial set of associated data buckets for that data bucket. The center distance is used to indicate the distance between the centers of two data buckets.

[0019] Remove the initial associated data buckets whose bucket center does not satisfy the triangle inequality relationship with the bucket center of the data bucket from the initial associated data bucket set to obtain the candidate associated data bucket set of the data bucket.

[0020] Based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set, the neighbor data bucket is selected from the candidate associated data bucket set. The recall loss ratio is used to indicate the loss of recall ratio when pruning the candidate associated data bucket.

[0021] In one optional implementation, the initial associated data buckets that do not satisfy the triangle inequality relationship with the data bucket are determined through the following steps:

[0022] For any of the initial associated data buckets, if the center distance between the initial associated data bucket and the data bucket is greater than the preset threshold and the sum of the bucket radius of the initial associated data bucket and the bucket radius of the data bucket, it is determined that the initial associated data bucket and the data bucket do not satisfy the triangle inequality relationship.

[0023] In one optional implementation, the recall loss ratio of each candidate associated data bucket is determined through the following steps:

[0024] For each of the initial associated data buckets, the intersection volume between the sphere corresponding to the initial associated data bucket and the sphere corresponding to the data bucket is determined. The sphere corresponding to the initial associated data bucket has its center as the bucket center and its radius as the bucket radius. The sphere corresponding to the data bucket has its center as the bucket center and its radius as the sum of the radius of the data bucket and the preset threshold.

[0025] Based on the intersection volume corresponding to the initial associated data bucket and the sum of the intersection volumes corresponding to each initial associated data bucket, the recall loss ratio of the candidate associated data bucket is determined.

[0026] In one optional implementation, the step of selecting the neighbor data bucket from the candidate associated data bucket set based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set includes:

[0027] In descending order of distance from the center of the data bucket, the recall rate loss ratios of each candidate associated data bucket in the candidate associated data bucket set are sequentially accumulated until the sum of the accumulated recall rate loss ratios reaches the preset recall rate loss.

[0028] The candidate associated data buckets accumulated when the preset recall rate loss is reached are deleted from the candidate associated data bucket set to obtain the neighbor data bucket.

[0029] In one optional implementation, determining the processing order for writing the plurality of data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache includes:

[0030] The target quantity is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache;

[0031] The data bucket with the largest out-degree among the multiple data buckets is used as the first data bucket to be written into the target cache;

[0032] Determine the correlation index of each of the other data buckets to be sorted, and take the data bucket with the largest correlation index as the second data bucket to be written into the target cache. The correlation index is used to indicate the number of common neighbor data buckets between the other data buckets to be sorted and the next target number of data buckets in the currently sorted data buckets.

[0033] Repeat the step of determining the association index of each of the other data buckets to be sorted until all the data buckets are sorted, and obtain the processing order when writing the data buckets to the target cache.

[0034] In one optional implementation, the method further includes:

[0035] During the process of writing each data bucket and its corresponding neighboring data bucket from the storage area to the target cache in the order described in the processing order, if the target cache is saturated and not all data buckets have been written to the target cache, the data bucket with the latest next write time in the target cache is deleted, and the data bucket writing continues in the order described in the processing order.

[0036] According to a second aspect of this application, a vector data similarity query apparatus is provided, the apparatus comprising:

[0037] The vector bucketing module is used to divide multiple vector data to be queried into multiple data buckets, and store the vector data included in each data bucket into the corresponding storage area. Each data bucket includes at least one vector data.

[0038] The association construction module is used to determine the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each of the data buckets and its corresponding neighboring data buckets, and generate a data bucket dependency graph.

[0039] The task orchestration module is used to determine the processing order of writing the multiple data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache;

[0040] The task execution module is used to write each of the data buckets and their corresponding neighbor data buckets from the storage area to the target cache in the order of processing, and to determine the distance between two vector data belonging to each of the data buckets and their corresponding neighbor data buckets, and to filter out vector pairs with a distance less than or equal to a preset threshold from the multiple vector data.

[0041] According to a third aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the vector data similarity query method described in the first aspect above.

[0042] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the vector data similarity query method described in the first aspect.

[0043] The vector data similarity query method, apparatus, device, and medium provided in this application divide multiple vector data into data buckets to achieve batch processing and targeted sorting, significantly improving local data consistency. This allows read and write operations to replace random access with continuous batch processing, thereby greatly reducing I / O overhead and improving single-machine performance. By combining neighbor data buckets to generate a bucket dependency graph and determining the processing order when writing multiple data buckets to the target cache, the storage layout is dynamically optimized, further reducing random I / O, enhancing adaptability and throughput performance. While ensuring high recall, it effectively reduces memory usage, breaks through capacity bottlenecks, and helps overcome the shortcomings of dynamic updates and disk I / O optimization, significantly reducing operation and maintenance and hardware investment.

[0044] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0045] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a vector data similarity query method in an exemplary embodiment of this application;

[0047] Figure 2 This is a schematic diagram illustrating a process for determining neighbor data buckets, as shown in an exemplary embodiment of this application;

[0048] Figure 3 This is a schematic diagram illustrating a vector data similarity query process according to an exemplary embodiment of this application;

[0049] Figure 4 This is a schematic diagram of a vector data similarity query device shown in an exemplary embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0053] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0054] Research has shown that the goal of SSJ is to find all vector pairs in a large-scale vector dataset whose distance is less than or equal to a preset threshold. Operationally, SSJ can be understood as performing an Approximate Nearest Neighbor (ANN) search on every vector in the dataset. Specifically, for each vector in the dataset, it is used as the query vector, and its neighbors (i.e., all other vectors whose distance to the query vector is less than or equal to the preset threshold) are searched throughout the dataset. Finally, all the found vector pairs that meet the criteria are aggregated and output. Therefore, SSJ essentially extends a single ANN query task into a batch-based global scan and matching process covering the entire dataset.

[0055] When performing SSJ, methods such as ε-grid methods, distributed similarity connections, or disk-based vector similarity search are often used.

[0056] Ec-grid methods are a class of techniques specifically designed for connecting high-dimensional vectors based on similarity. The core idea is to divide the entire vector space into regular grid cells of size ε. The algorithm first preprocesses all vectors in the dataset, mapping them to their respective grid cells, and then sorts and groups the vectors according to their grid coordinates. When performing joins, it utilizes a key filtering principle: only vectors located in adjacent or identical grid cells can potentially become ε-neighbors. Therefore, the algorithm only needs to examine vector pairs within these "nearby" grid cells, thus avoiding a brute-force scan of the entire dataset.

[0057] In distributed similarity joins, cluster joins, for example, represent a typical scheme for handling large-scale similarity joins using a distributed computing framework. It first selects a set of centroids and partitions the dataset based on these centroids, distributing the data across multiple computing nodes. On each node, candidate pairs are filtered out. Subsequently, a reducer phase is used to validate these cross-partition candidate pairs.

[0058] In disk-based vector similarity search, Disk-Optimized Approximate Nearest Neighbor Search (DiskANN) is an advanced billion-level vector retrieval system designed for solid-state disk (SSD) storage. It employs a hybrid memory-SSD architecture. Specifically, it maintains a proximity graph index composed of compressed vectors in memory for fast query routing; while the complete, high-precision vectors are stored on the SSD. When a query is received, the system first traverses the memory graph to find a set of candidate neighbors, then loads the full-precision vectors of these candidate neighbors from the SSD for final accurate distance calculation and result refinement.

[0059] However, the ε-grid method suffers from shortcomings in scalability and efficiency. As the vector dimension increases, the number of grids required grows exponentially, resulting in a large number of empty grids and extremely low space utilization. This not only incurs huge memory overhead but also makes grid-based sorting and grouping inefficient. More importantly, during execution, to verify potential neighbors across grids, it is often necessary to repeatedly read data blocks from different grids from disk, leading to severe issues of duplicate data access and caching incompatibility, making it difficult to efficiently apply to modern large-scale, high-dimensional vector datasets.

[0060] Distributed similarity connections are costly to communicate and complex to deploy. Because candidate pairs may span different data partitions, a large amount of intermediate results (such as candidate vector lists or collision counts) must be transmitted over the network; this "expensive inter-machine communication" becomes a major performance bottleneck. Furthermore, building and maintaining a large distributed cluster requires significant hardware and operational investment, far less economical and easier to deploy than a standalone solution. For many application scenarios, this resource consumption is unnecessary.

[0061] Disk-based vector similarity search suffers from severe disk I / O bottlenecks. There are two main reasons: First, the random access pattern of its nearest-neighbor graph index leads to significant read amplification, meaning that loading an entire disk page is required to read a tiny vector, resulting in substantial I / O waste. Second, during global scanning, the same data block is repeatedly loaded due to different query requests, which traditional caching strategies cannot handle, leading to extremely low cache hit rates. For example, disk I / O time accounts for over 70% of DiskANN's total execution time, preventing it from efficiently completing batch processing tasks.

[0062] It is evident that SSJ often requires multiple data reads and writes, resulting in a significant waste of input / output (I / O) and high maintenance and hardware investment.

[0063] Based on the above research, this application provides a vector data similarity query method, apparatus, device and medium. By dividing multiple vector data into data buckets, batch processing and targeted sorting are achieved, significantly improving local data consistency. Read and write operations are replaced by continuous batch processing instead of random access, thereby greatly reducing I / O overhead and improving single-machine performance. By combining neighbor data buckets to generate a bucket dependency graph and determining the processing order when writing multiple data buckets to the target cache, the storage layout is dynamically optimized, further reducing random I / O and significantly reducing operation and maintenance and hardware investment.

[0064] To facilitate understanding of this embodiment, a detailed description of the vector data similarity query method disclosed in this application embodiment will be provided first. The execution entity of the vector data similarity query method provided in this application embodiment is generally an electronic device with a certain computing power. This electronic device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, this vector data similarity query method can be implemented by a processor calling computer-readable instructions stored in memory.

[0065] The following description, in conjunction with the accompanying drawings, illustrates a vector data similarity query method provided in an embodiment of this application.

[0066] See Figure 1 The diagram shown is a flowchart illustrating a vector data similarity query method according to an exemplary embodiment of this application. Figure 1 As shown in the figure, the vector data similarity query method provided in this embodiment includes steps S101 to S104, wherein:

[0067] S101: Divide the multiple vector data to be queried into multiple data buckets, and store the vector data included in each data bucket into the corresponding storage area. Each data bucket includes at least one of the vector data.

[0068] The storage area refers to the corresponding file of the data bucket on the disk.

[0069] Traditional index-based methods suffer from severe read amplification and significant I / O bandwidth waste when accessing single vector data, as the vector data size is much smaller than the disk page size. This step employs access batching, dividing multiple vector data into multiple data buckets and storing them contiguously on disk. By using the data bucket as the smallest unit of data partitioning and storage, subsequent data access is no longer targeted at individual vector data, but rather at the entire data bucket in a batch sequential read. This reduces read amplification and optimizes the physical layout on disk to support efficient batch I / O.

[0070] In some possible implementations, dividing the multiple vector data to be queried into multiple data buckets includes:

[0071] Multiple target vector data are determined from multiple vector data to be queried according to a preset sampling ratio, and each target vector data serves as the center of a data bucket;

[0072] For each vector data other than the target vector data among the plurality of vector data, the vector data is assigned to the data bucket to which the target vector data that is closest to the vector data belongs.

[0073] In the above steps, multiple target vector data can be randomly sampled from the multiple vector data to be queried according to a preset sampling ratio, serving as the bucket centers of the data buckets. The specific value of the preset sampling ratio can be determined based on the resource supply corresponding to the vector data similarity query, and is not limited here. For example, the preset sampling ratio can be any integer less than 0.1%.

[0074] In other implementations, clustering algorithms, such as K-means++ or mini-batch K-means, can be used to cluster multiple vector data to be queried, thereby determining multiple target vector data.

[0075] For each vector data other than the target vector data in the plurality of vector data, the target vector data that is closest to the target vector data is determined from the plurality of target vector data, and the data bucket to which the determined closest target data belongs is determined as the data bucket to which the vector data belongs.

[0076] Optionally, a search index can be constructed for the multiple target vector data, such as an ANN search index (e.g., a Hierarchical Navigable Small World (HNSW) graph). For each vector data other than the target vector data, the constructed search index is used to perform a search, for example, an ANN search, to determine the target vector data that is closest to that vector data. It is understood that the index constructed here is only used to accelerate the process of allocating vector data to data buckets and is not the final main index serving SSJ.

[0077] In other implementations, Locality-Sensitive Hashing (LSH) can be used to determine the hash value of each of the vector data. For each vector data other than the target vector data, the vector data is assigned to the data bucket to which the target vector data with the same hash value belongs.

[0078] In this way, by determining the target vector data as the bucket center through a preset sampling ratio and allocating the remaining vector data to the corresponding data buckets, efficient data bucketing is achieved. This helps to ensure the rationality of data distribution, enhance the continuity and directionality of data processing, and thus significantly reduce random I / O operations and improve single-machine performance.

[0079] In some possible implementations, storing the vector data included in each of the data buckets into corresponding storage areas includes:

[0080] Set a corresponding buffer for each of the data buckets;

[0081] For each data bucket, the vector data allocated to that data bucket is loaded into the corresponding buffer. When the buffer is saturated, the vector data in the buffer is written into the storage area corresponding to that data bucket.

[0082] In the above steps, a corresponding buffer can be set in memory for each data bucket. After any vector data is allocated to a data bucket, the vector data is appended to the buffer corresponding to the allocated data bucket instead of being written to the storage area immediately. When the current storage capacity of the buffer reaches its capacity limit, that is, when the buffer is full, the vector data in the entire buffer is treated as a continuous data block and written to the storage area corresponding to the data bucket through a sequential write operation.

[0083] In this way, multiple reads and writes of a single vector data are avoided, reducing read amplification caused by random access and laying the foundation for subsequent batch sequential I / O. At the same time, the introduction of the buffer optimizes memory usage efficiency and reduces memory consumption.

[0084] S102: Determine the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each data bucket and its corresponding neighboring data buckets, and generate a data bucket dependency graph.

[0085] In this step, for each data bucket, the neighboring data buckets of the data bucket can be determined, and a directed edge is constructed between the data bucket and each of its corresponding neighboring data buckets, so that each data bucket and its corresponding neighboring data bucket form a data bucket pair, thereby generating the data bucket dependency graph G(V,E), where G represents the bucket dependency graph, V represents the vertices, i.e. each data bucket, and E represents the edges, i.e. each data bucket pair.

[0086] In some possible implementations, determining the neighboring data buckets associated with each of the data buckets includes:

[0087] For each data bucket, a preset number of data buckets that are closest to the center of the data bucket are determined from other data buckets to form an initial set of associated data buckets for that data bucket. The center distance is used to indicate the distance between the centers of two data buckets.

[0088] Remove the initial associated data buckets whose bucket center does not satisfy the triangle inequality relationship with the bucket center of the data bucket from the initial associated data bucket set to obtain the candidate associated data bucket set of the data bucket.

[0089] Based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set, the neighbor data bucket is selected from the candidate associated data bucket set. The recall loss ratio is used to indicate the loss of recall ratio when pruning the candidate associated data bucket.

[0090] In the above steps, for each data bucket, the distance between the bucket center of the data bucket and the bucket centers of each other data bucket is determined as the center distance. A preset number of data buckets with the closest center distance to the data bucket are determined from the other data buckets. The preset number of data buckets with the closest center distance are used to form the initial associated data bucket set of the data bucket.

[0091] Optionally, the search index built when allocating data buckets in the foregoing embodiments can be reused to determine a preset number of data buckets that are closest to the center of the data bucket.

[0092] The specific value of the preset quantity can be determined based on the search range required for filtering neighbor data buckets, and is not limited here.

[0093] After determining the initial set of associated data buckets for the data bucket, it can be determined whether the bucket center of the data bucket satisfies the triangle inequality relationship with the bucket centers of each of the initial associated data buckets in the initial set. Initial associated data buckets whose bucket centers do not satisfy the triangle inequality relationship are removed from the initial set of associated data buckets, resulting in a candidate set of associated data buckets for the data bucket. Next, the recall loss ratio of each candidate associated data bucket in the candidate set is determined, and probabilistic pruning is performed on the candidate associated data bucket set based on the recall loss ratio of each candidate associated data bucket to determine the neighboring data buckets.

[0094] In other implementations, after determining the initial set of associated data buckets for the data bucket, other filtering methods, such as grid-based or inverted file (IVF) filtering strategies, are used to identify the neighboring data buckets.

[0095] In this way, the technical solution filters the initial set of associated data buckets by center distance, ensuring the physical proximity of the data buckets with neighboring data buckets. It eliminates candidate buckets that do not conform to spatial logic by using the triangle inequality relationship, avoiding interference from invalid associations to the query. It selects neighboring data buckets based on the recall loss ratio, which greatly reduces the amount of subsequent distance calculation and I / O overhead while ensuring the recall rate. Through a multi-level filtering mechanism, combined with deterministic mathematical filtering and probabilistic statistical estimation, it retains high-value associated data to the greatest extent, achieving efficient and adjustable candidate pair reduction, and effectively balancing query efficiency and result completeness.

[0096] In some possible implementations, the initial associated data buckets that do not satisfy the triangle inequality relationship with the data bucket are determined through the following steps:

[0097] For any of the initial associated data buckets, if the center distance between the initial associated data bucket and the data bucket is greater than the preset threshold and the sum of the bucket radius of the initial associated data bucket and the bucket radius of the data bucket, it is determined that the initial associated data bucket and the data bucket do not satisfy the triangle inequality relationship.

[0098] Here, the triangle inequality states that for any three points A, B, and C, the following relationship exists as shown in formula (1):

[0099] dist(A,C)≤dist(A,B)+dist(B,C)(1)

[0100] Where dist(A,C) represents the distance between points A and C, dist(A,B) represents the distance between points A and B, and dist(B,C) represents the distance between points B and C.

[0101] Assuming that data bucket bi contains vector data x and data bucket bj contains vector data y, if the distance between vector data x and vector data y is less than or equal to a preset threshold, then vector data x and vector data y have the relationship shown in the following formula (2):

[0102] dist(x,y)≤ε (2)

[0103] Where dist(x,y) represents the distance between vector data x and vector data y, and ε represents the preset threshold.

[0104] Since the triangle inequality is monotonic, the data buckets bi and bj have the relationship shown in the following formula (3):

[0105] dist(c_bi,c_bj)≤dist(c_bi,x)+dist(x,y)+dist(c_bj,y) (3)

[0106] Where c_bi represents the center of data bucket bi, c_bj represents the center of data bucket bj, dist(c_bi,c_bj) represents the distance between the centers of data bucket bi and data bucket bj, dist(c_bi,x) represents the distance between the center of data bucket bi and vector data x, dist(x,y) represents the distance between vector data x and vector data y, and dist(c_bj,y) represents the distance between the center of data bucket bj and vector data y.

[0107] Since vector data x belongs to data bucket bi, vector data x and data bucket bi have the following relationship as shown in formula (4):

[0108] dist(c_bi,x)≤r_bi(4)

[0109] Where dist(c_bi,x) represents the distance between the center of data bucket bi and the vector data x, and r_bi represents the bucket radius of data bucket bi.

[0110] Since the vector data y belongs to the data bucket bj, the vector data y and the data bucket bj have the relationship shown in the following formula (5):

[0111] dist(c_bj,y)≤r_bj(5)

[0112] Where dist(c_bj,y) represents the distance between the center of data bucket bj and the vector data y, and r_bj represents the radius of data bucket bj.

[0113] Substituting formulas (2), (4), and (5) into formula (3), we obtain formula (6) as shown below:

[0114] dist(c_bi,c_bj)≤r_bi+ε+r_bj(6)

[0115] Where dist(c_bi,c_bj) represents the distance between the center of data bucket bi and the center of data bucket bj, r_bi represents the radius of data bucket bi, ε represents the preset threshold, and r_bj represents the radius of data bucket bj.

[0116] It is understandable that only when the centers of the two data buckets satisfy the triangle inequality relationship shown in formula (6) can there be a vector pair with a distance less than or equal to the preset threshold between the two data buckets.

[0117] For any of the initial associated data buckets, if the center distance between the initial associated data bucket and the data bucket is greater than the preset threshold and the sum of the bucket radius of the initial associated data bucket and the data bucket, it is determined that the initial associated data bucket and the data bucket do not satisfy the triangle inequality relationship; otherwise, it can be determined that the initial associated data bucket and the data bucket satisfy the triangle inequality relationship.

[0118] Optionally, the bucket radius of any data bucket can be determined by the following steps:

[0119] Determine the distance between each vector data in the data bucket and the center of the bucket, and determine the largest distance as the bucket radius of the data bucket.

[0120] In this way, the above method effectively realizes the judgment of the triangle inequality relationship. Through geometric constraints, it ensures that the spatial relationship between related data buckets conforms to the logic of the triangle inequality, thereby improving query efficiency.

[0121] In some possible implementations, the recall loss ratio for each candidate associated data bucket is determined through the following steps:

[0122] For each of the initial associated data buckets, the intersection volume between the sphere corresponding to the initial associated data bucket and the sphere corresponding to the data bucket is determined. The sphere corresponding to the initial associated data bucket has its center as the bucket center and its radius as the bucket radius. The sphere corresponding to the data bucket has its center as the bucket center and its radius as the sum of the radius of the data bucket and the preset threshold.

[0123] Based on the intersection volume corresponding to the initial associated data bucket and the sum of the intersection volumes corresponding to each initial associated data bucket, the recall loss ratio of the candidate associated data bucket is determined.

[0124] Here, it can be assumed that vector data whose distance to the vector data in the data bucket is less than or equal to a preset threshold are uniformly distributed in the spheres corresponding to the neighboring data buckets, thereby determining the recall loss ratio in the manner of this embodiment.

[0125] Alternatively, the recall loss ratio can be determined based on the actual data distribution. For example, for data known to be Gaussian distributed, the recall loss ratio can be determined by matching the Gaussian distribution to achieve different trade-offs between recall and efficiency.

[0126] Specifically, for vector data x in data bucket b, any vector data y whose distance is less than or equal to a preset threshold must satisfy d(x,y)≤ε. Since the maximum distance between vector data x and the bucket center c_b of data bucket b is the bucket radius r_b, and vector data y belongs to data bucket k, the maximum distance between vector data y and the bucket center c_k of data bucket k is the bucket radius r_k. Therefore, the maximum distance between vector data y and c_b is r_b+ε. This means that all potential vector data whose distance to the vector data in bucket b is less than or equal to the preset threshold must fall within the hypersphere B(c_b,r_b+ε) centered at c_b and with a radius of r_b+ε.

[0127] For example, the recall loss ratio of the candidate associated data bucket can be determined by formula (7):

[0128] β(j)=Volume(B(c_b,r_b+ε)∩B(c_k,r_k)) / Volume(U) (7)

[0129] Where β(j) represents the recall loss ratio, and Volume(B(c_b,r_b+ε)∩B(c_k,r_k)) represents the intersection volume between the sphere B(c_k,r_k) corresponding to the initial associated data bucket and the sphere B(c_b,r_b+ε) corresponding to the data bucket. The sphere B(c_k,r_k) corresponding to the initial associated data bucket is centered at the bucket center c_k of the initial associated data bucket and centered at the bucket center c_k of the initial associated data bucket. The radius r_k is the radius of the sphere. The sphere B(c_b,r_b+ε) corresponding to the data bucket has the bucket center c_b as its center and the sum of the radius of the data bucket and the preset threshold r_b+ε as its radius. Volume(U) represents the sum of the intersection volumes of the initial associated data buckets, U=U_{k∈L}(B(c_b,r_b+ε)∩B(c_k,r_k)), where L represents the initial associated data bucket set.

[0130] In this way, the recall loss ratio is quantified by determining the ratio of the intersection volume of candidate associated data buckets to the sum of the total intersection volume. By using geometric spatial relationships, the abstract recall loss is transformed into a calculable volume ratio, thus providing an objective quantitative basis for screening neighbor data buckets. In this way, through precise modeling of spatial geometry, the traditional coarse screening based on experience or fixed thresholds is avoided, and data buckets that have a significant impact on recall can be identified more accurately, optimizing computational efficiency while ensuring the comprehensiveness of the results.

[0131] In some possible implementations, the step of selecting the neighbor data bucket from the candidate associated data bucket set based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set includes:

[0132] In descending order of distance from the center of the data bucket, the recall rate loss ratios of each candidate associated data bucket in the candidate associated data bucket set are sequentially accumulated until the sum of the accumulated recall rate loss ratios reaches the preset recall rate loss.

[0133] The candidate associated data buckets accumulated when the preset recall rate loss is reached are deleted from the candidate associated data bucket set to obtain the neighbor data bucket.

[0134] In the above steps, the recall loss ratios of each candidate associated data bucket in the candidate associated data bucket set can be accumulated sequentially in descending order of distance from the center of the data bucket. Once the total accumulated recall loss ratio reaches the preset recall loss, it indicates that the tolerance for recall loss has reached its upper limit, and the accumulation can be stopped. The candidate associated data buckets accumulated when the preset recall loss is reached are deleted from the candidate associated data bucket set, and the remaining data buckets in the deleted candidate associated data bucket set are determined as the neighbor data buckets.

[0135] The preset recall loss can be the difference between 1 and the target recall. The specific data of the target recall can be determined according to the vector data similarity query requirements, and there is no restriction here.

[0136] In this way, by excluding data buckets that have a significant impact on recall rate loss through the above methods, the evaluation of unnecessary data buckets is effectively reduced, thereby improving query efficiency.

[0137] To more clearly illustrate the process of determining neighbor data buckets, see [link to documentation]. Figure 2 This is a schematic diagram illustrating a process for determining neighbor data buckets, as shown in an exemplary embodiment of this application. Figure 2 As shown, for each data bucket, an initial set of associated data buckets is determined based on the center distance. This initial set is then filtered according to the triangle inequality to determine a candidate set of associated data buckets. Finally, based on the recall loss ratio of each candidate data bucket, the candidate set of associated data buckets is further filtered to determine neighboring data buckets. Specific steps are described in the preceding embodiments and will not be repeated here.

[0138] S103: Based on the data bucket dependency graph and the capacity of the target cache, determine the processing order when writing the multiple data buckets into the target cache.

[0139] Here, when SSJ is regarded as a series of independent queries, the same memory blocks are often loaded repeatedly, making disk I / O a performance bottleneck. In this step, based on the data bucket dependency graph and the capacity of the target cache, the global future access information provided by the data bucket dependency graph is used to determine the processing order of the data bucket pairs in the data bucket dependency graph, so as to reduce the number of times data is loaded from the storage area to the target cache during the process of writing the multiple data buckets to the target cache.

[0140] In some possible implementations, determining the processing order for writing the plurality of data buckets to the target cache based on the data bucket dependency graph and the capacity of the target cache includes:

[0141] The target quantity is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache;

[0142] The data bucket with the largest out-degree among the multiple data buckets is used as the first data bucket to be written into the target cache;

[0143] Determine the correlation index of each of the other data buckets to be sorted, and take the data bucket with the largest correlation index as the second data bucket to be written into the target cache. The correlation index is used to indicate the number of common neighbor data buckets between the other data buckets to be sorted and the next target number of data buckets in the currently sorted data buckets.

[0144] Repeat the step of determining the association index of each of the other data buckets to be sorted until all the data buckets are sorted, and obtain the processing order when writing the data buckets to the target cache.

[0145] In the above steps, the target number is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache, which represents the neighboring data buckets that the target cache can accommodate.

[0146] For example, the target quantity can be determined by formula (8):

[0147] w=C / d_avg (8)

[0148] Where w represents the number of targets, C represents the capacity of the target cache, and d_avg represents the average out-degree of the data bucket dependency graph.

[0149] The data bucket with the highest out-degree among the multiple data buckets is selected as the first data bucket to be written into the target cache. The association index of each of the other data buckets to be sorted is determined.

[0150] For example, the correlation index can be determined by formula (9):

[0151] score(v)=Σ|N(P[j])∩N(v)|(9)

[0152] Where score(v) represents the correlation index of other data buckets v to be sorted, P[j] represents the data bucket at the j-th position in the currently sorted data buckets, N(P[j]) represents the neighboring data buckets of data bucket P[j], N(v) represents the neighboring data buckets of data bucket v, and N(P[j])∩N(v) represents the number of common neighboring data buckets of data bucket P[j] and data bucket v, where the value of j ranges from max(1,iw) to i-1), where i represents the current position to be sorted, and w represents the target number.

[0153] The correlation index quantifies the correlation between other data buckets to be sorted and recently sorted data buckets (i.e., the last target number of data buckets in the currently sorted data buckets). A higher correlation index means that the other data buckets to be sorted share more neighboring data buckets with recently sorted data buckets. After determining the first data bucket, each subsequent sorting operation adds the data bucket with the highest correlation index to the processing order. This maximizes the guarantee that shared neighboring data buckets, after being loaded into the target cache during the first processing, remain in the target cache during the second processing, thus enabling reuse, avoiding redundant disk reads, and effectively improving cache utilization.

[0154] The data bucket with the largest correlation index is used as the second data bucket to be written into the target cache. The step of determining the correlation index of each other data bucket to be sorted is repeated until all the data buckets are sorted, thus obtaining the processing order when writing the data buckets into the target cache.

[0155] To more clearly illustrate the processing order when writing the multiple data buckets to the target cache, the following example is provided.

[0156] For example, the data bucket dependency graph includes 6 data buckets (b1, b2, b3, b4, b5, b6). The neighboring data buckets of data bucket b1 are N(b1) = {b5, b6}, the neighboring data buckets of data bucket b2 are N(b2) = {b4, b5}, the neighboring data buckets of data bucket b3 are N(b3) = {b4, b6}, the neighboring data buckets of data bucket b4 are N(b4) = {b2, b3}, the neighboring data buckets of data bucket b5 are N(b5) = {b1, b2}, and the neighboring data buckets of data bucket b6 are N(b6) = {b1, b3}. Assuming the target cache capacity is C = 3 buckets and the average out-degree d_avg ≈ 2, the target quantity w = C / d_avg = 1.5. Since the data bucket is used as the smallest unit for data partitioning and storage in this embodiment, the target quantity is rounded down to w = 1 when it is a decimal. This means that when deciding on the next sorted data bucket, only the overlap between neighboring data buckets of the previously sorted data bucket is considered.

[0157] First, the data bucket with the largest out-degree is selected as the first data bucket to be written to the target cache. In this example, b1, b2, b3, b4, b5, and b6 each have two neighboring data buckets, so their out-degrees are all 2. Therefore, b1 is arbitrarily chosen as the first data bucket. The current processing order is T = [b1]. The set of other data buckets to be sorted is VR = {b2, b3, b4, b5, b6}.

[0158] Next, select the second data bucket (i=2) to write to the target cache. You can select a data bucket v from VR to maximize the correlation index score(v). At this time, the value range of j is from max(1,iw)=max(1,2-1)=1 to i-1=2-1=1, so j=1. Therefore, score(v)=|N(P[1])∩N(v)|=|N(b1)∩N(v)|. score(b2)=|N(b1)∩N(b2)|=|{b5,b6}∩{b4,b5}|=|{b5}|=1, score(b3)=|N(b1)∩N(b3 )|=|{b5,b6}∩{b4,b6}|=|{b6}|=1, score(b4)=|N(b1)∩N(b4)|=|{b5,b6}∩{b2,b3}|=| |=0, score(b5)=|N(b1)∩N(b5)|=|{b5,b6}∩{b1,b2}}|=| |=0, score(b6)=|N(b1)∩N(b6)|=|{b5,b6}∩{b1,b3}=| |=0. The correlation index of b2 and b3 is the largest and both are 1. Choose b2 as the second data bucket. Update the processing order T=[b1,b2], and update the set of other data buckets to be sorted, VR={b3,b4,b5,b6}.

[0159] Next, select the third data bucket (i=3) to write to the target cache. A data bucket v can be selected from VR to maximize the correlation index score(v). At this time, the value range of j is from max(1,iw)=max(1,3-1)=2 to i-1=3-1=2, so j=2. Therefore, score(v)=|N(P[2])∩N(v)|=|N(b2)∩N(v)|. score(b3)=|N(b2)∩N(b3)|=|{b4,b5}∩{b4,b6}|=|{b4}|=1, score(b4)=|N(b2)∩N(b4)|=|{b4,b5}∩{b2,b3}=| |=0, score(b5)=|N(b2)∩N(b5)|=|{b4,b5}∩{b1,b2}=| |=0, score(b6)=|N(b2)∩N(b6)|=|{b4,b5}∩{b1,b3}=| |=0. b3 has the highest correlation index, so b3 is selected as the third data bucket. Update the processing order T=[b1,b2,b3], and update the set of other data buckets to be sorted, VR={b4,b5,b6}. Continue the above process until the final processing order is obtained.

[0160] In this example, during the sorting process, when determining the data buckets other than the first data bucket, priority is given to data buckets that share common neighbor data buckets with the next target number of data buckets in the currently sorted data buckets. This helps to increase the local access continuity when writing data buckets from the storage area to the target cache, thereby improving the cache hit rate.

[0161] By employing the processing order determined in the above manner, if two data buckets share a large number of neighboring data buckets, then when these two data buckets are processed consecutively, their shared neighboring data buckets, loaded into the target cache during the first processing, can be reused during the second processing, thus avoiding redundant disk reads. In this way, by ensuring that data buckets with shared neighboring data buckets are processed consecutively, the temporal locality of the memory cache is maximized, reducing the number of cache misses.

[0162] In other implementations, other graph algorithms designed to maximize node neighborhood overlap, such as spectral clustering or community detection algorithms, may be used to determine the processing order when writing the multiple data buckets into the target cache.

[0163] S104: Write each data bucket and its corresponding neighbor data bucket from the storage area to the target cache in the order of processing, and determine the distance between two vector data belonging to each data bucket and its corresponding neighbor data bucket, and filter out vector pairs with a distance less than or equal to a preset threshold from the multiple vector data.

[0164] Here, the processing order is predetermined and fully known before the execution vector pair filtering. It is not a dynamically generated access flow in real time, but a complete execution plan derived from the data bucket dependency graph and the capacity of the target cache. A future access sequence can be generated based on this processing order, with a sequence length of 2|E|, where |E| is the number of edges in the bucket dependency graph (i.e., the number of data bucket pairs). When processing an edge (u,v) connecting data bucket u and data bucket v, both data bucket u and data bucket v need to be written from the storage area to the target cache.

[0165] Specifically, after each data bucket and its corresponding neighbor data bucket are written from the storage area to the target cache, the distance between vector data belonging to two different data buckets is determined, and it is determined whether the distance is less than or equal to the preset threshold. Vector pairs with a distance less than or equal to the preset threshold are then filtered out.

[0166] The specific value of the preset threshold can be determined according to the needs of vector data similarity query, and is not limited here.

[0167] In some possible implementations, the method further includes:

[0168] During the process of writing each data bucket and its corresponding neighboring data bucket from the storage area to the target cache in the order described in the processing order, if the target cache is saturated and not all data buckets have been written to the target cache, the data bucket with the latest next write time in the target cache is deleted, and the data bucket writing continues in the order described in the processing order.

[0169] In the above steps, each data bucket and its corresponding neighboring data bucket can be written from the storage area to the target cache sequentially according to the processing order. If the target cache is saturated and not all data buckets are written to the target cache due to excessive overhead, a situation may arise where the target cache is full and not all data buckets are written to the target cache. In this case, for each data bucket already written to the target cache, the next write time for each data bucket can be determined, the data bucket with the latest next write time can be deleted, and the data bucket writing can continue according to the processing order. Unlike traditional cache management methods, such as Least Recently Used (LRU) and Least Frequently Used (LFU), this method can reduce cache invalidation problems caused by a lack of a global view and improve data writing performance.

[0170] Specifically, for each data bucket that has been written to the target cache, if any data bucket does not need to be written to the target cache again in the future, the next write time corresponding to that data bucket can be set as the target time. The target time can be later than the next write time corresponding to the last data bucket in the processing order, so that when the capacity of the target cache is saturated and the data buckets have not all been written to the target cache, the data bucket can be evicted as soon as possible, reducing unnecessary storage space occupation.

[0171] Optionally, the writing and deletion of data buckets can be assisted by maintaining a future access index table and a max-heap. The future access index table records all occurrences of each data bucket in the future access sequence when it is written to the target cache according to the processing order. The future access index table can be constructed based on the future access sequence. For example, if the future access sequence S=[b1,b2,b1,b3], then the future access index table P includes three sub-future access index sequences: P[b1]=[0,2], P[b2]=[1], and P[b3]=[3], indicating that data bucket b1 is accessed at the 0th and 2nd positions of the future access sequence, data bucket b2 is accessed at the 1st position of the future access sequence, and data bucket b3 is accessed at the 3rd position of the future access sequence.

[0172] The max-heap is used to record the key values ​​of each data bucket that has been written to the target cache. The key value is used to indicate the next write time of the corresponding data bucket. For example, according to the processing order, if the data bucket b to be written is not in the target cache and the target cache is not saturated, the data bucket b can be written from the storage area to the target cache, the number of times the data bucket b has been accessed c[b] is determined, and the key value of the data bucket b is recorded in the max-heap Q. Here, if the number of times data bucket b has been accessed, c[b], is less than the length len(P[b]) of the corresponding sub-future access index sequence P[b], it means that data bucket b will appear in the future access sequence S in the future, that is, data bucket b will need to be written to the target cache again in the future. In this case, the key value corresponding to data bucket b is P[b][c[b]]. If the number of times data bucket b has been accessed, c[b], is equal to len(P[b]), it means that data bucket b will not appear in the future access sequence S in the future, that is, data bucket b will not need to be written to the target cache again in the future. In this case, the key value corresponding to data bucket b is greater than the value of the position of the last accessed data bucket in S. This allows the data bucket to be preferentially deleted from the target cache when the capacity of the target cache is saturated and the data bucket has not been fully written to the target cache.

[0173] If the data bucket to be written is already in the target cache, there is no need to repeat the data bucket write; only the access count and key value of the data bucket need to be updated. If the target cache is saturated, the data bucket with the largest corresponding key value in the target cache can be deleted, and the key value corresponding to that data bucket can be removed from the max heap, thereby writing the data bucket to be written.

[0174] To more clearly illustrate the process of deleting data buckets when the target cache is saturated, the following example will be used to explain it.

[0175] Exemplarily, the processing order is T = [b1, b2, b3], the neighbor buckets of bucket b1 are N(b1) = {b5, b6}, the neighbor buckets of bucket b2 are N(b2) = {b4, b5}, and the neighbor buckets of bucket b3 are N(b3) = {b4, b6}. Therefore, the future access sequence S = [b1, b5, b1, b6, b2, b4, b2, b5, b3, b4, b3, b6]. According to the future access sequence S, the future access index table P can be constructed: P[b1] = [0, 2], P[b5] = [1, 7], P[b6] = [3, 11], P[b2] = [4, 6], P[b4] = [5, 9], P[b3] = [8, 10]. The corresponding number of accessed times c[b] can be maintained for each bucket, initially 0, and it automatically increments by 1 each time it is accessed. If the target cache capacity is 3 and the first 4 accesses (i.e., S[0] to S[3]: b1, b5, b1, b6) have been processed, the cache content is {b1, b5, b6} at this time. The number of accessed times for each bucket is c[b1] = 2 (accessed at positions 0 and 2 in S), c[b5] = 1 (accessed at position 1 in S), and c[b6] = 1 (accessed at position 3 in S).

[0176] Next, it is to write S[4] = b2. At this time, the target cache is full and b2 is not in the target cache, so bucket deletion needs to be performed. Determine the next write time of each bucket in the target cache. For b1: c[b1] = 2, len(P[b1]) = 2, c[b1] = len(P[b1]), indicating that b1 will not be accessed in the future, and the key value can be set to 12 (a value greater than the last position 11 in S); for b5: c[b5] = 1, len(P[b5]) = 2, c[b5] < len(P[b5]), the next access is at P[b5][1] = 7; for b6: c[b6] = 1, len(P[b6]) = 2, c[b6] < len(P[b6]), the next access is at P[b6][1] = 11. At this time, Q includes (b1, 12), (b6, 11), (b5, 7). Delete b1 with the largest key value, write b2, c[b2] = 1, the key value is P[b2][1] = 6, and add (b2, 6) to Q.

[0177] Next, it is to write S[5] = b4. At this time, the target cache is full and b4 is not in the cache. At this time, the cache content is {b5, b6, b2}, and bucket deletion needs to be performed. At this time, Q includes (b5, 7), (b6, 11), (b2, 6). Delete b6 with the largest key value and write b4. Continue the above process until the final processing order is obtained.

[0178] In this way, when the target cache is saturated and the data buckets have not been fully written to the target cache, the data bucket with the latest next write time in the target cache is deleted, thereby reducing the cache miss rate and reducing the disk I / O performance bottleneck.

[0179] To more clearly illustrate the process of vector data similarity query, see [link to documentation]. Figure 3 This is a schematic diagram illustrating a vector data similarity query process, as shown in an exemplary embodiment of this application. Figure 3 As shown in the illustration, this embodiment employs a collaborative, batch-processing-optimized system, maximizing data access efficiency through global task scheduling. First, vector bucketing is performed, dividing the multiple vector data to be queried into multiple data buckets. The vector data within each data bucket is stored in its corresponding storage area. Then, neighboring data buckets with relationships to each data bucket are identified to generate a data bucket dependency graph. Based on the data bucket dependency graph and the target cache capacity, graph reordering is performed to determine the processing order for writing the multiple data buckets into the target cache. Data is then written according to this processing order, and vector pairs with a distance less than or equal to a preset threshold are selected. Specific steps are described in the preceding embodiment and will not be repeated here.

[0180] In this embodiment, vector data similarity query is treated as a unified batch processing task, rather than a series of independent queries. Through a series of collaborative processes such as vector bucketing, data bucket dependency graph construction, and graph reordering, the entire process from data organization to computation execution is systematically optimized.

[0181] This embodiment can be executed on a single machine, making full use of the advantages of large-capacity, low-cost SSDs. It can process billions of data points on a single machine. Compared with distributed solutions, it can not only reduce complex cluster management, high inter-machine communication costs and hardware costs, but also reduce communication bottlenecks in a distributed environment.

[0182] The vector data similarity query method provided in this application divides multiple vector data into data buckets, realizes batch processing and targeted sorting, significantly improves local data consistency, and enables read and write operations to replace random access with continuous batch processing, thereby greatly reducing I / O overhead and improving single-machine performance. It combines neighbor data buckets to generate a bucket dependency graph and determines the processing order when writing multiple data buckets to the target cache, dynamically optimizes the storage layout, further reduces random I / O, enhances adaptability and throughput performance, effectively reduces memory usage while ensuring high recall, breaks through capacity bottlenecks, helps overcome the shortcomings of dynamic updates and disk I / O optimization, and significantly reduces operation and maintenance and hardware investment.

[0183] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0184] Corresponding to the embodiments of the aforementioned vector data similarity query method, this application also provides embodiments of a vector data similarity query device.

[0185] Please see Figure 4 This is a schematic diagram illustrating a vector data similarity query device according to an exemplary embodiment of this application. Figure 4 As shown in the figure, the vector data similarity query device 400 provided in this application embodiment includes:

[0186] The vector bucketing module 401 is used to divide multiple vector data to be queried into multiple data buckets, and store the vector data included in each data bucket into the corresponding storage area, wherein each data bucket includes at least one vector data.

[0187] The association construction module 402 is used to determine the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each of the data buckets and their corresponding neighboring data buckets, and generate a data bucket dependency graph.

[0188] The task orchestration module 403 is used to determine the processing order of writing the multiple data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache;

[0189] The task execution module 404 is used to write each of the data buckets and their corresponding neighbor data buckets from the storage area to the target cache in the order of processing, and to determine the distance between two vector data belonging to each of the data buckets and their corresponding neighbor data buckets, and to filter out vector pairs with a distance less than or equal to a preset threshold from the plurality of vector data.

[0190] In some possible implementations, the vector bucketing module 401, when used to divide multiple vector data to be queried into multiple data buckets, is specifically used for:

[0191] Multiple target vector data are determined from multiple vector data to be queried according to a preset sampling ratio, and each target vector data serves as the center of a data bucket;

[0192] For each vector data other than the target vector data among the plurality of vector data, the vector data is assigned to the data bucket to which the target vector data that is closest to the vector data belongs.

[0193] In some possible implementations, the vector bucketing module 401, when storing the vector data included in each data bucket to its corresponding storage area, is specifically used for:

[0194] Set a corresponding buffer for each of the data buckets;

[0195] For each data bucket, the vector data allocated to that data bucket is loaded into the corresponding buffer. When the buffer is saturated, the vector data in the buffer is written into the storage area corresponding to that data bucket.

[0196] In some possible implementations, the association building module 402, when determining neighboring data buckets that are associated with each of the data buckets, is specifically used for:

[0197] For each data bucket, a preset number of data buckets that are closest to the center of the data bucket are determined from other data buckets to form an initial set of associated data buckets for that data bucket. The center distance is used to indicate the distance between the centers of two data buckets.

[0198] Remove the initial associated data buckets whose bucket center does not satisfy the triangle inequality relationship with the bucket center of the data bucket from the initial associated data bucket set to obtain the candidate associated data bucket set of the data bucket.

[0199] Based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set, the neighbor data bucket is selected from the candidate associated data bucket set. The recall loss ratio is used to indicate the loss of recall ratio when pruning the candidate associated data bucket.

[0200] In some possible implementations, the association building module 402 is used to determine the initial associated data buckets that do not satisfy the triangle inequality relationship with the data bucket through the following steps:

[0201] For any of the initial associated data buckets, if the center distance between the initial associated data bucket and the data bucket is greater than the preset threshold and the sum of the bucket radius of the initial associated data bucket and the bucket radius of the data bucket, it is determined that the initial associated data bucket and the data bucket do not satisfy the triangle inequality relationship.

[0202] In some possible implementations, the association building module 402 is used to determine the recall loss ratio of each candidate association data bucket through the following steps:

[0203] For each of the initial associated data buckets, the intersection volume between the sphere corresponding to the initial associated data bucket and the sphere corresponding to the data bucket is determined. The sphere corresponding to the initial associated data bucket has its center as the bucket center and its radius as the bucket radius. The sphere corresponding to the data bucket has its center as the bucket center and its radius as the sum of the radius of the data bucket and the preset threshold.

[0204] Based on the intersection volume corresponding to the initial associated data bucket and the sum of the intersection volumes corresponding to each initial associated data bucket, the recall loss ratio of the candidate associated data bucket is determined.

[0205] In some possible implementations, when the association construction module 402 is used to filter the neighbor data bucket from the candidate association data bucket set based on the recall loss ratio of each candidate association data bucket in the candidate association data bucket set, it is specifically used for:

[0206] In descending order of distance from the center of the data bucket, the recall rate loss ratios of each candidate associated data bucket in the candidate associated data bucket set are sequentially accumulated until the sum of the accumulated recall rate loss ratios reaches the preset recall rate loss.

[0207] The candidate associated data buckets accumulated when the preset recall rate loss is reached are deleted from the candidate associated data bucket set to obtain the neighbor data bucket.

[0208] In some possible implementations, the task orchestration module 403 is specifically used for:

[0209] The target quantity is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache;

[0210] The data bucket with the largest out-degree among the multiple data buckets is used as the first data bucket to be written into the target cache;

[0211] Determine the correlation index of each of the other data buckets to be sorted, and take the data bucket with the largest correlation index as the second data bucket to be written into the target cache. The correlation index is used to indicate the number of common neighbor data buckets between the other data buckets to be sorted and the next target number of data buckets in the currently sorted data buckets.

[0212] Repeat the step of determining the association index of each of the other data buckets to be sorted until all the data buckets are sorted, and obtain the processing order when writing the data buckets to the target cache.

[0213] In some possible implementations, the task execution module 404 is further configured to:

[0214] During the process of writing each data bucket and its corresponding neighboring data bucket from the storage area to the target cache in the order described in the processing order, if the target cache is saturated and not all data buckets have been written to the target cache, the data bucket with the latest next write time in the target cache is deleted, and the data bucket writing continues in the order described in the processing order.

[0215] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0216] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0217] Based on the same technical concept, this application also provides a computer device 500, referring to... Figure 5 The diagram shown is a schematic representation of the structure of a computer device according to an exemplary embodiment of this application, comprising:

[0218] The processor 510, memory 520, and bus 530 are included. The memory 520 is used to store execution instructions and includes main memory 521 and external memory 522. The main memory 521, also known as internal memory, is used to temporarily store the operation data in the processor 510 and the data exchanged with external memory 522 such as hard disk. The processor 510 exchanges data with external memory 522 through main memory 521.

[0219] In this embodiment, the memory 520 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 510. That is, when the electronic device 500 is running, the processor 510 communicates with the memory 520 through the bus 530, or the processor 510 communicates with the memory 520 through other means, so that the processor 510 executes the application code stored in the memory 520, and then executes the steps of the vector data similarity query method described in any of the foregoing embodiments.

[0220] The memory 520 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0221] Processor 510 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0222] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 500. In other embodiments of this application, the electronic device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0223] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the vector data similarity query method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0224] This disclosure also provides a computer program product, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the vector data similarity query method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0225] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0226] Furthermore, embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0227] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0228] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0229] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0230] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0231] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0232] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0233] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for vector data similarity query, characterized in that, The method includes: The multiple vector data to be queried are divided into multiple data buckets, and the vector data included in each data bucket is stored in the corresponding storage area. Each data bucket includes at least one of the vector data. Identify the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each data bucket and its corresponding neighboring data buckets, and generate a data bucket dependency graph; Based on the data bucket dependency graph and the capacity of the target cache, determine the processing order when writing the multiple data buckets into the target cache; According to the processing order, each data bucket and its corresponding neighbor data bucket are written from the storage area to the target cache in sequence, and the distance between two vector data belonging to each data bucket and its corresponding neighbor data bucket is determined. Vector pairs with a distance less than or equal to a preset threshold are selected from the multiple vector data. The step of determining the processing order for writing the multiple data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache includes: The target quantity is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache; The data bucket with the largest out-degree among the multiple data buckets is used as the first data bucket to be written into the target cache; Determine the correlation index of each of the other data buckets to be sorted, and take the data bucket with the largest correlation index as the second data bucket to be written into the target cache. The correlation index is used to indicate the number of common neighbor data buckets between the other data buckets to be sorted and the next target number of data buckets in the currently sorted data buckets. Repeat the step of determining the association index of each of the other data buckets to be sorted until all the data buckets are sorted, and obtain the processing order when writing the data buckets to the target cache.

2. The method according to claim 1, characterized in that, The process of dividing the multiple vector data to be queried into multiple data buckets includes: Multiple target vector data are determined from multiple vector data to be queried according to a preset sampling ratio, and each target vector data serves as the center of a data bucket; For each vector data other than the target vector data among the plurality of vector data, the vector data is assigned to the data bucket to which the target vector data that is closest to the vector data belongs.

3. The method according to claim 1, characterized in that, The step of storing the vector data included in each of the data buckets into their respective storage areas includes: Set a corresponding buffer for each of the data buckets; For each data bucket, the vector data allocated to that data bucket is loaded into the corresponding buffer. When the buffer is saturated, the vector data in the buffer is written into the storage area corresponding to that data bucket.

4. The method according to claim 1, characterized in that, The step of determining the neighboring data buckets that are associated with each of the data buckets includes: For each data bucket, a preset number of data buckets that are closest to the center of the data bucket are determined from other data buckets to form an initial set of associated data buckets for that data bucket. The center distance is used to indicate the distance between the centers of two data buckets. Remove the initial associated data buckets whose bucket center does not satisfy the triangle inequality relationship with the bucket center of the data bucket from the initial associated data bucket set to obtain the candidate associated data bucket set of the data bucket. Based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set, the neighbor data bucket is selected from the candidate associated data bucket set. The recall loss ratio is used to indicate the loss of recall ratio when pruning the candidate associated data bucket.

5. The method according to claim 4, characterized in that, The following steps are used to determine the initial associated data buckets that do not satisfy the triangle inequality relationship with the given data bucket: For any of the initial associated data buckets, if the center distance between the initial associated data bucket and the data bucket is greater than the preset threshold and the sum of the bucket radius of the initial associated data bucket and the bucket radius of the data bucket, it is determined that the initial associated data bucket and the data bucket do not satisfy the triangle inequality relationship.

6. The method according to claim 4, characterized in that, The recall loss ratio for each candidate associated data bucket is determined by the following steps: For each of the initial associated data buckets, the intersection volume between the sphere corresponding to the initial associated data bucket and the sphere corresponding to the data bucket is determined. The sphere corresponding to the initial associated data bucket has its center as the bucket center and its radius as the bucket radius. The sphere corresponding to the data bucket has its center as the bucket center and its radius as the sum of the radius of the data bucket and the preset threshold. Based on the intersection volume corresponding to the initial associated data bucket and the sum of the intersection volumes corresponding to each initial associated data bucket, the recall loss ratio of the candidate associated data bucket is determined.

7. The method according to claim 4, characterized in that, The step of selecting the neighbor data bucket from the candidate associated data bucket set based on the recall loss ratio of each candidate associated data bucket in the candidate associated data bucket set includes: In descending order of distance from the center of the data bucket, the recall rate loss ratios of each candidate associated data bucket in the candidate associated data bucket set are sequentially accumulated until the sum of the accumulated recall rate loss ratios reaches the preset recall rate loss. The candidate associated data buckets accumulated when the preset recall rate loss is reached are deleted from the candidate associated data bucket set to obtain the neighbor data bucket.

8. The method according to claim 1, characterized in that, The method further includes: During the process of writing each data bucket and its corresponding neighboring data bucket from the storage area to the target cache in the order described in the processing order, if the target cache is saturated and not all data buckets have been written to the target cache, the data bucket with the latest next write time in the target cache is deleted, and the data bucket writing continues in the order described in the processing order.

9. A vector data similarity query device, characterized in that, The device includes: The vector bucketing module is used to divide multiple vector data to be queried into multiple data buckets, and store the vector data included in each data bucket into the corresponding storage area. Each data bucket includes at least one vector data. The association construction module is used to determine the neighboring data buckets that are associated with each of the data buckets, construct directed edges between each of the data buckets and its corresponding neighboring data buckets, and generate a data bucket dependency graph. The task orchestration module is used to determine the processing order of writing the multiple data buckets into the target cache based on the data bucket dependency graph and the capacity of the target cache; The task execution module is used to write each of the data buckets and their corresponding neighbor data buckets from the storage area to the target cache in the order of processing, and to determine the distance between two vector data belonging to each of the data buckets and their corresponding neighbor data buckets, and to filter out vector pairs with a distance less than or equal to a preset threshold from the multiple vector data. The task orchestration module is specifically used for: The target quantity is determined based on the ratio of the average out-degree of the data bucket dependency graph to the capacity of the target cache; The data bucket with the largest out-degree among the multiple data buckets is used as the first data bucket to be written into the target cache; Determine the correlation index of each of the other data buckets to be sorted, and take the data bucket with the largest correlation index as the second data bucket to be written into the target cache. The correlation index is used to indicate the number of common neighbor data buckets between the other data buckets to be sorted and the next target number of data buckets in the currently sorted data buckets. Repeat the step of determining the association index of each of the other data buckets to be sorted until all the data buckets are sorted, and obtain the processing order when writing the data buckets to the target cache.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the vector data similarity query method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the vector data similarity query method according to any one of claims 1 to 8.

Images