Diversity vector retrieval method, electronic device, and program product

Abstract

The present disclosure provides a diversity vector retrieval method, an electronic device and a program product, and relates to the technical field of computers. The diversity vector retrieval method comprises: determining a target cutoff threshold of a database vector set by a dynamic optimization algorithm for database vectors contained in the database vector set; determining the spatial distance between each pair of database vectors in the database vector set, and constructing a probability filter according to the vector identifier combinations of the pairs of database vectors with a spatial distance less than the target cutoff threshold; performing distance estimation on the received query vector and each compressed vector cached in the memory, and performing beam search according to the distance estimation result and the disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors; and performing homogenization filtering on the candidate database vectors in the candidate vector set according to the vector identifier combinations contained in the probability filter to obtain a target vector set.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a diversity vector retrieval method, electronic device, and program product. Background Technology

[0002] Vector retrieval technology is widely used in recommendation systems, image retrieval, natural language processing, and retrieval enhancement generation. To avoid retrieval results falling into "information cocoons" (i.e., returning a large amount of highly similar homogeneous content), existing technologies have introduced diverse vector retrieval schemes. Taking the LotusFilter algorithm as an example, it typically performs a query on a graph index adjacency list (such as a Hierarchical Navigable Small World (HNSW)). It first recalls a set of candidate vectors, and then uses a pre-constructed truncation table (which precisely records all redundant vector pairs with a spatial distance less than a certain fixed threshold) to filter the candidate vectors online, removing vectors that are too similar to the selected results, thereby achieving diverse recommendations.

[0003] However, the aforementioned existing technologies employ a pure in-memory architecture, requiring the complete graph index adjacency list, all original high-dimensional vector data, and the truncated table to reside permanently in memory. When processing massive amounts of data, this consumes hundreds or even thousands of GB of DRAM (Dynamic Random Access Memory), making it difficult to implement at low cost in real-world production environments. Furthermore, precisely storing all redundant vector pairs in the truncated table occupies a huge amount of memory space, further exacerbating the memory bottleneck. Summary of the Invention

[0004] This disclosure provides a diversity vector retrieval method, electronic device, and program product.

[0005] According to one aspect of this disclosure, a diversity vector retrieval method is provided, comprising: For the database vectors contained in the database vector set, a target truncation threshold for the database vector set is determined by a dynamic optimization algorithm; Determine the spatial distance between any two database vectors in the database vector set, and construct a probability filter based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold; The received query vector is compared with each compressed vector cached in memory for distance estimation. Based on the distance estimation results and the disk-based graph index, beam search is performed to obtain a candidate vector set containing multiple candidate database vectors. The compressed vectors are obtained by compressing the database vectors in the database vector set, and the graph index is constructed based on the database vectors in the database vector set. Based on the vector identifier combination contained in the probability filter, the candidate database vectors in the candidate vector set are subjected to homogenization filtering to obtain the target vector set.

[0006] According to one technical solution, a target truncation threshold for the database vector set is determined through a dynamic optimization algorithm. Then, the spatial distance between each pair of database vectors in the set is determined. A probabilistic filter is constructed based on the vector identifiers of database vector pairs whose spatial distance is less than the target truncation threshold. This probabilistic filter replaces the precisely stored truncation table, enabling the storage of redundant vector pairs with minimal memory overhead. Furthermore, the graph index is stored on disk, while only compressed vectors obtained from database vector compression are cached in memory. When a query vector is received, the distance between the query vector and each compressed vector is estimated. A beam search is then performed based on the distance estimation results and the graph index to obtain a candidate vector set, significantly reducing memory consumption and lowering hardware costs. Finally, the probabilistic filter is used to perform homogenization filtering on the candidate vector set, accurately eliminating redundant results and ensuring the diversity of the target vector set.

[0007] According to at least one embodiment of the diversity vector retrieval method of this disclosure, for database vectors contained in a database vector set, a target truncation threshold for the database vector set is determined by a dynamic optimization algorithm, including: Extract a portion of database vectors from the database vector set as a training query set, and calculate the spatial distance between each pair of database vectors in the training query set; Using the maximum spatial distance as the right boundary and zero as the left boundary, we obtain the initial search interval; Based on the initial search interval, multiple rounds of dynamic optimization are performed to determine the target truncation threshold of the database vector set. In each round of dynamic optimization, multiple candidate truncation thresholds are uniformly sampled within the current search interval. The comprehensive cost score of the retrieval results under homogeneity filtering based on each candidate truncation threshold is determined. The comprehensive cost score is the weighted sum of the search cost and diversity cost of the retrieval results. The candidate truncation threshold with the smallest comprehensive cost score is taken as the current optimal truncation threshold. The current search interval is then halved based on the position of the current optimal truncation threshold within the current search interval.

[0008] According to the technical solution of this embodiment, there is no need to rely on prior experience to set a fixed cutoff threshold. Through dynamic optimization algorithm, the optimal cutoff threshold, i.e. the target cutoff threshold, can be determined based on the distribution characteristics of database vectors and retrieval performance requirements (based on the weighted average of search cost and diversity cost).

[0009] According to the diversity vector retrieval method of at least one embodiment of the present disclosure, during multi-round dynamic optimization, the number of samples of the candidate truncation threshold in the last round is greater than the number of samples of the candidate truncation threshold in other rounds.

[0010] According to the technical solution of this embodiment, a fine search can be performed, avoiding the omission of the true optimal cutoff threshold due to sparse sampling.

[0011] According to at least one embodiment of the diversity vector retrieval method of this disclosure, the received query vector is distance-estimated with each compressed vector cached in memory, and beam search is performed based on the distance estimation results and a disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors, including: Based on the received query vector, calculate the approximate distance between the query vector and each compressed vector cached in memory; The database vectors corresponding to the N compressed vectors with the smallest approximate distance are selected as the entry nodes for beam search, where N is an integer greater than 0; Beam search is performed on the disk-based graph index based on the entry node to obtain a candidate vector set containing multiple candidate database vectors.

[0012] According to the technical solution of this embodiment, the approximate distance is calculated using compressed vectors in memory, and the entry node is accurately selected based on the approximate distance. This avoids blind or global random searches in a large graph index, and significantly reduces the computational overhead and I / O latency of the initial search.

[0013] According to at least one embodiment of the diversity vector retrieval method of this disclosure, a probability filter is constructed based on a combination of vector identifiers of database vector pairs whose spatial distance is less than the target truncation threshold, including: Traverse the spatial distance between each pair of database vectors in the database vector set, and determine the database vector pairs whose spatial distance is less than the target truncation threshold as homogeneous vector pairs. The vector identifiers of the homogenized vector pairs are combined as input keys and input into multiple different hash functions to obtain multiple mapping position indices; The positions in the preset bit array corresponding to the multiple mapping position indices are updated from 0 to 1 to obtain the probability filter.

[0014] According to the technical solution of this embodiment, by utilizing the bitwise operation characteristics of probabilistic data structures, a large and redundant precise truncated hash table can be replaced with minimal memory overhead, significantly reducing its hardware implementation cost.

[0015] According to at least one embodiment of the diversity vector retrieval method of this disclosure, a target vector set is obtained by homogenizing candidate database vectors in the candidate vector set based on vector identifier combinations included in the probability filter, including: For the candidate database vectors in the candidate vector set, extract the vector identifier combination between the current candidate database vector and each target database vector that has been added to the result set, and use it as the query key value; Each of the query keys is sequentially input into the probability filter for querying. If the query result corresponding to any of the query keys is a hit, the current candidate database vector is discarded. If the query results corresponding to all the query keys are not hits, the current candidate database vector is added to the result set as the target database vector. When the number of target database vectors in the result set reaches the target number, the current result set is output as the target vector set.

[0016] According to the technical solution of this embodiment, by utilizing the bitwise operation characteristics of the probability filter, it is possible to quickly complete the homogeneous identification and interception of a large number of candidate database vectors with constant time complexity, thereby reducing computational latency and memory overhead.

[0017] According to at least one embodiment of the diversity vector retrieval method of this disclosure, after constructing a probability filter, the diversity vector retrieval method further includes: In response to a vector deletion event for a set of database vectors, the vector identifier of the deleted database vector is determined, and the vector identifier combination associated with the vector identifier of the deleted database vector is removed from the probability filter to obtain an updated probability filter. Alternatively, in response to a vector deletion event for a database vector set, the probability filter is marked as pending reconstruction, and the probability filter is reconstructed based on the updated database vector set when preset reconstruction conditions are met.

[0018] According to the diversity vector retrieval method of at least one embodiment of the present disclosure, the probability filter includes any one of a Bloom filter, a Cuckoo filter, or a Quotient filter.

[0019] The technical solution of this embodiment can be adapted to different application scenarios.

[0020] According to another aspect of this disclosure, a diversity vector retrieval apparatus is provided, comprising: The threshold optimization module is used to determine the target truncation threshold of the database vector set through a dynamic optimization algorithm for the database vectors contained in the database vector set. A probability filter construction module is used to determine the spatial distance between any two database vectors in the database vector set, and to construct a probability filter based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold. The search module is used to estimate the distance between the received query vector and each compressed vector cached in memory, and to perform beam search based on the distance estimation result and the disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors. The compressed vectors are obtained by compressing the database vectors in the database vector set, and the graph index is constructed based on the database vectors in the database vector set. The diversity recommendation module is used to perform homogenization filtering on candidate database vectors in the candidate vector set based on the vector identifier combination contained in the probability filter, so as to obtain the target vector set.

[0021] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs a diversity vector retrieval method according to any embodiment of this disclosure.

[0022] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the diversity vector retrieval method of any embodiment of this disclosure.

[0023] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the diversity vector retrieval method of any embodiment of this disclosure.

[0024] The beneficial effect of this disclosed technical solution is that it enables efficient and diversified vector retrieval with low hardware cost in the case of large-scale vector data (such as billions of data points). Attached Figure Description

[0025] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0026] Figure 1 This is a flowchart illustrating a diversity vector retrieval method according to one embodiment of the present disclosure.

[0027] Figure 2 This is a flowchart illustrating step S110 of a diversity vector retrieval method according to one embodiment of the present disclosure.

[0028] Figure 3 This is a flowchart illustrating step S130 of a diversity vector retrieval method according to one embodiment of the present disclosure.

[0029] Figure 4 This is a schematic diagram of the process of constructing a probability filter in a diversity vector retrieval method according to one embodiment of the present disclosure.

[0030] Figure 5 This is a flowchart illustrating step S140 of a diversity vector retrieval method according to one embodiment of the present disclosure.

[0031] Figure 6 This is a schematic structural block diagram of a diversity vector retrieval device according to one embodiment of the present disclosure.

[0032] Figure 7 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0033] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0034] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Existing technologies are limited by the massive memory consumption brought about by pure memory architecture, making it difficult to achieve low hardware cost deployment while ensuring retrieval quality.

[0036] To address this, this disclosure proposes the following technical solution: A dynamic optimization algorithm determines the target truncation threshold of the database vector set. Then, the spatial distance between each pair of database vectors in the set is determined. A probabilistic filter is constructed based on the vector identifiers of database vector pairs whose spatial distance is less than the target truncation threshold. This probabilistic filter replaces the precisely stored truncation table, enabling the storage of redundant vector pairs with minimal memory overhead. Furthermore, the graph index is stored on disk, while compressed vectors obtained from database vector compression are cached in memory. When a query vector is received, the distance between the query vector and each compressed vector is estimated. A beam search is then performed based on the distance estimation results and the graph index to obtain a candidate vector set, significantly reducing memory consumption and lowering hardware costs. Finally, the probabilistic filter is used to perform homogenization filtering on the candidate vector set, accurately eliminating redundant results and ensuring the diversity of the target vector set.

[0037] Therefore, the technical solution disclosed herein can achieve efficient and diversified vector retrieval with low hardware cost in the case of large-scale vector data (such as billions of data points).

[0038] Figure 1 This is a flowchart illustrating a diversity vector retrieval method according to one embodiment of the present disclosure. This diversity vector retrieval method can be applied to a terminal device or a server, wherein the terminal device includes, but is not limited to, one or more of smartphones, tablets, laptops, and desktop computers.

[0039] like Figure 1 As shown, the diversity vector retrieval method preferably includes steps S110 to S140.

[0040] In step S110, a target truncation threshold for the database vector set is determined by a dynamic optimization algorithm for the database vectors contained in the database vector set.

[0041] The database vector set is a pre-stored collection of raw vector data to be retrieved. Examples include image feature vectors and text feature vectors, which are the target objects of the retrieval operation.

[0042] The truncation threshold is a distance threshold used to determine whether two database vectors constitute a redundant vector pair. When the spatial distance between two database vectors is less than the truncation threshold, they are considered highly homogeneous or redundant and should be filtered out in diversity vector retrieval.

[0043] In this implementation, a pre-set dynamic optimization algorithm determines the optimal truncation threshold, or target truncation threshold, for the database vector set based on the spatial distance between each pair of database vectors. This solves the technical problem of difficulty in adapting to the spatial distance distribution span between different modalities of data (such as image feature vectors and text feature vectors) due to relying on manual experience to set the truncation threshold, effectively balancing retrieval efficiency and result diversity.

[0044] In step S120, the spatial distance between each pair of database vectors in the database vector set is determined, and a probability filter is constructed based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold.

[0045] Spatial distance refers to the geometric distance between two database vectors in the feature space, which can be measured using Euclidean distance or cosine distance.

[0046] The vector identifier is a unique identifier corresponding to each database vector, used to distinguish different database vectors. Preferably, the vector identifier can be an integer, a string, or other encoding that can uniquely identify a database vector.

[0047] In this implementation, all database vector pairs in the database vector set are traversed, and the spatial distance between the two database vectors in each pair is calculated. Then, database vector pairs with a spatial distance less than the target truncation threshold are selected and identified as redundant vector pairs.

[0048] Next, the vector identifiers of each database vector in each redundant vector pair are extracted and combined to obtain the vector identifier combination corresponding to that redundant vector pair. Then, based on the vector identifier combinations of all redundant vector pairs, a probabilistic filter is constructed. This probabilistic filter is a probabilistic data structure based on hash mapping and bit arrays, which can quickly determine whether a key-value pair belongs to a certain set (i.e., whether it is added to the probabilistic filter) with minimal memory overhead.

[0049] Preferably, the probabilistic filter includes any one of a Bloom filter, a Cuckoo filter, or a quotient filter. A Bloom filter inserts elements (such as vector identifier combinations) by mapping them to a bit array using multiple hash functions and setting the corresponding bits to 1. During a query, it checks if all mapped bits are 1 to determine if the element may exist (allowing false positives) or definitely does not exist (no false negatives). A Cuckoo filter supports element deletion, storing a compressed fingerprint of the element instead of its complete information. It uses two candidate buckets and a "kick-out" mechanism to resolve hash collisions. During insertion, if both buckets are full, an existing fingerprint is randomly kicked out and relocated. During a query, it only needs to check if a matching fingerprint exists in either bucket. A quotient filter splits the hash value into a quotient and a remainder. The quotient is used to locate the bucket (slot), and the remainder is stored as a fingerprint in the bucket. Collisions are managed through metadata (such as whether it is occupied, whether it is consecutive, etc.). Its core function is to achieve compact storage and fast member query, support deletion operations, require only a single hash function, and has good cache locality.

[0050] Those skilled in the art can choose different implementation forms of probability filters according to implementation needs, thereby adapting to different application scenarios.

[0051] In step S130, the distance between the received query vector and each compressed vector cached in memory is estimated, and beam search is performed based on the distance estimation result and the disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors. The compressed vectors are obtained by compressing the database vectors in the database vector set, and the graph index is constructed based on the database vectors in the database vector set.

[0052] The compressed vector is a vector with a very small volume obtained by compressing the original database vector. Preferably, this disclosure uses a product quantization compression method to compress the database vector. Specifically, the high-dimensional database vector is divided into several sub-vectors, each of which is independently subjected to cluster quantization. The index (short code) of the cluster center is used to replace the original sub-vector, thereby compressing the original high-dimensional database vector into a short code (i.e., a compressed vector) of tens to hundreds of bytes. It should be understood that the compressed vector retains the distance characteristics of the original database vector and can be used to quickly estimate the approximate distance between it and the query vector. This approximate distance is a distance estimate calculated based on the query vector and the compressed vector. It has the characteristics of fast calculation speed and low memory consumption, but its accuracy is lower than the precise spatial distance based on the original database vector.

[0053] The graph index is an index structure based on a nearest-neighbor graph, where each database vector corresponds to a node in the graph index, and nodes are connected by directed edges to form an approximate nearest-neighbor graph. The adjacency list in the graph index records a list of neighbor node IDs for each node. In this embodiment, the graph index is stored on disk rather than in memory.

[0054] In this implementation, the database vectors in the database vector set are compressed in advance to obtain the corresponding compressed vectors and cached in memory. At the same time, a corresponding graph index is constructed based on the database vectors in the database vector set and stored in the graph index on a hard disk (such as a solid-state drive).

[0055] Upon receiving the input query vector, the approximate distance between the query vector and each compressed vector is first calculated using the compressed vectors in memory. Then, the database vectors corresponding to the compressed vectors with the smallest approximate distances are selected as the entry nodes for beam searching, thereby performing beam searching in a disk-based graph index.

[0056] Specifically, this beam search maintains a candidate list of fixed capacity (i.e., beamwidth). In each iteration, it selects unexpanded nodes from the candidate list and visits their neighboring nodes. It calculates the spatial distance between neighboring nodes and the query vector and updates the candidate list (retaining only the first few nodes with the smallest distance within the first beamwidth), repeating this process until convergence or a preset number of iterations is reached. At this point, based on the nodes included in the candidate list, it obtains the corresponding database vector for each node as a candidate database vector, thus obtaining a candidate vector set.

[0057] In step S140, the candidate database vectors in the candidate vector set are subjected to homogenization filtering based on the vector identifier combination contained in the probability filter to obtain the target vector set.

[0058] Homogeneity filtering is a process of removing database vectors from the candidate vector set that are too similar to the selected database vectors in the feature space (i.e., the spatial distance is less than the target truncation threshold). Homogeneity filtering ensures that the spatial distance between any two target database vectors in the final target vector set is not less than the target truncation threshold, thereby achieving diversity in the retrieval results.

[0059] In this implementation, after the candidate vector set is retrieved, an empty target vector set is initialized, and the number of targets in the target vector set is set (e.g., 10). Next, the candidate database vectors in the candidate vector set are read sequentially (e.g., according to their approximate distance from the query vector in ascending order). Vector identifier combinations between the current candidate database vector and each target database vector already added to the target vector set are extracted, and these extracted vector identifier combinations are used for querying in a probability filter. If the query result is a match, it is determined that a homogeneous vector corresponding to the current candidate database vector already exists in the target vector set, so the current candidate database vector is discarded. If the query result is a miss, the current candidate database vector is added to the target vector set as a target database vector.

[0060] Preferably, when the number of target database vectors in the target vector set reaches the preset target number, the homogeneous filtering is stopped, and the target database vectors in the current target vector set are output.

[0061] Therefore, according to the diversity vector retrieval method of this disclosure, a target truncation threshold for the database vector set is determined through a dynamic optimization algorithm. Then, the spatial distance between each pair of database vectors in the database vector set is determined. A probabilistic filter is constructed based on the vector identifiers of database vector pairs whose spatial distance is less than the target truncation threshold. Using the probabilistic filter instead of a precisely stored truncation table allows for the storage of redundant vector pairs with minimal memory overhead. Furthermore, the graph index is stored on disk, and only compressed vectors obtained from database vector compression are cached in memory. When a query vector is received, the distance between the query vector and each compressed vector is estimated, and a beam search is performed based on the distance estimation results and the graph index to obtain a candidate vector set, significantly reducing memory consumption and lowering hardware costs. Then, the probabilistic filter is used to perform homogenization filtering on the candidate vector set, accurately eliminating redundant results and ensuring the diversity of the target vector set. The diversity vector retrieval method of this disclosure can achieve efficient and diverse vector retrieval with low hardware costs even with large-scale vector data (such as billions of data points).

[0062] In some embodiments of this disclosure, step S110, determining the target truncation threshold of the database vector set using a dynamic optimization algorithm for the database vectors contained in the database vector set, preferably includes steps S111 to S113. Please refer to [link / reference needed]. Figure 2 .

[0063] In step S111, a portion of the database vectors are extracted from the database vector set as a training query set, and the spatial distance between each pair of database vectors in the training query set is calculated.

[0064] In step S112, the maximum spatial distance is used as the right boundary and zero is used as the left boundary to obtain the initial search interval.

[0065] In step S113, multiple rounds of dynamic optimization are performed based on the initial search interval to determine the target truncation threshold of the database vector set. In each round of dynamic optimization, multiple candidate truncation thresholds are uniformly sampled within the current search interval to determine the comprehensive cost score of the retrieval results under homogeneity filtering based on each candidate truncation threshold. The comprehensive cost score is the weighted sum of the search cost and diversity cost of the retrieval results. The candidate truncation threshold with the smallest comprehensive cost score is taken as the current optimal truncation threshold, and the current search interval is halved based on the position of the current optimal truncation threshold in the current search interval.

[0066] In this implementation, a portion of the database vectors are extracted from the database vector set to form a training query set. For example, 2% of the database vectors in the database vector set are randomly selected to construct the training query set.

[0067] Based on the constructed training query set, calculate the spatial distance between each pair of database vectors in the training query set, and record the maximum spatial distance. Use this maximum spatial distance as the right boundary and zero as the left boundary to obtain the initial search interval.

[0068] Next, multiple rounds of dynamic optimization are performed based on the initial search interval to determine the target cutoff threshold for the database vector set. Specifically, in each round of dynamic optimization, several candidate cutoff thresholds (e.g., 5) are uniformly sampled within the current search interval (or the initial search interval if it is the first round of dynamic optimization). For example, the current search interval can be divided into five equal sub-intervals, and the midpoint value of each sub-interval can be selected as a candidate cutoff threshold; alternatively, after dividing the current search interval into equal parts, the boundary values ​​between adjacent sub-intervals can be selected as candidate cutoff thresholds.

[0069] After determining candidate truncation thresholds within the current search interval, a complete retrieval process is executed once or multiple times for each candidate truncation threshold to determine the comprehensive cost score of the retrieval results under homogeneity filtering based on each candidate truncation threshold. Preferably, for each candidate truncation threshold, a corresponding probabilistic filter is constructed according to the aforementioned probabilistic filter construction method. Then, using database vectors in the training query set or pre-defined vectors as query vectors, a complete retrieval process is executed (including approximate distance calculation with compressed vectors, beam search based on graph index, and homogeneity filtering based on the probabilistic filter).

[0070] Based on the execution results of the retrieval process, a comprehensive cost score is determined for the retrieval results corresponding to each candidate truncation threshold. Preferably, this comprehensive cost score is a weighted sum of the search cost and diversity cost of the retrieval results. The search cost reflects the computational resources and I / O overhead incurred to obtain the corresponding retrieval results. In one embodiment, the search cost includes the total number of graph nodes accessed during the query, the total number of distance calculations, and / or the number of disk I / O reads. If multiple factors are included, the weighted sum of these factors can be used as the search cost.

[0071] Diversity cost is used to reflect the degree of redundancy or homogeneity among vectors in the search results. In one embodiment, the diversity cost is determined based on the average or minimum spatial distance between the two database vectors in all vector pairs in the search results. The larger the average or minimum spatial distance, the more dispersed the search results are, the better the diversity, and the lower the diversity cost; conversely, the smaller the average or minimum spatial distance, the more concentrated the search results are, the worse the diversity, and the higher the diversity cost.

[0072] After determining the comprehensive cost score corresponding to the retrieval results of each candidate truncation threshold, the candidate truncation threshold with the smallest comprehensive cost score is taken as the current optimal truncation threshold. The current search interval is then halved based on the position of the current optimal truncation threshold within the current search interval, serving as the current search interval for the next round of dynamic optimization. When the number of rounds of dynamic optimization reaches a pre-set threshold (e.g., 5 rounds), the current optimal truncation threshold from the last round of dynamic optimization is output as the target truncation threshold.

[0073] Therefore, without relying on prior experience to set a fixed cutoff threshold, the optimal cutoff threshold, i.e. the target cutoff threshold, can be determined by a dynamic optimization algorithm based on the distribution characteristics of the database vectors and the retrieval performance requirements (based on a weighted average of search cost and diversity cost).

[0074] In one implementation, when halving the current search interval based on the position of the current optimal truncation threshold within the current search interval, the length of the current search interval is halved, centered on the position of the current optimal truncation threshold within the current search interval, to obtain the halved current search interval for the next round. It should be noted that if the new boundary exceeds the boundary of the original current search interval (e.g., less than the left boundary or greater than the right boundary of the original current search interval), the new current search interval is truncated to the left or right boundary of the original current search interval accordingly.

[0075] In other implementations, when halving the current search interval based on its position, the optimal truncation threshold is first determined to be located in the left or right half of the current search interval. Specifically, the optimal truncation threshold is compared with the midpoint of the current search interval. If the optimal truncation threshold is less than the midpoint, it is determined that the optimal truncation threshold is located in the left half of the current search interval; if the optimal truncation threshold is greater than the midpoint, it is determined that the optimal truncation threshold is located in the right half of the current search interval.

[0076] If the current optimal cutoff threshold is located in the left half of the search interval, then the left half of the current search interval is used as the current search interval for the next round of dynamic optimization. If the current optimal cutoff threshold is located in the right half of the search interval, then the right half of the current search interval is used as the current search interval for the next round of dynamic optimization. It should be noted that if the current optimal cutoff threshold is exactly located at the midpoint of the current search interval, then the length of the current search interval can be halved using the current optimal cutoff threshold as the midpoint to obtain the current search interval for the next round of dynamic optimization.

[0077] In some embodiments of this disclosure, during the multi-round dynamic optimization process, as the current search interval gradually converges, the sampling density of the candidate cutoff threshold is increased in the last round of dynamic optimization. That is, the number of candidate cutoff threshold samples in the last round of dynamic optimization is set to be greater than the number of candidate cutoff threshold samples in other rounds (i.e., all rounds of dynamic optimization before the last round). Preferably, the number of candidate cutoff threshold samples in the last round of dynamic optimization is 2 to 4 times the number of candidate cutoff threshold samples in other rounds.

[0078] For example, if there are five rounds of dynamic optimization, the number of candidate cutoff thresholds sampled in the first four rounds of dynamic optimization is 5, and the number of candidate cutoff thresholds sampled in the last round (i.e., the 5th round) is 10 or 20.

[0079] Therefore, when the search interval is large in the early rounds of dynamic optimization, a smaller number of candidate cutoff threshold samples are used, and only a small number of retrieval evaluations are performed in each round, significantly reducing the number of retrieval evaluations. In the final round of dynamic optimization, since the search interval has narrowed, a larger number of candidate cutoff threshold samples is used to perform a fine-grained search, avoiding the omission of the true optimal cutoff threshold due to sparse sampling.

[0080] In some embodiments of this disclosure, step S130, which involves estimating the distance between the received query vector and each compressed vector cached in memory, and performing beam search based on the distance estimation results and a disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors, preferably includes steps S131 to S133. Please refer to [reference needed]. Figure 3 .

[0081] In step S131, the approximate distance between the received query vector and each compressed vector cached in memory is calculated based on the query vector.

[0082] In step S132, the database vectors corresponding to the N compressed vectors with the smallest approximate distance are selected as the entry nodes for beam search, where N is an integer greater than 0.

[0083] In step S133, beam search is performed on the disk-based graph index according to the entry node to obtain a candidate vector set containing multiple candidate database vectors.

[0084] The approximate distance is an estimate calculated based on the compressed vectors, rather than the precise distance in the original vector space. Preferably, the approximate distance between the query vector and each compressed vector is determined by asymmetric distance computation (ADC).

[0085] In this implementation, when a user's query vector is received, the approximate distance between the query vector and each compressed vector cached in memory is first calculated. Preferably, the query vector q is divided into the same number of subvectors as the compressed vectors (e.g., M subvectors). For the i-th subvector, the distance between the subvector and all cluster centers (e.g., 256 centers) in the i-th subspace is pre-calculated to form a distance lookup table.

[0086] For each compressed vector c in memory, it consists of M sub-vector indices (each index ranges from 0 to 255). To find the approximate distance between vector q and compressed vector c, iterate through i from 1 to M, look up the distance value in the distance lookup table corresponding to the i-th sub-vector of vector q and the i-th sub-vector index c[i] of compressed vector c, and then sum the M distance values.

[0087] After determining the approximate distance between the query vector and each compressed vector, the compressed vectors are sorted in ascending order of approximate distance, and the vector identifiers of the database vectors corresponding to the first N compressed vectors (i.e. the N compressed vectors with the smallest approximate distance) and their node information in the graph index are obtained as the entry nodes for beam search.

[0088] Then, a beam search is performed on the disk-based graph index based on the entry nodes to retrieve a candidate vector set. In one implementation, during the beam search, a candidate list is maintained. Initially, N entry nodes are added to the candidate list, and each entry node is arranged in ascending order of its approximate distance to the query vector. Simultaneously, a set of visited nodes is maintained to avoid redundant expansion. The beamwidth B (B is a positive integer, such as B=1200) is set to indicate that at most B closest nodes are retained in the candidate list.

[0089] Next, an iterative loop is entered, selecting nodes from the candidate list that have not yet been expanded. The neighbor node identifier lists of these nodes are read from disk asynchronously using I / O. While waiting for I / O to complete, the distance to the queried neighbor nodes is estimated using the compressed vector in memory, and the candidate list is updated based on the distance estimation results. Specifically, when updating the candidate list, the queried neighbor nodes are merged with the nodes in the current candidate list. For newly added neighbor nodes, the approximate distance between the compressed vector corresponding to that neighbor node and the query vector is calculated. The nodes in the merged candidate list (including newly added neighbor nodes) are sorted in ascending order of their corresponding approximate distances, and the top B nodes are selected to construct a new candidate list. Simultaneously, nodes whose neighbor nodes have been queried are added to the visited set.

[0090] Repeat the above process until the candidate list no longer changes (i.e., all neighboring nodes of all nodes in the candidate list have been visited) or the preset maximum number of iterations is reached. After the iteration is completed, select the database vectors corresponding to the top S nodes (i.e., the S nodes whose corresponding compressed vectors have the smallest approximate distance to the query vector) from the final candidate list as candidate database vectors, construct a candidate vector set, and output it.

[0091] Therefore, by using compressed vectors in memory to calculate approximate distances and accurately selecting entry nodes based on these approximate distances, blind or global random searches in a large graph index are avoided, significantly reducing the computational overhead and I / O latency of the initial search.

[0092] In some embodiments of this disclosure, a probability filter is constructed based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold, preferably including steps S410 to S430. Please refer to [link / reference]. Figure 4 .

[0093] In step S410, the spatial distance between each pair of database vectors in the database vector set is traversed, and database vector pairs with a spatial distance less than the target truncation threshold are determined to be homogeneous vector pairs.

[0094] In step S420, the vector identifier combination of the homogenized vector pair is used as the input key value and input into multiple different hash functions to obtain multiple mapping position indices.

[0095] In step S430, the positions in the preset bit array corresponding to the plurality of mapping position indices are updated from 0 to 1 to obtain a probability filter.

[0096] In this implementation, the database vectors in the database vector set are grouped pairwise to obtain several database vector pairs. The spatial distance (such as Euclidean distance or cosine distance) between the two database vectors in each database vector pair is calculated. Then, the spatial distance of each database vector pair is compared with a target truncation threshold. If the spatial distance is less than the target truncation threshold, the database vector pair is determined to be a homogeneous vector pair (or a redundant vector pair).

[0097] Extract the vector identifiers of the two database vectors from each homogeneous vector pair and combine (e.g., concatenate) them into a vector identifier combination, which is then used as the input key. For each homogeneous vector pair, its corresponding input key is sequentially input into multiple different hash functions to obtain the corresponding number of mapping position indices. Next, in a pre-maintained bit array (initially, all bits in the bit array are zero), the positions corresponding to the multiple mapping position indices are updated from 0 to 1 (if they were originally 1, they remain unchanged). This write operation is repeated until all homogeneous vector pairs have been processed, thus obtaining the probability filter.

[0098] At this point, the constructed probability filter can support subsequent member query operations. That is, for any given combination of vector identifiers, the same multiple hash functions are used to calculate its mapping position index. If all positions in the bit array corresponding to the mapping position index are 1, it is determined that the combination of vector identifiers has most likely been recorded (i.e., it is a homogeneous vector pair). If any position is 0, it is determined that the combination of vector identifiers has not been recorded (i.e., it is a non-homogeneous vector pair).

[0099] Therefore, by leveraging the bitwise operation characteristics of probabilistic data structures, a large and redundant precise truncated hash table can be replaced with minimal memory overhead, significantly reducing its hardware deployment cost. Furthermore, the probabilistic filter is constructed through a combination of offline computation and hash mapping, enabling the rapid interception and filtering of homogeneous vector pairs in constant time complexity during the subsequent online retrieval stage, achieving a balance between low retrieval latency and high diversity assurance.

[0100] In some embodiments of this disclosure, step S140, which involves performing homogenization filtering on candidate database vectors in the candidate vector set based on the vector identifier combinations included in the probability filter to obtain the target vector set, preferably includes steps S141 to S143. Please refer to [reference needed]. Figure 5 .

[0101] In step S141, for the candidate database vectors in the candidate vector set, the vector identifier combination between the current candidate database vector and the target database vector that has been added to the result set is extracted and used as the query key value.

[0102] In step S142, the key value to be queried is input into the probability filter for querying. If the query result is a match, the current candidate database vector is discarded. If the query result is a miss, the current candidate database vector is added to the result set as the target database vector.

[0103] In step S143, when the number of target database vectors in the result set reaches the target number, the current result set is output as the target vector set.

[0104] The result set is a dynamically maintained collection during the homogenization filtering process. Initially empty, it temporarily stores the database vectors that have passed the homogenization filtering, i.e., the target database vectors. Preferably, the target database vectors in the result set are arranged in ascending order of their addition order or the approximate distance between their compressed vectors and the query vectors.

[0105] In this implementation, after the candidate vector set is retrieved, the candidate database vectors contained in the candidate vector set are subjected to homogenization filtering according to a pre-constructed probability filter. Specifically, the candidate database vectors contained in the candidate vector set are arranged in ascending order according to the order of their vector identifiers or according to the approximate distance between the corresponding compressed vector and the query vector. At the same time, a pre-set target number (e.g., 10) is obtained, and the result set is initialized to an empty set.

[0106] Next, the candidate database vectors in the candidate vector set are traversed in order. For the first candidate database vector encountered, it is directly added to the result set as the target database vector. For the second and subsequent candidate database vectors encountered, the vector identifier combination between the currently encountered candidate database vector and each target database vector already added to the result set is extracted and used as the query key value.

[0107] Next, for one or more query keys, the query keys are sequentially input into the probability filter for querying. If any query key corresponds to a match (i.e., any target database vector forms a homogeneous vector pair with the current candidate database vector), the current candidate database vector is discarded. If all query keys result in a miss (meaning no target database vector forms a homogeneous vector pair with the current candidate database vector), the current candidate database vector is added to the result set as the target database vector.

[0108] When a target database vector is added to the result set, the number of target database vectors already added to the current result set is determined and compared with the pre-set target number. If the target number is equal to the target number, the traversal of candidate database vectors stops, and the current result set is output as the target vector set.

[0109] Therefore, by leveraging the bitwise operation characteristics of probabilistic filters, it is possible to quickly identify and intercept a large number of candidate database vectors in constant time complexity, thereby reducing computational latency and memory overhead.

[0110] In some embodiments of this disclosure, after constructing the probability filter, the diversity vector retrieval method preferably further includes: in response to a vector deletion event for a set of database vectors, determining the vector identifier of the deleted database vector, and deleting the vector identifier combination associated with the vector identifier of the deleted database vector from the probability filter to obtain an updated probability filter.

[0111] In this implementation, the constructed probability filter is a cuckoo filter. When a vector deletion event is detected in the database vector set, the vector identifier of the deleted database vector is obtained. Based on the vector identifier of the deleted database vector, the probability filter retrieves the vector identifier combination associated with the vector identifier of the deleted database vector (the search can be performed on the vector identifier combination recorded when constructing the probability filter based on the vector identifier of the deleted database vector).

[0112] For each vector identifier combination containing the identifier of a deleted database vector, it is removed from the probabilistic filter. Specifically, for each vector identifier combination, it is hashed to determine the corresponding fingerprint. Then, a hash function is used to calculate the indices of two or more candidate buckets corresponding to that vector identifier combination. Based on the determined candidate bucket indices, an entry matching the fingerprint is searched from the candidate buckets corresponding to those indices. If an entry matching the fingerprint is found, that entry is removed from the candidate bucket. If no matching entry is found, it means that the vector identifier combination is not included in the probabilistic filter, and no deletion is required.

[0113] Therefore, by constructing a cuckoo filter, which removes fingerprints from the candidate bucket, the homogeneous filtering data structure can be dynamically maintained without affecting other homogeneous vector pairs without requiring a full reconstruction, thus reducing maintenance costs.

[0114] In some embodiments of this disclosure, after constructing the probability filter, the diversity vector retrieval method preferably further includes: in response to a vector deletion event for the database vector set, marking the probability filter as a state to be rebuilt, and reconstructing the probability filter according to the updated database vector set when preset reconstruction conditions are met.

[0115] In this implementation, when a probabilistic filter does not support native deletion operations (such as a Bloom filter), upon detecting a vector deletion event targeting a database vector set, the current probabilistic filter is marked as pending reconstruction (e.g., the status flag corresponding to the probabilistic filter is set to "pending reconstruction"). After marking the probabilistic filter as pending reconstruction, the existing probabilistic filter continues to be used for homogeneous filtering, and the online retrieval service is not immediately interrupted.

[0116] Then, continuously monitor whether the current system status meets the preset reconstruction conditions. Preferably, the reconstruction conditions include one or more of the following: system idle (e.g., CPU utilization or I / O load is consistently less than a certain threshold), reaching the upper limit of the number of vectors to be deleted (e.g., the number of deleted database vectors reaches a predetermined percentage of the original database vector set), timed conditions (e.g., automatically triggering reconstruction at preset time intervals), or manually triggering reconstruction.

[0117] When any reconstruction condition is detected, the updated database vector set (i.e., excluding all deleted database vectors) is obtained, and combined with the target truncation threshold (which can be re-determined according to the aforementioned method based on the current database vector set), the probability filter is reconstructed according to the aforementioned probability filter construction method.

[0118] Preferably, after the new probability filter is reconstructed, the old probability filter is replaced by the newly constructed probability filter in an atomic operation (e.g., by pointer swapping or version number update) to ensure that there is no inconsistency in the replacement process where some queries use the old filter and some queries use the new filter.

[0119] Therefore, by using a delayed reconstruction strategy, the reconstruction overhead is shifted to system idle periods or batch processing, ensuring high availability and extremely low latency of the online retrieval service.

[0120] Preferably, when the probability filter is a Bloom filter, when a new database vector is added to the database vector set, the spatial distance between the new database vector and all existing database vectors is calculated. Homogeneous vector pairs with a spatial distance less than the target truncation threshold are selected. The vector identifier combination of each homogeneous vector pair is used as a key value and mapped to the bit array of the current probability filter through multiple hash functions. The corresponding position is marked as 1, thereby realizing the incremental update of the probability filter, which synchronously records the homogeneity relationship of the new database vector without full reconstruction.

[0121] Figure 6 This is a schematic structural block diagram of a diversity vector retrieval device according to one embodiment of the present disclosure. Figure 6 As shown, the diversity vector retrieval device 600 preferably includes a threshold optimization module 610, a probability filter construction module 620, a search module 630, and a diversity recommendation module 640.

[0122] The threshold optimization module 610 is used to determine the target truncation threshold of the database vector set using a dynamic optimization algorithm; the probability filter construction module 620 is used to determine the spatial distance between each pair of database vectors in the database vector set, and construct a probability filter based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold; the search module 630 is used to estimate the distance between the received query vector and each compressed vector cached in memory, and perform beam search based on the distance estimation result and a disk-based graph index to obtain a candidate vector set containing multiple candidate database vectors, wherein the compressed vectors are obtained by compressing the database vectors in the database vector set, and the graph index is constructed based on the database vectors in the database vector set; the diversity recommendation module 640 is used to perform homogeneous filtering on the candidate database vectors in the candidate vector set based on the vector identifier combination contained in the probability filter to obtain a target vector set.

[0123] The aforementioned diversity vector retrieval device can be in the form of computer software, and each module of the aforementioned diversity vector retrieval device can be implemented through computer software modules.

[0124] In some embodiments of this disclosure, the probability filter construction module 620 is further configured to, in response to a vector deletion event for a database vector set, determine the vector identifier of the deleted database vector, delete the vector identifier combination associated with the vector identifier of the deleted database vector from the probability filter, and obtain an updated probability filter; or, in response to a vector deletion event for a database vector set, mark the probability filter as pending reconstruction, and reconstruct the probability filter according to the updated database vector set when preset reconstruction conditions are met.

[0125] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0126] The execution subject of the diversity vector retrieval method in the specific embodiments of this disclosure can be an electronic device such as a mobile phone or computer.

[0127] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the diversity vector retrieval method of any of the embodiments described above.

[0128] Figure 7 This is a schematic structural block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0129] like Figure 7 As shown, the hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0130] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0131] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0132] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0133] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0134] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0139] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0140] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A diversity vector retrieval method, characterized in that, include: For the database vectors contained in the database vector set, a target truncation threshold for the database vector set is determined by a dynamic optimization algorithm; Determine the spatial distance between any two database vectors in the database vector set, and construct a probability filter based on the vector identifier combination of database vector pairs whose spatial distance is less than the target truncation threshold; The received query vector is compared with the distances of each compressed vector cached in memory. Based on the distance estimation results and a disk-based graph index, a beam search is performed to obtain a candidate vector set containing multiple candidate database vectors. The compressed vectors are obtained by compressing database vectors from this set, and the graph index is constructed based on the database vectors in the set. Based on the vector identifier combination contained in the probability filter, the candidate database vectors in the candidate vector set are subjected to homogenization filtering to obtain the target vector set; Specifically, for the database vectors contained in the database vector set, a target truncation threshold for the database vector set is determined through a dynamic optimization algorithm, including: Extract a portion of database vectors from the database vector set as a training query set, and calculate the spatial distance between each pair of database vectors in the training query set; Using the maximum spatial distance as the right boundary and zero as the left boundary, we obtain the initial search interval; and Based on the initial search interval, multiple rounds of dynamic optimization are performed to determine the target truncation threshold of the database vector set. In each round of dynamic optimization, multiple candidate truncation thresholds are uniformly sampled within the current search interval. The comprehensive cost score of the retrieval results under homogeneity filtering based on each candidate truncation threshold is determined. The comprehensive cost score is the weighted sum of the search cost and diversity cost of the retrieval results. The candidate truncation threshold with the smallest comprehensive cost score is taken as the current optimal truncation threshold. The current search interval is then halved based on the position of the current optimal truncation threshold within the current search interval. In the process of multi-round dynamic optimization, the number of samples for the candidate cutoff threshold in the last round is greater than the number of samples for the candidate cutoff threshold in other rounds.

2. The diversity vector retrieval method as described in claim 1, characterized in that, The received query vector is compared with the distances of each compressed vector cached in memory. Based on the distance estimation results and the disk-based graph index, a beam search is performed to obtain a candidate vector set containing multiple candidate database vectors, including: Based on the received query vector, calculate the approximate distance between the query vector and each compressed vector cached in memory; The database vectors corresponding to the N compressed vectors with the smallest approximate distance are selected as the entry nodes for beam searching, where N is an integer greater than 0; and Beam search is performed on the disk-based graph index based on the entry node to obtain a candidate vector set containing multiple candidate database vectors.

3. The diversity vector retrieval method as described in claim 1, characterized in that, A probability filter is constructed based on the vector identifiers of database vector pairs whose spatial distance is less than the target truncation threshold, including: Traverse the spatial distance between each pair of database vectors in the database vector set, and determine the database vector pairs whose spatial distance is less than the target truncation threshold as homogeneous vector pairs. The vector identifiers of the homogenized vector pairs are combined as input keys and input into multiple different hash functions to obtain multiple mapping position indices; and The positions in the preset bit array corresponding to the multiple mapping position indices are updated from 0 to 1 to obtain the probability filter.

4. The diversity vector retrieval method as described in claim 3, characterized in that, Based on the vector identifier combinations included in the probability filter, homogenization filtering is performed on the candidate database vectors in the candidate vector set to obtain the target vector set, including: For the candidate database vectors in the candidate vector set, extract the vector identifier combination between the current candidate database vector and each target database vector that has been added to the result set, and use it as the query key value; Each of the query keys is sequentially input into the probability filter for querying. If the query result for any of the query keys is a match, the current candidate database vector is discarded. If the query results for all the query keys are not a match, the current candidate database vector is added to the result set as the target database vector. When the number of target database vectors in the result set reaches the target number, the current result set is output as the target vector set.

5. The diversity vector retrieval method as described in claim 1, characterized in that, After constructing the probability filter, the diversity vector retrieval method further includes: In response to a vector deletion event for a set of database vectors, the vector identifier of the deleted database vector is determined, and the vector identifier combination associated with the vector identifier of the deleted database vector is removed from the probability filter to obtain an updated probability filter. Alternatively, in response to a vector deletion event for a database vector set, the probability filter is marked as pending reconstruction, and the probability filter is reconstructed based on the updated database vector set when preset reconstruction conditions are met.

6. The diversity vector retrieval method as described in claim 5, characterized in that, The probability filter includes any one of a Bloom filter, a Cuckoo filter, or a quotient filter.

7. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the diversity vector retrieval method according to any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the diversity vector retrieval method according to any one of claims 1 to 6.

Images