Retrieval method and electronic equipment
By using the cluster center vector and the second data table deduplication method in the vector database, the vector duplication loading and calculation problems caused by inverted indexes are solved, and a more efficient query process is achieved.
Patent Information
- Application Number
- CN202411521490.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-03-28
AI Technical Summary
In the prior art, inverted indexes cause the boundary vector or all vectors to be repeatedly placed in multiple nearby clusters in the vector database, resulting in repeated loading and repeated calculations of vectors in memory disk mixed query scenarios, wasting transmission resources and computing resources.
By obtaining the search vector, determining the N cluster center vectors that match, querying the first data table to determine multiple vectors in the cluster, then querying the second data table to deduplicate and determine the deduplicate vector, and finally adding the deduplicate vector as a candidate vector to the second data table to determine the set of result vectors corresponding to the search vector.
It effectively avoids duplicate loading and repeated calculation of vectors, reduces the waste of input and output resources and vector computing resources, and improves query efficiency.
Smart Images
Figure CN119719098A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a retrieval method and an electronic device. Background Art
[0002] With the rapid iteration and evolution of big model technology and applications, the Retrieval-Augmented Generation (RAG) architecture has emerged to solve the problems of poor data real-time performance, data privacy security, and illusion in public big models. As the core component of the RAG solution, the performance of the vector database is extremely important. As the mainstream vector query method, the clustering-based inverted index not only supports memory queries for small data sets, but also supports memory-disk hybrid queries for large data sets, which plays a vital role in vector databases and vector computing.
[0003] However, the inverted index usually puts boundary vectors or all vectors repeatedly into multiple adjacent clusters. In the memory-disk hybrid query scenario, after finding the N clusters closest to the query vector in the memory, the vectors of each cluster are read into the memory in sequence and brute force queries are performed one by one. This solution will load the same vector from the disk index multiple times for different clusters, and calculate the distance between the query vector and the duplicate vector multiple times, wasting transmission resources and computing resources. Summary of the invention
[0004] The present disclosure provides a retrieval method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.
[0005] According to a first aspect of the present disclosure, a retrieval method is provided, the method comprising:
[0006] Obtain a search vector, and determine N cluster center vectors that match the search vector;
[0007] According to the N cluster center vectors, a first data table is queried to determine a plurality of vectors within the N clusters; the first data table is used to record a mapping relationship between the cluster center vectors and the vectors within the clusters;
[0008] Querying a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record information of at least one candidate vector;
[0009] The deduplicated vector is added as a candidate vector to a second data table, and a result vector set corresponding to the search vector is determined according to the second data table.
[0010] According to a second aspect of the present disclosure, a retrieval device is provided, the device comprising:
[0011] An acquisition module, used to acquire a search vector and determine N cluster center vectors that match the search vector;
[0012] A first determination module, configured to query a first data table according to the N cluster center vectors to determine a plurality of vectors within the N clusters; the first data table is configured to record a mapping relationship between the cluster center vectors and the vectors within the clusters;
[0013] A second determination module, configured to query a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record information of at least one candidate vector;
[0014] A processing module is used to add the deduplicated vector as a candidate vector to a second data table, and determine a result vector set corresponding to the search vector according to the second data table.
[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.
[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0021] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0022] Figure 1 A schematic diagram of a process flow of a search method provided by an embodiment of the present disclosure is shown;
[0023] Figure 2 A schematic diagram of a process flow of a search method provided by an application embodiment of the present disclosure is shown;
[0024] Figure 3 A schematic diagram of the structure of a retrieval device provided by an embodiment of the present disclosure is shown;
[0025] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0026] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0027] Figure 1 A flowchart of a search method provided by an embodiment of the present disclosure; Figure 1 As shown, the method can be applied to any device with a retrieval function, and the method includes:
[0028] Step 101: Obtain a search vector and determine N cluster center vectors that match the search vector;
[0029] Step 102: query a first data table according to the N cluster center vectors to determine a plurality of vectors within the N clusters; the first data table is used to record a mapping relationship between cluster center vectors and vectors within the clusters;
[0030] Step 103: query a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record information of at least one candidate vector;
[0031] Step 104: Add the deduplicated vector as a candidate vector to a second data table, and determine a result vector set corresponding to the search vector according to the second data table.
[0032] In some embodiments, adding the deduplicated vector as a candidate vector to the second data table includes:
[0033] Determine information of the deduplicated vector, the information including an identifier, a reading status, and a comparison status;
[0034] Add the information of the deduplicated vector to the second data table.
[0035] Here, the ID (Identity document) is a unique identification number for each vector.
[0036] The read status indicates whether the vector has been read from the disk into the memory. When adding to the second data table, the memory address of the vector data can be used to indicate that it has been read; for unread vectors, the read status can be empty or any number (such as 0) to indicate that it has not been read.
[0037] The comparison status indicates whether the vector has been compared with the search vector. When added to the second data table, any character or letter may be used to represent it, for example, 1 indicates that the vector has been compared, and 0 indicates that the vector has not been compared.
[0038] Here, the information of the deduplicated vectors is recorded in the second data table, and the result vector set corresponding to the search vector can be determined directly based on the second data table. In this way, the disk reading operation can be skipped for the vectors that have been read, and the comparison operation can be skipped for the vectors that have been compared, so as to avoid repeated operations and thus avoid wasting precious input and output (I / O) resources and vector computing resources.
[0039] In some embodiments, determining a result vector set corresponding to the search vector according to the second data table includes:
[0040] Determine a candidate vector to be compared according to information of each candidate vector;
[0041] The candidate vector to be compared is read from a disk or a memory, and the candidate vector to be compared is compared with the search vector.
[0042] Here, according to the comparison status of each candidate vector, it can be determined which candidate vectors have been compared and which candidate vectors have not been compared. For the compared candidate vectors, the comparison operation can be skipped and the candidate vectors to be compared can be directly determined.
[0043] For the candidate vectors to be compared, it is believed that the vector data may have been read from the disk and saved in the memory, or the vector data may not have been read from the disk into the memory. It can be determined whether to read the candidate vectors to be compared from the disk or the memory in combination with the reading status of each candidate vector to be compared, and then compare the candidate vectors to be compared with the retrieval vector.
[0044] Here, reading the candidate vector to be compared from the disk means reading the vector data of the candidate vector to be compared from the disk, and the read vector data can also be saved in the memory for subsequent comparison with the search vector. In addition, the second data table can also be updated, specifically, the memory address of the vector data is added to the second data table.
[0045] In this way, the disk reading operation is skipped for the read vectors and the comparison operation is skipped for the compared vectors according to the second data table, thereby avoiding repeated operations and wasting precious input and output resources and vector computing resources.
[0046] Here, determining the result vector set corresponding to the search vector according to the second data table may further include:
[0047] Determine a comparison result corresponding to each candidate vector, wherein the comparison may be performed by distance calculation based on an approximate nearest neighbor rule, and the comparison result is the distance between the candidate vector and the search vector, and the distance may reflect the similarity between the two;
[0048] Sort each candidate vector by distance from small to large, and select M cluster center vectors that are closest to the search vector (i.e., most similar). M can be any positive integer, which is set based on user needs.
[0049] In some embodiments, reading the candidate vector to be compared from a disk includes:
[0050] Determine a vector in the second data table that is in the same cluster as the candidate vector to be compared;
[0051] Determine a read address range of the candidate vector to be compared and the vector of the same cluster;
[0052] The candidate vector to be compared and the vectors of the same cluster are read from a disk according to the read address range.
[0053] Here, the read address of each candidate vector can be obtained by querying a pre-established index file, and the index file may include: an identifier of each vector and a read address corresponding to each vector.
[0054] Here, when determining the vectors in the second data table that are in the same cluster as the candidate vector to be compared, the first data table can be combined, that is, the first data table is used to determine the vectors in the same cluster as the candidate vector to be compared, and then which ones are in the second data table are further determined. In this way, the vectors in the second data table that are in the same cluster as the candidate vector to be compared can be determined.
[0055] Of course, the second data table can also record the mapping relationship between the cluster center vector and the vector within the cluster. For example, when the information of the deduplicated vector is added to the second data table, the identifier of the cluster center vector corresponding to each vector is also added to the second data table as the information of the vector. When applied, the vector in the second data table that is in the same cluster as the candidate vector to be compared can be directly determined according to the second data table.
[0056] Here, considering that the vector data in the same cluster will be stored continuously, if there are multiple vectors in a single cluster that need to be read from the disk, the disk reading operations can be merged to reduce the number of disk readings. Specifically, after determining the candidate vector to be compared, the vector in the second data table that is in the same cluster as the candidate vector to be compared can be determined, and read from the disk according to the read address range of the candidate vector to be compared and other vectors in the same cluster. In this way, multiple candidate vectors can be read at a time, thereby reducing the number of disk readings.
[0057] In some embodiments, the method further comprises:
[0058] Filtering the N cluster center vectors according to a third data table to obtain filtered cluster center vectors; the third data table is used to record vectors to be filtered and / or deleted;
[0059] Accordingly, the first data table is queried according to the N cluster center vectors to determine multiple vectors within the N clusters, including:
[0060] The first data table is queried according to the filtered cluster center vector to determine a plurality of vectors within the filtered cluster.
[0061] Here, the third data table may include the identifier of the vector to be filtered and / or the identifier of the vector to be deleted. After determining the N cluster center vectors, the third data table may be queried according to the identifier of each cluster center vector. If it is consistent with the identifier in the third data table, it is determined to be the cluster center vector to be filtered or deleted. If it is different from the identifier in the third data table, it is considered to be a valid vector, which can be retained and added to the result vector set corresponding to the search vector.
[0062] The third data table may be a bit set generated by the search device based on a filter query or vector deletion function, recording the identifiers of vectors not to be considered, and may also include vectors selected by the user to be deleted during the search process.
[0063] In this way, by filtering, relevant vectors can be excluded in the process of calculating the number of valid vectors and generating a result vector set, thereby avoiding the appearance of vectors to be filtered and / or deleted in the final result vector set.
[0064] In some embodiments, the method further comprises:
[0065] The multiple vectors in the N clusters are filtered according to the third data table to obtain filtered vectors; the third data table is used to record the vectors to be filtered and / or the deleted vectors.
[0066] Here, before performing a brute force search on the multiple vectors included in the N clusters, vector filtering may be performed according to the third data table to exclude vectors that are not to be considered, thereby avoiding the presence of vectors to be filtered and / or deleted in the final result vector set.
[0067] It should be noted that after filtering the N cluster center vectors, that is, querying the first data table according to the filtered cluster center vectors to determine multiple vectors in the filtered cluster, the multiple vectors in the filtered cluster can be filtered according to the third data table to obtain the filtered vectors.
[0068] In some embodiments, querying the second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors, determining the deduplicated vectors, and adding the deduplicated vectors as candidate vectors to the second data table may include:
[0069] For N cluster center vectors, query the second data table in turn according to the identifier of each cluster center vector, and if the identifier of the cluster center vector is not repeated with the identifier of the vector in the second data table, add the cluster center vector as a candidate vector to the second data table;
[0070] For the multiple vectors, the second data table is queried in turn according to the identifier of each vector, and if the identifier of the vector in the second data table is not repeated, the vector is added to the second data table as a candidate vector.
[0071] Specifically, the multiple vectors within the N clusters include: one or more vectors within each of the N clusters.
[0072] Here, a cluster usually includes: a cluster center vector and multiple non-center vectors.
[0073] Among them, the cluster center vector is the representative point of the cluster, which represents the overall characteristics of the cluster; the non-center vectors are other specific data points in the cluster, which are close to the cluster center vector and have similar characteristics or attributes.
[0074] Therefore, the composition of a cluster can be understood as a center (cluster center vector) and multiple members (non-center vectors), which together define the properties and shape of the cluster.
[0075] Here, considering that the inverted index usually puts the boundary vector or all vectors repeatedly into multiple adjacent clusters to improve the recall rate, the above-mentioned deduplication method based on the second data table is provided. Of course, in addition to the above-mentioned method, it is also possible to first determine whether there is a duplication based on the identifiers of the N cluster center vectors and the identifiers of multiple vectors. If there is a duplication, the repeated part is first removed. If there is no duplication, the second data table is queried in turn to determine the deduplicated vector.
[0076] In this way, in the mixed query scenario of memory and disk, the N clusters closest to the search vector are first found in the memory, and then the vectors in each cluster are read into the memory in sequence and queried one by one. At the same time, the problem of repeated loading and repeated calculation caused by repeated vectors is avoided, and the waste of I / O resources and vector calculation resources is avoided.
[0077] In some embodiments, the method further comprises:
[0078] If the total number of candidate vectors in the second data table is less than the target threshold, determining other cluster center vectors that the search vector matches;
[0079] According to the center vectors of the other clusters, the first data table is searched to determine a plurality of vectors in the other clusters;
[0080] According to the center vector of the other cluster and the multiple vectors in the other cluster, a second data table is searched to determine the vector after deduplication;
[0081] The deduplicated vector is added to the second data table as a candidate vector.
[0082] Here, considering that the number of vectors in the second data table may not be sufficient to obtain a result vector set, the number of vectors in the second data table can be judged. If it is less than a preset threshold, the number of cluster center vectors can be readjusted, and then the number of deduplicated vectors can be adjusted.
[0083] For example, after determining the N cluster center vectors as above, the N cluster center vectors are the N cluster center vectors closest to the search vector, and K other cluster center vectors can be further determined. The other cluster center vectors are the K cluster center vectors closest to the search vector except the N cluster center vectors.
[0084] In this way, by adjusting the number of cluster center vectors, the number of vectors after deduplication is increased, and a certain number of result vectors is provided as much as possible.
[0085] In some embodiments, the method further comprises:
[0086] Get the expected number of vectors and the multiple of the number of candidate vectors;
[0087] The target threshold is determined according to the expected number of vectors and a multiple of the number of candidate vectors.
[0088] Here, the expected number of vectors is recorded as topk, the candidate vector number multiple (candidate num ratio) is recorded as Num, and the target threshold can be topk*Num. This design takes into account that the number of candidate vectors is positively correlated with the recall rate, and has a relatively small relationship with the search vector. Using the candidate vector number multiple and topk to find a suitable target threshold can reduce the IO load while ensuring the recall rate and improve the search performance.
[0089] Therefore, it is proposed here to determine a target threshold by multiples of the expected number of vectors and the number of candidate vectors to obtain a result vector that meets the expected recall rate.
[0090] In some embodiments, determining N cluster center vectors that the search vector matches includes:
[0091] Obtain at least one cluster center vector according to the index file;
[0092] The at least one cluster center vector is compared with the search vector, and N cluster center vectors that are most similar to the search vector are determined according to the comparison result.
[0093] Here, the index file can be pre-constructed based on existing vectors, and the index file can be used to record multiple cluster center vectors and information of each cluster center vector. For example, the information of the cluster center vector includes: identification, memory address of vector data, cluster to which it belongs, cluster identification, etc.
[0094] After obtaining the search vector, the search vector is compared with each cluster center vector to obtain the comparison result of each cluster center vector, wherein the comparison can be based on the rule of approximate nearest neighbor to calculate the distance, and the distance between the search vector and the cluster center vector represents the similarity between the two. After sorting each cluster center vector from small to large in distance, select the N cluster center vectors with the closest distance to the search vector (i.e., the most similar). Among them, N can be any positive integer, and there is no limit on the value, such as N is 1, 2, 3, etc.
[0095] In this way, the cluster center vector closest to the search vector can be quickly determined based on the index file, and then the vectors in each cluster can be obtained, so as to subsequently determine the result vector set corresponding to the search vector.
[0096] Figure 2 A flowchart of a search method provided by an application embodiment of the present disclosure; Figure 1 As shown, the method includes:
[0097] Step 201, obtaining query data; here, the query data may include: query vector q, expected vector number (topk), expected candidate entity number multiple (Candidate entity num ratio), bit set (Bitset);
[0098] Here, the query vector q can be determined according to the content that the user needs to query.
[0099] The expected number of vectors may be determined based on specific application scenarios, user requirements and preferences, etc., or may be input by the user, and is not limited here.
[0100] The expected multiple of the number of candidate entities may be automatically generated by the retrieval system according to preset algorithms and strategies, and the specific situation depends on the design and application scenario of the retrieval system.
[0101] Bitset is a bit set that records the vectors that you want to delete or filter.
[0102] Step 202, determine N cluster center vectors; here, based on the rule of approximate nearest neighbor, find the N best cluster center vectors for the query vector q, and sort the cluster center vectors from small to large according to the distance.
[0103] Step 203, looping and processing N cluster center vectors;
[0104] Step 204: determine whether the cluster center vector appears for the first time in this query (here, it can be determined whether it is in the searchvectorMap); and also determine whether it has not been deleted or filtered (here, it can be determined whether this vector has been marked in the Bitset);
[0105] If the cluster center vector has not appeared in this query (not in the searchvectorMap) and has not been deleted or filtered (Bitset has not marked this vector), then go to step 205; otherwise, return to step 203 and start processing the next cluster center vector;
[0106] Step 205, adding the information of the cluster center vector to searchvectorMap; here, the identifier of the cluster center vector and the memory address of the vector data can be recorded in the searchvectorMap data structure, and at the same time, the cluster center vector is added to the result vector set (i.e., resultset, the minimum heap result vector set);
[0107] Step 206, looping in sequence to obtain the vectors Xi contained in the N cluster centers from postinglistvectorMap;
[0108] Step 207: determine whether the vector appears for the first time in this query (here, it can be determined whether it is in the searchvectorMap), and whether it has not been deleted or filtered (here, it can be determined whether the vector has been marked in the Bitset);
[0109] If the vector has not appeared in this query (not in searchvectorMap) and has not been deleted or filtered out (Bitset does not mark this vector), then go to step 208; if the vector has appeared in this query (in searchvectorMap, that is, a duplicate vector), or has been deleted or filtered out (bitset marks this vector), then delete the vector, that is, do not record the vector in searchvectorMap, return to step 206, and continue to process the next vector.
[0110] Step 208: Record the information of the vector in searchvectorMap; here, the identification, reading status and comparison status of the vector may be added to searchvectorMap.
[0111] Step 209, determine whether the number of vectors in searchvectorMap reaches a threshold;
[0112] In the above process, the number of vectors contained in searchvectorMap is determined to determine whether the number of vectors reaches a threshold, which may be an expected number (i.e., a multiple of the expected number of candidate entities*topk); if the number of vectors reaches the expected number, then proceed to step 210. If the vectors contained in the N clusters have been processed, but the number of vectors contained in searchvectorMap has not yet reached the expected number, then increase N, and N=N+topk, and return to step 202.
[0113] Step 210: Record the P cluster center vectors that have been processed so far.
[0114] Step 211, concurrently obtain the vectors Xi contained in P clusters from postinglistvectorMap;
[0115] Step 212: determine whether the vector Xi is in searchvectorMap and the memory address is empty;
[0116] If the vector Xi is in the searchvectorMap, and the vector in the searchvectorMap corresponds to the memory address of the vector (the vector Xi has been read into the memory), go to step 213;
[0117] If the vector Xi is in the searchvectorMap and the vector in the searchvectorMap does not correspond to a memory address of the vector (the vector Xi has not been read into the memory), then go to step 2141;
[0118] Step 213, determine whether the vector Xi has not been matched; if it has not been matched (there is no matching mark corresponding to the vector in searchvectorMap, or the matching mark indicates that it is not matched), go to step 215; if it has been matched, go to step 211.
[0119] Step 2141, determine whether there are vectors of the same cluster that have not been read from the disk, if so, proceed to step 2142, read multiple vectors at one time, if not, proceed to step 2143, read vector Xi;
[0120] Here, disk reading refers to reading this vector Xi from the disk index into the memory and recording the memory address of the vector into searchvectorMap.
[0121] Here, considering that the vector data of each vector in the same cluster will be stored continuously, if multiple vectors in a single cluster need to be read from the disk, the disk reading operations can be merged to reduce the number of disk reading times.
[0122] For example, if the memory addresses of vectors Xi, Xi+1, Xi+2, ..., Xi+m in the same cluster are not in the searchvectorMap, that is, they have not been read from the disk, then execute step 2142, specifically read the vector data of vectors Xi, Xi+1, Xi+2, ..., Xi+m from the disk index to the memory address addr1 at one time, add these vectors to the searchvectorMap, and set the addresses of the searchvectorMap vectors Xi, Xi+1, Xi+2, ..., Xi+m to addr1, addr1+sizeof(vector data), ..., addr1+m*sizeof(vector data), respectively.
[0123] If the same cluster vector is not considered or the same cluster vector has been read from disk, execute step 2143, specifically read the vector data from the disk index to the memory address addr1, add the vector to searchvectorMap, and set the memory address of the vector in searchvectorMap to addr1.
[0124] Step 215: compare with the query vector and add to the result vector set;
[0125] Here, the distance between the query vector q and this vector Xi can be calculated. As the comparison result, this vector is marked in searchvectorMap as having been compared and added to the result vector set.
[0126] Step 216 , determine whether the comparison of the vectors included in the P clusters is completed. If yes, proceed to step 217 , otherwise return to step 211 .
[0127] Step 217: Output the result vector set.
[0128] The solution provided by the embodiment of the present disclosure solves the problem of vector duplication by introducing two MAPs, postinglistvectorMap and searchvectorMap, during the query process. Among them, postinglistvectorMap can be constructed in the index loading stage to record the mapping relationship between the cluster center vector and the vectors within the cluster (such as the cluster center vector ID=>list of other vector IDs within the cluster). It can be constructed based on the disk index or the mapping relationship generated during the index construction process. SearchvectorMap is used to record which vectors have been loaded / compared during the query process; its key is the vector ID, and its value is the memory address of the vector data and whether the distance with the query vector has been calculated (i.e., the comparison mark or comparison status). During the query process, for each vector within the cluster, first determine whether the searchvectorMap contains this vector. If it contains this vector, it means that this vector has been considered before, and the disk reading operation and distance calculation operation of this vector can be skipped. This avoids repeated reading and / or comparison of the same vector, and solves the problem of wasting I / O resources and vector computing resources caused by brute force queries.
[0129] The disclosed embodiment also provides a method for constructing an index file, including: constructing a cluster tree for all vectors, selecting one or more appropriate cluster center nodes from the cluster tree according to specified rules, and saving the cluster center vector and its identifier in a cluster center vector file. Build an index for all cluster center nodes so that a suitable cluster center can be quickly found for the query variable, and save this index in the index file of the cluster center vector. For other non-cluster center vectors, search for the topx cluster center vectors through the index of the cluster center vector, add them to the clusters represented by these cluster center vectors, save the mapping relationship between these non-cluster center vectors and the cluster center vector, and save these non-cluster center vectors in a disk index file. For the convenience of subsequent analysis, the mapping relationship between the cluster center vector and other non-cluster center vectors can also be placed in an index file separately.
[0130] The above only describes an example of building an index file. In actual application, any method can be used to build the required index file, and there is no limitation here.
[0131] The disclosed embodiment provides a method for loading an index, including: loading a cluster center vector file and an index file of the cluster center vector into a memory, parsing a mapping relationship between the cluster center vector and other vectors and a file offset of the vector data from a disk index file, or directly reading the mapping relationship and the file offset of the vector data from a mapping index file, and putting them into a postinglistvectorMap data structure in the memory.
[0132] The present disclosure provides a disk index format, which may include the following parameters:
[0133] Disk index metadata: vector dimension, number of clusters, cluster metadata array file offset address.
[0134] The cluster metadata array may include:
[0135] The file offset address of the first vector of cluster A, the number of vectors in cluster A;
[0136] The file offset address of the first vector of cluster B, the number of vectors in cluster B;
[0137] …
[0138] And so on.
[0139] Each cluster contains data such as:
[0140] The data of cluster A may include: the ID of vector A1; the data of vector A1; the ID of vector A2; the data of vector A2; ... and so on.
[0141] The data of cluster B may include: the ID of vector B1; the data of vector B1; the ID of vector B2; the data of vector B2; ...; and so on.
[0142] The format of the cluster center vector and cluster intra-vector mapping file can be as follows:
[0143] The number of cluster centers;
[0144] The ID of cluster A, the address of the first vector of cluster A;
[0145] ID of vector A1;
[0146] ID of vector A2;
[0147] ..., and so on, vectors of other clusters A;
[0148] The ID of cluster B, the address of the first vector of cluster B;
[0149] ID of vector B1;
[0150] ID of vector B2;
[0151] ..., and so on, the vectors of other clusters B;
[0152] If there are other clusters, the format is the same as above.
[0153] Figure 3 A schematic diagram of the structure of a retrieval device provided by an embodiment of the present disclosure is shown; Figure 3 As shown, the retrieval device comprises:
[0154] An acquisition module, used to acquire a search vector and determine N cluster center vectors that match the search vector;
[0155] A first determination module, configured to query a first data table according to the N cluster center vectors to determine a plurality of vectors within the N clusters; the first data table is configured to record a mapping relationship between the cluster center vectors and the vectors within the clusters;
[0156] A second determination module, configured to query a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record information of at least one candidate vector;
[0157] A processing module is used to add the deduplicated vector as a candidate vector to a second data table, and determine a result vector set corresponding to the search vector according to the second data table.
[0158] In some embodiments, the processing module is used to determine information of the deduplicated vector, the information including an identifier, a reading status, and a comparison status;
[0159] Add the information of the deduplicated vector to the second data table.
[0160] In some embodiments, the processing module is used to determine the candidate vector to be compared based on the information of each candidate vector;
[0161] The candidate vector to be compared is read from a disk or a memory, and the candidate vector to be compared is compared with the search vector.
[0162] In some embodiments, the processing module is used to determine vectors in the second data table that are in the same cluster as the candidate vector to be compared;
[0163] Determine a read address range of the candidate vector to be compared and the vector of the same cluster;
[0164] The candidate vector to be compared and the vectors of the same cluster are read from a disk according to the read address range.
[0165] In some embodiments, the first determination module is further used to filter the N cluster center vectors according to a third data table to obtain filtered cluster center vectors; the third data table is used to record vectors to be filtered and / or deleted;
[0166] Accordingly, the first data table is queried according to the N cluster center vectors to determine multiple vectors within the N clusters, including:
[0167] The first data table is queried according to the filtered cluster center vector to determine a plurality of vectors within the filtered cluster.
[0168] In some embodiments, the first determination module is further used to filter the multiple vectors in the N clusters according to a third data table to obtain filtered vectors; the third data table is used to record the vectors to be filtered and / or deleted.
[0169] In some embodiments, the acquisition module is further used to determine other cluster center vectors that the search vector matches if the total number of candidate vectors in the second data table is less than a target threshold;
[0170] The first determination module is further used to query the first data table according to the center vectors of the other clusters to determine multiple vectors in the other clusters;
[0171] The second determination module is further used to query the second data table according to the other cluster center vector and the multiple vectors in the other cluster to determine the deduplicated vector;
[0172] The processing module is further configured to add the deduplicated vector as a candidate vector to the second data table.
[0173] In some embodiments, the acquisition module is further used to obtain the expected number of vectors and multiples of the number of candidate vectors;
[0174] The target threshold is determined according to the expected number of vectors and a multiple of the number of candidate vectors.
[0175] In some embodiments, the acquisition module is used to acquire at least one cluster center vector according to the index file;
[0176] The at least one cluster center vector is compared with the search vector, and N cluster center vectors that are most similar to the search vector are determined according to the comparison result.
[0177] It is understandable that when the retrieval device provided in the above embodiment implements the corresponding retrieval method, the above processing can be assigned to different program modules as needed to complete all or part of the above-described processing. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0178] An embodiment of the present disclosure provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be triggered to execute the retrieval method provided by the embodiment of the present disclosure.
[0179] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM; it may also be various devices including one or any combination of the above memories.
[0180] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, model, subroutine or other unit suitable for use in a computing environment.
[0181] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.
[0182] An embodiment of the present disclosure provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the retrieval method described in the present disclosure is implemented.
[0183] Figure 4 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 4As shown, the electronic device 40 includes: a processor 401 and a memory 402 for storing a computer program that can be run on the processor; when the processor 401 is used to run the computer program, it executes the retrieval method provided by the embodiment of the present disclosure.
[0184] In actual application, the electronic device 40 may further include: at least one network interface 403. The various components in the electronic device 40 are coupled together via a bus system 404. It is understood that the bus system 404 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 4 In the figure, various buses are labeled as bus system 404. There may be at least one processor 401. The network interface 403 is used for wired or wireless communication between the electronic device 40 and other devices.
[0185] The memory 402 in the embodiment of the present disclosure is used to store various types of data to support the operation of the electronic device 40 .
[0186] The method disclosed in the above embodiment of the present disclosure can be applied to the processor 401, or implemented by the processor 401. The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 401 or the instruction in the form of software. The above processor 401 can be a general processor, a digital signal processor (DSP, DiGital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiment of the present disclosure. The general processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a storage medium, which is located in the memory 402. The processor 401 reads the information in the memory 402 and completes the steps of the above method in combination with its hardware.
[0187] In some embodiments, the electronic device 40 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.
[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0189] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0190] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A retrieval method, comprising: Obtain a search vector, and determine N cluster center vectors that match the search vector; According to the N cluster center vectors, a first data table is queried to determine a plurality of vectors within the N clusters; the first data table is used to record a mapping relationship between the cluster center vectors and the vectors within the clusters; Querying a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record information of at least one candidate vector; The deduplicated vector is added as a candidate vector to a second data table, and a result vector set corresponding to the search vector is determined according to the second data table.
2. The method according to claim 1, adding the deduplicated vector as a candidate vector to the second data table, comprising: Determine information of the deduplicated vector, the information including an identifier, a reading status, and a comparison status; Add the information of the deduplicated vector to the second data table.
3. The method according to claim 1, determining the result vector set corresponding to the search vector according to the second data table, comprising: Determine a candidate vector to be compared according to information of each candidate vector; The candidate vector to be compared is read from a disk or a memory, and the candidate vector to be compared is compared with the search vector.
4. The method according to claim 3, reading the candidate vector to be compared from the disk, comprising: Determine a vector in the second data table that is in the same cluster as the candidate vector to be compared; Determine a read address range of the candidate vector to be compared and the vector of the same cluster; The candidate vector to be compared and the vectors of the same cluster are read from a disk according to the read address range.
5. The method according to claim 1, further comprising: Filtering the N cluster center vectors according to the third data table to obtain filtered cluster center vectors; The third data table is used to record vectors to be filtered and / or deleted; Accordingly, the first data table is queried according to the N cluster center vectors to determine multiple vectors within the N clusters, including: The first data table is queried according to the filtered cluster center vector to determine a plurality of vectors within the filtered cluster.
6. The method according to claim 1, further comprising: Filtering the multiple vectors in the N clusters according to the third data table to obtain filtered vectors; The third data table is used to record vectors to be filtered and / or deleted.
7. The method according to claim 1, further comprising: If the total number of candidate vectors in the second data table is less than the target threshold, determining other cluster center vectors that the search vector matches; According to the center vectors of the other clusters, the first data table is searched to determine a plurality of vectors in the other clusters; According to the center vector of the other cluster and the multiple vectors in the other cluster, a second data table is searched to determine the vector after deduplication; The deduplicated vector is added to the second data table as a candidate vector.
8. The method according to claim 7, further comprising: Get the expected number of vectors and the multiple of the number of candidate vectors; The target threshold is determined according to the expected number of vectors and a multiple of the number of candidate vectors.
9. The method according to claim 1, determining N cluster center vectors that the search vector matches comprises: Obtain at least one cluster center vector according to the index file; The at least one cluster center vector is compared with the search vector, and N cluster center vectors that are most similar to the search vector are determined according to the comparison result.
10. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute: Obtain a search vector, and determine N cluster center vectors that match the search vector; Querying a first data table according to the N cluster center vectors to determine a plurality of vectors within the N clusters; The first data table is used to record the mapping relationship between the cluster center vector and the cluster internal vector; Querying a second data table according to the identifiers of the N cluster center vectors and the identifiers of the multiple vectors to determine the vector after deduplication; the second data table is used to record at least one candidate vector; The deduplicated vector is added as a candidate vector to a second data table, and a result vector set corresponding to the search vector is determined according to the second data table.