Vector retrieval method and device, electronic equipment and computer storage medium

Abstract

The invention relates to the field of data retrieval, in particular to a vector retrieval method and device, electronic equipment and a storage medium. Obtaining a first similarity between the query vector and the compression index vector; the compression index vector is obtained by compressing the original vector according to a preset compression strategy in advance; obtaining a plurality of compression index vectors as candidate compression index vectors according to the first similarity; querying in a pre-constructed compression index table according to the candidate compression index vectors, and obtaining an original vector corresponding to each candidate compression index vector as a candidate original vector; obtaining a second similarity between the candidate original vector and the query vector; and obtaining at least one candidate original vector as a target vector according to the second similarity. Compared with the prior art, the method has the advantages that the original vectors are compressed, so that the resource bottleneck of retrieval is avoided, and the retrieval precision is improved through two times of retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data retrieval, and more particularly, to a vector retrieval method and device, electronic equipment and storage medium. BACKGROUND

[0002] Vector retrieval refers to a process of finding data corresponding to the closest vector pair according to the semantic requirements of a query vector from a vector database. The existing vector retrieval technology usually directly retrieves the query according to the requirements in the vector database. This retrieval method has high efficiency when the amount of data in the vector database is small, and when the amount of data required for retrieval is small. However, in today's highly informationized Internet environment, the data size in the vector database is constantly expanding, which causes a bottleneck in computer memory when using the existing vector retrieval technology for large-scale retrieval in the vector database, ultimately resulting in reduced retrieval efficiency.

[0003] Therefore, there is an urgent need for a technology that can effectively improve the retrieval efficiency of large-scale vector retrieval. SUMMARY

[0004] The present application provides a data retrieval method, device, electronic equipment and storage medium for effectively improving the efficiency of large-scale vector retrieval.

[0005] According to a first aspect of the present application, a vector retrieval method is provided, the method comprising: obtaining a query vector; obtaining a first similarity between the query vector and a compressed index vector; the compressed index vector is obtained by compressing an original vector according to a preset compression strategy in advance; obtaining a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity; querying the candidate compressed index vectors in a pre-constructed compressed index table to obtain original vectors corresponding to each of the candidate compressed index vectors as candidate original vectors; the compressed index table records the mapping relationship between the compressed index vector and the original vector; obtaining a second similarity between the candidate original vectors and the query vector; obtaining at least one candidate original vector as a target vector according to the second similarity.

[0006] Optionally, the compressed index vector is obtained by compressing the original vector according to the preset compression strategy in advance, comprising: dividing the original vector into a plurality of sub-vectors according to a preset order; dividing the sub-vectors of all original vectors into a plurality of vector groups; obtaining at least one reference vector of the vector group. In the vector group, a reference vector closest to the sub-vector is taken as an approximate vector of the sub-vector; The approximate vectors of all sub-vectors of the original vector are spliced to obtain a compressed index vector of the original vector; The compressed index vector of the original vector and the mapping relationship between the original vector and the compressed index vector are recorded, wherein if the compressed index vectors of multiple original vectors are the same, one of the same compressed index vectors is recorded and the multiple original vectors are respectively mapped to the recorded compressed index vector.

[0007] Optionally, the division of all sub-vectors of the original vector into a plurality of vector groups comprises: According to the division order of the sub-vector in the corresponding original vector, the corresponding sub-vectors of all original vectors are divided into a plurality of vector groups; wherein the sub-vectors of the same division order are divided into the same vector group.

[0008] Optionally, the obtaining of at least one reference vector of the vector group comprises: Clustering all sub-vectors in the vector group to obtain at least one vector cluster of the vector group; Obtaining the center vector of the vector cluster as the reference vector of the vector group.

[0009] Optionally, the obtaining of a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity comprises: The compressed index vectors are sorted according to the size of the first similarity; According to the sorted compressed index vectors, a plurality of compressed index vectors with the largest first similarity are obtained as the candidate compressed index vectors.

[0010] Optionally, the obtaining of at least one candidate original vector as a target vector according to the second similarity comprises: The candidate original vectors are sorted according to the size of the second similarity; According to the sorted candidate original vectors, at least one candidate original vector with the largest second similarity is obtained as the target vector.

[0011] Optionally, the compressed index vector is stored in a first resource storage area, and the compressed index table and the original vector are stored in a second resource storage area; The data access performance of the first resource storage area is better than that of the second resource storage area.

[0012] According to a second aspect of the present application, a vector retrieval device is provided, the device comprising: a query obtaining module configured to obtain a query vector; a first similarity calculation module configured to obtain a first similarity between the query vector and a compressed index vector; the compressed index vector is obtained by compressing an original vector according to a preset compression strategy in advance; a candidate compressed index obtaining module configured to obtain a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity; a candidate original vector obtaining module configured to query a compressed index table constructed in advance according to the candidate compressed index vectors, and obtain original vectors corresponding to the candidate compressed index vectors as candidate original vectors; the compressed index table records a mapping relationship between compressed index vectors and original vectors; a second similarity calculation module configured to obtain a second similarity between the candidate original vectors and the query vector; a target vector obtaining module configured to obtain at least one candidate original vector as a target vector according to the second similarity.

[0013] According to a third aspect of the present application, an electronic device is provided, comprising: a memory configured to store one or more computer programs; a processor, when the one or more computer programs are executed by the processor, implements the vector retrieval method of the first aspect described above.

[0014] According to a fourth aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer instructions, the computer instructions are used to make the processor execute and implement the vector retrieval method of the first aspect described above.

[0015] Based on any one of the above aspects, the vector retrieval method, device, electronic device and computer storage medium provided by the embodiments of the present application perform primary retrieval based on compressed original vectors, avoid direct retrieval in a large amount of data, can effectively avoid memory bottleneck, and further can effectively improve the efficiency of retrieval; the candidate compressed index vectors obtained by the primary retrieval are used for secondary retrieval with the original vectors, and the target vector can be accurately extracted from the compressed candidate compressed index vectors. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0017] Figure 1 A schematic application scenario diagram of the vector retrieval method provided in the present embodiment.

[0018] Figure 2 A step flowchart of the vector retrieval method provided in the present embodiment.

[0019] Figure 3 A step flowchart of the compressed index vector compression provided in the present embodiment.

[0020] Figure 4 A functional module diagram of the vector compression device provided in the present embodiment.

[0021] Figure 5 A structural diagram of the electronic device provided in the present embodiment. DETAILED DESCRIPTION

[0022] The drawings of the present application are only used for exemplary illustration, and cannot be understood as the limitation of the present application. In order to better illustrate the following embodiments, some components in the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures in the drawings and their descriptions can be omitted.

[0023] In order to make the person in the art better understand the present application scheme, the following will combine the drawings in the embodiments of the present application, and the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should belong to the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Vector retrieval refers to the process of finding the most similar vector from a vector database based on the semantic requirements of a query vector. Existing vector retrieval technologies typically involve directly searching the vector database based on the requirements, such as calculating the similarity between one or more query vectors and all stored vectors in the database, and then finding the vector with the highest similarity to the query vector.

[0026] This retrieval method is highly efficient when the amount of data in the vector database is small, and when the amount of data required for retrieval is small. However, in today's highly information-driven internet environment, the data scale in vector databases is constantly expanding. This causes computer memory to become a bottleneck when performing large-scale searches using existing vector retrieval technologies, ultimately leading to a decrease in retrieval efficiency.

[0027] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0028] This is an exemplary schematic diagram illustrating an application scenario of a vector retrieval method provided in an embodiment of this application. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.

[0029] In one possible implementation, server 100 and terminal 200 may each execute the vector retrieval method provided in the embodiments of this application. Alternatively, the vector retrieval method provided in the embodiments of this application may be executed partly in server 100 and partly in terminal 200.

[0030] like Figure 2As shown, the embodiment provides a vector retrieval method, which can include the following steps: S1: obtaining a query vector; In the embodiment, the query vector is used for querying in a preset vector database, wherein the vector database pre-stores original vectors to be queried. It can be understood that the query vector matches the format of the original vectors in the vector database, and the query vector can be one or more; further, in the embodiment, a large-scale vector can be input for querying, and the large-scale vector contains a large number of query vectors.

[0031] S2: obtaining a first similarity between the query vector and a compressed index vector; In the embodiment, the compressed index vector is obtained by compressing original vectors according to a preset compression strategy; the compressed index vector has a certain mapping relationship with the original vectors in the vector database. One compressed index vector can map multiple original vectors; in the embodiment, the compressed index vector can be understood as the common features of all original vectors it maps, or as the compressed index vector being similar to all original vectors it maps.

[0032] Therefore, the original vectors in the vector database that have a certain similarity relationship with the query vector can be obtained by calculating the first similarity between the query vector and the compressed index vector. In addition, the compressed index vector can be stored in a compressed vector database different from the vector database, and since the compressed index vector maps multiple original vectors, the number of compressed index vectors stored in the compressed vector database is less than the number of original vectors in the vector database. Therefore, querying the query vector in the compressed index vector can effectively avoid the resource bottleneck of querying and improve the efficiency of querying, even if the number of query vectors is large.

[0033] In a real-time manner, as shown in Figure 3 The compressed index vector is obtained by compressing original vectors according to a preset compression strategy, which can include: A1: dividing the original vectors into several sub-vectors uniformly according to a preset order; In the embodiment, the original vectors can be divided into several sub-vectors in a preset order according to the dimensions of the original vectors, that is, the number of the divided sub-vectors is determined according to the dimensions of the original vectors. The dimensions of the original vectors in the vector database are the same. It can be understood that the original vectors need to be preprocessed when being stored in the vector database, so that the dimensions of all the original vectors stored in the vector database are uniform. The preprocessing can include padding, truncation, feature extraction conversion, etc., which are not limited in the embodiment.

[0034] In an example, assuming that a 128-dimensional vector is divided into four sub-vectors in order, four sub-vectors of each division order are obtained, and the dimensions of each sub-vector of each division order are 32 dimensions.

[0035] By unifying the dimensions of the sub-vectors, subsequent grouping calculation can be better facilitated.

[0036] A2: dividing all the sub-vectors of the original vectors into several vector groups; In the embodiment, the step A2 can include: According to the division order of the sub-vectors in the corresponding original vectors, the sub-vectors corresponding to all the original vectors are divided into several vector groups; wherein the sub-vectors of the same division order are divided into the same vector group.

[0037] In an example, assuming that each of the original vectors is divided into four sub-vectors in a preset order, that is, each of the original vectors contains a sub-vector of division order 1, a sub-vector of division order 2, a sub-vector of division order 3, and a sub-vector of division order 4 after division. Then, the sub-vectors of division order 1 in each of the original vectors are divided into the same vector group, the sub-vectors of division order 2 are divided into another vector group, and so on.

[0038] By dividing the sub-vectors of the same division order into the same vector group, all the sub-vectors in the vector group can always express consistent semantic or feature dimensions, thereby ensuring the alignment of the sub-vectors in structure and facilitating subsequent calculation.

[0039] A3: obtaining at least one reference vector of the vector group; In the embodiment, the step A3 can include: Clustering all the sub-vectors in the vector group to obtain at least one vector cluster of the vector group; and obtaining the center vector of the vector cluster as the reference vector of the vector group.

[0040] By clustering in the vector group, the internal relations between the sub-vectors in the vector group can be further discovered, and similar sub-vectors can be classified, so that all the sub-vectors in each vector cluster have a certain similarity, and the reference vector as the center vector of the vector cluster has a certain similarity with all the sub-vectors in the corresponding vector cluster. It can be understood that if a vector is similar to the reference vector, the vector is similar to all the sub-vectors in the vector cluster corresponding to the reference vector, and therefore the reference vector can be used to represent all the sub-vectors in the corresponding vector cluster.

[0041] In the embodiment, the clustering can be implemented by using a k-means clustering method, which is an unsupervised learning algorithm and is mainly used to automatically divide a group of unlabeled data into K clusters so that the data in the same cluster is as similar as possible. In the embodiment, the value of K can be set in advance.

[0042] In the embodiment, the calculation of the center vector of the vector cluster can be implemented by using a nearest neighbor search or a mean center calculation, which is not further limited herein.

[0043] A4: In the vector group, a reference vector closest to the sub-vector is taken as an approximate vector of the sub-vector; In the embodiment, as described above, since the reference vector is the center vector of the vector cluster in the vector group, the reference vector can represent each sub-vector in the vector cluster, and the reference vector closest to the sub-vector is the center vector of the same vector cluster as the sub-vector. By taking the reference vector closest to the sub-vector as the approximate vector of the sub-vector, the sub-vector of each original vector can be "replaced" by the corresponding approximate vector, so that multiple sub-vectors can be represented by one approximate vector, and data compression is achieved.

[0044] A5: The approximate vectors of all the sub-vectors of the original vector are spliced to obtain a compressed index vector of the original vector; As described above, the approximate vector can represent multiple sub-vectors, and therefore, by splicing the approximate vectors corresponding to all the sub-vectors of the original vector, the compressed index vector can represent the original vector and can also be mapped with multiple original vectors, so that vector compression is achieved.

[0045] A6: Record the compressed index vector of the original vector and the mapping relationship between the original vector and the compressed index vector, wherein if the compressed index vectors of multiple original vectors are the same, one of the same compressed index vectors is recorded and the multiple original vectors are respectively mapped with the recorded compressed index vector.

[0046] As described above, since the compressed index vector can map multiple original vectors, when recording the compressed index vector, for the case that one compressed index vector maps multiple original vectors, only this one compressed index vector needs to be stored, and a large amount of original data is compressed into a smaller compressed index vector, effectively realizing the compression of vector data.

[0047] For example, assuming that there are the following 5 original vectors of 12 dimensions: Original vector v1=[12,3,4,5,6,7,8,9,10,11,12] Original vector v2=[1,2,3,5,5,6,7,9,9,10,11,13] Original vector v3=[8,9,10,11,1,2,3,4,5,6,7,8] Original vector v4=[8,9,10,12,1,2,3,5,5,6,7,9] Original vector v5=[2,3,4,5,6,7,8,9,10,11,12,13] Divide the 5 original vectors into 3 4-dimensional vectors respectively, denoted as Original vector v1: [1,2,3,4][5,6,7,8][9,10,11,12] Original vector v2: [1,2,3,5][5,6,7,9][9,10,11,13] Original vector v3: [8,9,10,11][1,2,3,4][5,6,7,8] Original vector v4: [8,9,10,12][1,2,3,5][5,6,7,9] Original vector v5: [2,3,4,5][6,7,8,9][10,11,12,13] The sub-vectors of each partition order of the five original vectors are respectively partitioned into the same vector group, taking the sub-vectors of the partition order 1 as an example, the obtained vector group includes the five sub-vectors: v1.1: [1, 2, 3, 4], v1.2: [1, 2, 3, 5], v1.3: [8, 9, 10, 11], v1.4: [8, 9, 10, 12] and v1.5: [2, 3, 4, 5], the five sub-vectors in the vector group are clustered to obtain two vector clusters, one vector cluster includes the sub-vectors v1.1, v1.2 and v1.5 (representing small values), and the other vector cluster includes the sub-vectors v1.3 and v1.4, the center vector of the first vector cluster is calculated as [1.3, 2.3, 3.3, 4.5], and the center vector of the second vector cluster is calculated as [8.5, 9.5, 10.5, 11.5].

[0048] In the same way, the center vectors of the vector groups corresponding to the partition order 2 are calculated as [5.3, 6.3, 7.3, 8.7] and [1.0, 2.0, 3.0, 4.5], and the center vectors of the vector groups corresponding to the partition order 3 are calculated as [9.0, 10.0, 11.0, 12.5], [5.0, 6.0, 7.0, 8.5] and [10, 11, 12, 13].

[0049] Then, for the original vector v1, the distance between the sub-vectors of the partition order 1 and the center vectors [1.3, 2.3, 3.3, 4.5] and [8.5, 9.5, 10.5, 11.5] of the corresponding vector group is calculated, the center vector [1.3, 2.3, 3.3, 4.5] is closest to the sub-vectors of the partition order 1, so the reference vector of the sub-vectors of the partition order 1 is [1.3, 2.3, 3.3, 4.5], similarly, the sub-vectors of the partition order 2 are closest to the center vector [5.3, 6.3, 7.3, 8.7] of the corresponding vector group, so the reference vector of the sub-vectors of the partition order 2 is [5.3, 6.3, 7.3, 8.7], and the sub-vectors of the partition order 3 are closest to the center vector [9.0, 10.0, 11.0, 12.5] of the corresponding vector group, so the reference vector of the sub-vectors of the partition order 3 is [9.0, 10.0, 11.0, 12.5].

[0050] After obtaining the reference vectors corresponding to the sub-vectors of the original vector v1, the compressed index vectors of the original vector v1 can be obtained by concatenating the reference vectors in the order of the partition. In one preferred embodiment, in order to facilitate the representation, after obtaining the compressed index vectors, the integer bits of the values in the compressed index vectors can be retained and the decimal bits can be removed. For example, the compressed index vectors of the original vector v1 after processing are [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. By removing the decimal bits, the calculation can be facilitated.

[0051] It can be understood that the calculation of the original vector v2, the original vector v3, the original vector v4 and the original vector v5 can refer to the calculation of the original vector v1, which will not be described further. The compressed index vectors of the original vector v2 are [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], the compressed index vectors of the original vector v3 are [8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8], the compressed index vectors of the original vector v4 are [8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8], and the compressed index vectors of the original vector v5 are [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13].

[0052] The compressed index vectors of the original vector v1 and the original vector v2 are the same, and the compressed index vectors of the original vector v3 and the original vector v4 are the same. Therefore, when recording the compressed index vectors, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], [8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8] and [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13] can be recorded. The [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] is mapped to the original vector v1 and the original vector v2, the [8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8] is mapped to the original vector v3 and the original vector v4, and the [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13] is mapped to the original vector v5. By mapping the five original vectors to three compressed index vectors, the compression of the vector data storage is achieved.

[0053] S3: obtaining a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity; In the embodiment, the step S3 can include: ordering the compressed index vectors according to the size of the first similarity; and obtaining a plurality of compressed index vectors with the largest first similarity as the candidate compressed index vectors according to the ordered compressed index vectors.

[0054] By ordering the compressed index vectors according to the size of the first similarity, the candidate compressed index vectors can be obtained more intuitively and conveniently.

[0055] S4: querying the candidate compressed index vectors in a pre-constructed compressed index table to obtain original vectors corresponding to the candidate compressed index vectors as candidate original vectors. It can be understood that the compressed index table records the mapping relationship between the compressed index vectors and the original vectors. The mapping relationship can be obtained in the process of calculating the compressed index vectors in the above steps A1-A6, which will not be described further. By querying the candidate original vectors using the pre-constructed compressed index table, one or more candidate original vectors corresponding to the candidate compressed index vectors can be quickly obtained.

[0056] Since the compressed index vectors are obtained by "compression" although they are similar to the original vectors, they cannot be equivalent to the original vectors. Therefore, in the embodiment, more candidate original vectors can be obtained in the step S3, the data amount for subsequent screening is improved, and the accuracy of the query is further improved.

[0057] S5: obtaining a second similarity between the candidate original vectors and the query vector. As described above, since the compressed index vectors cannot be equivalent to the original vectors, the candidate original vectors obtained from the candidate compressed index vectors can not be accurately matched with the query vector. Therefore, after the candidate original vectors are extracted, the second similarity between the query vector and the candidate original vectors is further calculated, and the target vector matched with the query vector is extracted from the candidate original vectors.

[0058] S6: obtaining at least one candidate original vector as a target vector according to the second similarity.

[0059] In the embodiment, the step S6 can include: The candidate original vectors are sorted according to the sizes of the second similarities; and according to the sorted candidate original vectors, at least one candidate original vector with the largest second similarity is obtained as the target vector.

[0060] By sorting the candidate original vectors according to the sizes of the second similarities, the target vector can be more intuitively and conveniently obtained.

[0061] In a preferred embodiment, the compressed index vectors can be stored in a first resource storage area, i.e., the compressed index vector library can be arranged in the first resource storage area, and the compressed index table and the original vectors can be stored in a second resource storage area. The data access performance of the first resource storage area is superior to that of the second resource storage area. For example, the first resource storage area can be arranged in the memory of a computer system, and the second resource storage area can be arranged in the disk of the computer system. By arranging the compressed query with a small amount of data in the first resource storage area with high data access performance, the query efficiency for the compressed index vectors can be effectively improved, and the candidate compressed index vectors after the query can be queried in the second resource storage area, so that the bottleneck of the memory can be avoided, and the query efficiency can be improved.

[0062] As shown in Figure 4 The embodiment of the present application further provides a vector retrieval device. Optionally, the vector retrieval device can include A query obtaining module 11 is configured to obtain a query vector. In the embodiment, the query obtaining module 11 can be configured to perform step S1 shown in Figure 2 The specific description of the query obtaining module 11 can refer to the description of step S1.

[0063] A first similarity calculating module 12 is configured to obtain a first similarity between the query vector and a compressed index vector. In the embodiment, the first similarity calculating module 12 can be configured to perform step S2 shown in Figure 2 The specific description of the first similarity calculating module 12 can refer to the description of step S2.

[0064] A candidate compressed index obtaining module 13 is configured to obtain a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity. In the embodiment, the candidate compressed index obtaining module 13 can be configured to perform step S3 shown in Figure 2 The specific description of the candidate compressed index obtaining module 13 can refer to the description of step S3.

[0065] The candidate original vector obtaining module 14 is configured to query the pre-constructed compressed index table according to the candidate compressed index vector, and obtain an original vector corresponding to each candidate compressed index vector as a candidate original vector. In the embodiment, the candidate original vector obtaining module 14 can be configured to perform Figure 2 The step S4 is illustrated, and the specific description of the candidate original vector obtaining module 14 can refer to the description of the step S4.

[0066] The second similarity calculating module 15 is configured to obtain a second similarity between the candidate original vector and the query vector. In the embodiment, the second similarity calculating module 15 can be configured to perform Figure 2 The step S5 is illustrated, and the specific description of the second similarity calculating module 15 can refer to the description of the step S5.

[0067] The target vector obtaining module 16 is configured to obtain at least one candidate original vector as a target vector according to the second similarity.

[0068] In the embodiment, the target vector obtaining module 16 can be configured to perform Figure 2 The step S6 is illustrated, and the specific description of the target vector obtaining module 16 can refer to the description of the step S6.

[0069] In a preferred embodiment, the device can further comprise a vector compressing module 17 configured to compress the original vector to obtain the compressed index vector.

[0070] In the embodiment, the vector compressing module 17 can be configured to perform Figure 3 The steps A1-A6 are illustrated, and the specific description of the vector compressing module 17 can refer to the description of the steps A1-A6.

[0071] It can be understood that the above-mentioned device embodiments and the above-mentioned method embodiments can correspond to each other, and the similar description of the device embodiments can refer to the method embodiments. To avoid repetition, it will not be described here. The vector retrieval device provided in the embodiment of the present application can perform the vector retrieval method provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of performing the method. The function modules of the vector retrieval device can be realized by hardware, realized by software instructions, or realized by a combination of hardware and software modules.

[0072] In particular, each step of the method embodiments of the present application can be completed by integrated logic circuits of hardware in the processor and / or instructions in the form of software, the steps of the vector retrieval method of the embodiments of the present application can be directly embodied as hardware coding processor execution completion, or combined with hardware and software modules in the coding processor to complete execution. Alternatively, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps in the above method embodiments in combination with the hardware thereof.

[0073] The electronic device provided by the embodiments of the present application has a structure as shown in Figure 5 The electronic device can be the server 100 or the terminal 200 shown in the embodiments of the present application. Figure 1

[0074] The electronic device includes a memory 21, a processor 22, a communication module 23, and an input / output interface 24, etc. Optionally, the memory 21, the processor 22, the communication module 23, and the input / output interface 24 can be connected and communicated through a bus 25.

[0075] The memory 21 is used to store one or more computer programs and transmit the codes of the computer programs to the processor 22; when the one or more computer programs are executed by the processor 22, the vector retrieval method in the embodiments of the present application is realized.

[0076] Optionally, the electronic device can be connected to a network through the communication module 23 to communicate with other devices such as terminals or servers through the network to realize the interaction of data. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0077] ​Optionally, the electronic device can connect the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 24. The electronic device itself can have a display device, and can also be externally connected to other display devices through the input / output interface 24. Optionally, the storage device, such as a hard disk, etc., can also be connected through the input / output interface 24, so that the data in the electronic device can be stored in the storage device, or the data in the storage device can be read, and the data in the storage device can also be stored in the memory 21. It can be understood that the input / output interface 24 can be a wired interface or a wireless interface. According to different actual application scenarios, the devices connected with the input / output interface 24 can be a component of the electronic device, or can be an external device connected with the electronic device when needed.

[0078] Optionally, the memory 21 can be a volatile memory and / or a non-volatile memory. The volatile memory can be a random access memory, etc., and the non-volatile memory can be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0079] Optionally, the computer program stored in the processor 22 can be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to complete the method provided in the embodiment. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0080] Optionally, the processor 22 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 22 include but are not limited to a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and any appropriate controller, microcontroller, processor, etc. The processor 22 executes various methods and processes of the embodiment, exemplarily, such as a vector retrieval method of the embodiment of the present application.

[0081] Optionally, the bus 25 can include a channel for transmitting information. According to different functions, the bus 25 can be divided into an address bus, a data bus, a control bus, etc.

[0082] In an alternative implementation, the embodiments of the present application further provide a computer storage medium, which stores a computer program. The computer program is executed by a computer to enable the computer to perform the method of the above-mentioned method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 21 of the electronic device. When the computer program is executed by the processor 22, one or more steps of a vector retrieval method of the embodiments of the present application can be performed.

[0083] Optionally, the computer readable storage medium can be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0084] Obviously, the above-mentioned embodiments of the present application are merely examples for clearly illustrating the technical solutions of the present application, and are not intended to limit the specific implementation of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A vector retrieval method characterized by, The method comprises: obtaining a query vector; obtaining a first similarity between the query vector and a compressed index vector; the compressed index vector is obtained by compressing an original vector according to a preset compression strategy in advance; obtaining a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity; querying the candidate compressed index vectors in a pre-constructed compressed index table to obtain original vectors corresponding to the candidate compressed index vectors as candidate original vectors; the compressed index table records the mapping relationship between the compressed index vector and the original vector; obtaining a second similarity between the candidate original vectors and the query vector; obtaining at least one candidate original vector as a target vector according to the second similarity.

2. The vector retrieval method of claim 1, wherein, The compressed index vector is obtained by compressing an original vector according to a preset compression strategy in advance, comprising: dividing the original vector into a plurality of sub-vectors according to a preset order; dividing all the sub-vectors of the original vector into a plurality of vector groups; obtaining at least one reference vector of the vector group; in the vector group, the reference vector closest to the sub-vector is taken as the approximate vector of the sub-vector; splicing the approximate vectors of all the sub-vectors of the original vector to obtain the compressed index vector of the original vector; record the compressed index vector of the original vector and the mapping relationship between the original vector and the compressed index vector, wherein if the compressed index vectors of a plurality of original vectors are the same, one of the same compressed index vectors is recorded and a plurality of original vectors are respectively mapped to the recorded compressed index vector.

3. The vector retrieval method of claim 2, wherein, The sub-vectors of all the original vectors are divided into a plurality of vector groups, comprising: according to the division order of the sub-vector in the corresponding original vector, the sub-vectors corresponding to all the original vectors are divided into a plurality of vector groups; wherein the sub-vectors of the same division order are divided into the same vector group.

4. The vector retrieval method of claim 2, wherein, The at least one reference vector of the vector group is obtained, comprising: clustering all the sub-vectors in the vector group to obtain at least one vector cluster of the vector group; obtaining the center vector of the vector cluster as the reference vector of the vector group.

5. The vector retrieval method according to any one of claims 1-4, wherein, According to the first similarity, a plurality of compressed index vectors are obtained as candidate compressed index vectors, comprising: sorting the compressed index vectors according to the size of the first similarity; according to the sorted compressed index vectors, a plurality of compressed index vectors with the largest first similarity are obtained as the candidate compressed index vectors.

6. The vector retrieval method according to any one of claims 1-4, wherein, According to the second similarity, at least one candidate original vector is obtained as a target vector, comprising: sorting the candidate original vectors according to the size of the second similarity; according to the sorted candidate original vectors, at least one candidate original vector with the largest second similarity is obtained as the target vector.

7. The vector retrieval method according to any one of claims 1-4, characterized by, The compressed index vector is stored in a first resource storage area, and the compressed index table and the original vector are stored in a second resource storage area; the data access performance of the first resource storage area is better than that of the second resource storage area.

8. A vector retrieval apparatus characterized by comprising: The device comprises: a query acquisition module, configured to acquire a query vector; a first similarity calculation module, configured to acquire a first similarity between the query vector and a compressed index vector; the compressed index vector is obtained by compressing an original vector according to a preset compression strategy in advance; a candidate compressed index acquisition module, configured to acquire a plurality of compressed index vectors as candidate compressed index vectors according to the first similarity; a candidate original vector acquisition module, configured to acquire original vectors corresponding to the candidate compressed index vectors as candidate original vectors by querying a compressed index table constructed in advance according to the candidate compressed index vectors; the compressed index table records a mapping relationship between compressed index vectors and original vectors; a second similarity calculation module, configured to acquire a second similarity between the candidate original vectors and the query vector; a target vector acquisition module, configured to acquire at least one candidate original vector as a target vector according to the second similarity.

9. An electronic device, comprising: comprise: a memory, configured to store one or more computer programs; a processor, when the one or more computer programs are executed by the processor, implements the vector retrieval method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the processor execute and implement the vector retrieval method as claimed in any one of claims 1-7.

Images