Similarity search method and device

By allocating and indexing the feature vectors of the target to be matched on the computer cluster, and using similarity search tools for positioning search, the problems of low similarity search efficiency and large memory usage in the existing technology are solved, and large-scale data is realized. Efficient search.

CN113761017BActive Publication Date: 2024-12-03BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110119512.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-28
Publication Date
2024-12-03
Estimated Expiration
2041-01-28

AI Technical Summary

Technical Problem

When existing similarity search systems process billions of targets to be matched and millions of query targets, they are slow to create indexes, inefficient query efficiency, and consume a lot of memory. They cannot support offline queries or limited memory. scene.

Method used

By assigning the feature vectors of the target to be matched to multiple computing nodes in the computer cluster by category, creating an index based on the similarity search tool, and establishing a mapping relationship between the category and the index partition, positioning the index partition of the query target for searching, using Similarity search tools conduct parallel query to improve search efficiency and realize the ready-to-use and release of memory resources in Spark cluster.

Benefits of technology

It realizes efficient similarity search for billions of targets to be matched and millions of query targets, reducing search time, avoiding long-term memory usage, and supporting offline queries of large-scale data and memory finite memory scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113761017B_ABST
    Figure CN113761017B_ABST
Patent Text Reader

Abstract

The present invention discloses a similarity search method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining feature vectors of multiple targets to be matched, and distributing the feature vectors to multiple computing nodes in a computer cluster according to the categories to which the targets to be matched belong; creating an index for the feature vectors of each category of targets to be matched at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; establishing a mapping relationship between categories and index partitions; obtaining the feature vector of a query target, and determining the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; searching using the feature vector of the query target in the determined index partition to obtain a preset first number of targets to be matched that are similar to the query target. This implementation can support similarity searches for billions of targets to be matched and millions of query targets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a similarity search method and device. Background Art

[0002] Similarity search refers to obtaining one or more targets that are most similar to the input query target from a large number of targets to be matched. In the current similarity search field, due to technical reasons such as slow index creation and low query QPS (Queries Per Second), it is impossible to support searches for billions of targets to be matched and millions of query targets. At the same time, the existing similarity search system will occupy a large amount of memory for a long time, making it impossible to apply it to offline query services or scenarios with limited memory. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a similarity search method and device, which can improve search efficiency and support similarity search for billions of to-be-matched targets and millions of query targets.

[0004] To achieve the above objective, according to one aspect of the present invention, a similarity search method is provided.

[0005] The similarity search method of the embodiment of the present invention includes: obtaining feature vectors of multiple targets to be matched, and distributing the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; creating an index for the feature vectors of the targets to be matched in each category at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; establishing a mapping relationship between each category and the index partition where the feature vectors of the targets to be matched in the category are located; obtaining the feature vector of the query target, and determining the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; searching in the determined index partition using the feature vector of the query target to obtain a preset first number of targets to be matched that are similar to the query target as search results.

[0006] Optionally, the index is an index based on a preset similarity search tool; and the searching using the feature vector of the query target in the determined index partition comprises: for any determined index partition, inputting the feature vector of the query target into the similarity search tool for searching, and obtaining candidate targets in the index partition and the similarity between the candidate targets and the query targets; wherein the candidate targets are a preset second number of targets to be matched that have the greatest similarity with the query targets in the index partition.

[0007] Optionally, obtaining a preset first number of to-be-matched targets similar to the query target as search results includes: arranging the candidate targets in each determined index partition in descending order according to the similarity, and determining the first number of candidate targets arranged in front as search results.

[0008] Optionally, the computer cluster is a Spark cluster, the similarity search tool is Faiss, and the similarity is a similarity based on Euclidean distance or vector inner product.

[0009] To achieve the above objective, according to another aspect of the present invention, a similarity search device is provided.

[0010] The similarity search device of the embodiment of the present invention may include: an allocation unit, used to obtain feature vectors of multiple targets to be matched, and allocate the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; an index creation unit, used to create an index for the feature vectors of the targets to be matched in each category at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; a mapping relationship is established between each category and the index partition where the feature vectors of the targets to be matched in the category are located; an index partition determination unit, used to obtain the feature vector of the query target, and determine the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; a search unit, used to search using the feature vector of the query target in the determined index partition, and obtain a preset first number of targets to be matched that are similar to the query target as search results.

[0011] Optionally, the index is an index based on a preset similarity search tool; and the search unit can be further used to: for any determined index partition, input the feature vector of the query target into the similarity search tool for search, and obtain the candidate targets in the index partition and the similarity between the candidate targets and the query targets; wherein the candidate targets are a preset second number of targets to be matched in the index partition with the greatest similarity to the query targets.

[0012] Optionally, the search unit may be further configured to: arrange the candidate targets in each determined index partition in descending order according to the similarity, and determine a first number of candidate targets arranged in front as search results.

[0013] Optionally, the computer cluster is a Spark cluster, the similarity search tool is Faiss, and the similarity is a similarity based on Euclidean distance or vector inner product.

[0014] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0015] An electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the similarity search method provided by the present invention.

[0016] To achieve the above objective, according to yet another aspect of the present invention, a computer-readable storage medium is provided.

[0017] A computer-readable storage medium of the present invention stores a computer program, and when the program is executed by a processor, the similarity search method provided by the present invention is implemented.

[0018] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects:

[0019] Before executing the search, the feature vector of the target to be matched is pre-assigned to multiple computing nodes of a computer cluster (such as a Spark cluster) according to the category to which the target to be matched belongs, and an index based on a similarity search tool (such as Faiss) is created for the feature vector of the target to be matched of each computing node, and a mapping relationship between the category to which the target to be matched belongs and the corresponding index partition is established. When executing the search, one or more index partitions are first determined according to the category to which the query target belongs and the above-mentioned mapping relationship, and then a query is performed in each index partition based on the similarity search tool, and finally the query results of each index partition determined are comprehensively determined to obtain the final similarity search results. In this way, the similarity search tool can be effectively combined with the computer cluster to improve the efficiency of the similarity search, and the index partition corresponding to the query target is first located before the search to reduce the search range, and the search is carried out in parallel on multiple computing nodes and multiple index partitions to improve the computing efficiency. In addition, since the embodiment of the present invention is implemented based on a computer cluster such as Spark, memory resources are applied for when they are used and released when they are not used, and no additional application resources are required, thereby solving the problem that the existing similarity search technology occupies a large amount of memory for a long time. According to the above configuration, the embodiment of the present invention can support similarity search for billions of to-be-matched targets and millions of query targets.

[0020] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0022] Figure 1 Schematic diagram of the main steps of the similarity search method in an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a specific implementation of a similarity search method in an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of components of a similarity search device in an embodiment of the present invention;

[0025] Figure 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0026] Figure 5 It is a schematic diagram of the structure of an electronic device used to implement the similarity search method in the embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0029] Figure 1 Schematic diagram of main steps of a similarity search method according to an embodiment of the present invention.

[0030] like Figure 1 As shown, the similarity search method of the embodiment of the present invention can be specifically performed according to the following steps:

[0031] Step S101: Acquire feature vectors of multiple targets to be matched, and distribute the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong.

[0032] In the embodiment of the present invention, the target to be matched refers to the target used to match the input query target. The final search result of the similarity search exists in the target to be matched. The above target can be various objects such as items and users. The following will mainly use items as an example for explanation. For example, when it is necessary to search for similar items of an input item among billions of items, the billions of items are the target to be matched, and the input item is the query target.

[0033] In this step, first, the feature vector of each target to be matched can be obtained according to the name, description information and other data of each target to be matched, and the above feature vector can be used to identify the corresponding target to be matched. Specifically, the name, description information and other data of each target to be matched can be processed by word segmentation, and the data after word segmentation processing can be input into the pre-trained word vector generation model Word2vec, so as to obtain the feature vector of the target to be matched. It can be understood that the above word segmentation processing can adopt various existing word segmentation algorithms, and the feature vector can also be implemented by other vectorization models, and the present invention is not limited to this.

[0034] After obtaining the feature vector of the target to be matched, the feature vector can be distributed to multiple computing nodes in a preset computer cluster according to the category to which the target to be matched belongs. In specific applications, the above categories can be categories or categories of various granularities, and the category granularity used in this step can be set according to requirements. For example, for a women's windbreaker as the target to be matched, the categories of women's tops, women's clothing, and clothing of different granularities can all be used as its category.

[0035] The computer cluster in this step refers to a cluster that can perform big data computing and analysis, such as a Spark cluster (Spark is a known fast and general computing engine). The computer cluster is composed of multiple computing nodes, and the computing tasks are completed by the collaboration of each computing node. It can be understood that the above-mentioned computer cluster is not limited to the Spark cluster, but can also be a cluster based on a known computing engine such as Twister, HaLoop, etc.

[0036] In this step, allocation can be performed based on the preset maximum data processing capacity of a single computing node and the number of feature vectors contained in each category (the category to which the target to be matched belongs). After the allocation is completed, one computing node can correspond to one or more categories. For example, if the maximum data processing capacity of a single computing node is 1 million, and the data volume of category a is 1 million, then one computing node can be allocated to category a; if the data volumes of categories b and c are both 500,000, then one computing node can be allocated to categories b and c; if the data volume of category d is 2 million, then two computing nodes can be allocated to category d.

[0037] Step S102: create an index for the feature vector of each category of the target to be matched at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; establish a mapping relationship between each category and the index partition where the feature vector of the target to be matched in that category is located.

[0038] In this step, an index may be created for the data assigned to each computing node. In particular, the index is an index based on a preset similarity search tool. Exemplarily, the similarity search tool may be the known clustering and similarity search tool Faiss, or other applicable similarity search tools. The following description will be made by taking Faiss as an example.

[0039] Faiss's index can indicate the data segmentation and encoding methods and help implement specific queries. The index types include the following: IndexFlatL2 (index type based on Euclidean distance calculation), IndexFlatIP (index type based on dot product calculation), IndexHNSWFlat (index type based on layering), IndexIVFFlat (index type based on exact test), IndexLSH (index type based on local sensitive hashing), IndexScalarQuantizer (index type based on scalar quantization), IndexPQ (index type based on quantization), IndexIVFScalarQuantizer (index type of a combination method), IndexIVFPQ (index type of another combination method), IndexIVFPQR (optimized version of IndexIVFPQ).

[0040] Preferably, an index can be created for the feature vector data in each computing node based on the preset single index data volume, and the index type of each index can be determined according to actual needs. Continuing with the above example, if the single index data volume is 200,000, 5 indexes need to be created for the data of a computing node; if an exact match is required for 200,000 data of a certain category in a computing node, an IndexFlatL2 index can be created; if a fuzzy match is required for 200,000 data of a certain category in a computing node, an IndexLSH index can be created.

[0041] After the index is created, the feature vectors with the same index are in the same index partition corresponding to the index. The above index partition refers to the data storage space in the computing node corresponding to the same index. After that, a mapping relationship between each category and the index partition where the feature vector of the target to be matched in the category is located can be established to determine the index partition corresponding to each target to be matched and each query target. In practical applications, the following routing table can be used to represent the above mapping relationship.

[0042]

[0043]

[0044] Based on the above routing table, the index partition, index type and number of search results corresponding to any category (i.e., the number of candidate targets returned by the index partition, which will be described below) can be obtained. The index partition identifier is used to uniquely indicate the index partition, which can be composed of the corresponding computing node identifier and the index partition ID. For example, L1_03 indicates that the index partition is the index partition with ID 03 in the computing node L1. It can be understood that the above routing table can also store only the category field and the index partition identifier field.

[0045] Step S103: Acquire a feature vector of the query target, and determine the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship.

[0046] In an embodiment of the present invention, first, the name and description information of the query target can be segmented, and the data after the segmentation process can be input into a vectorization model such as Word2vec to obtain a feature vector of the query target. This action can be performed before, after or at the same time as step S101. Thereafter, the index partition corresponding to the category to which the query target belongs is determined in the above-mentioned routing table, and the subsequent data search is performed in the determined index partition. Through the above settings, the search range can be reduced, thereby improving the calculation efficiency.

[0047] Step S104: searching in the determined index partition using the feature vector of the query target to obtain a preset first number of to-be-matched targets similar to the query target as search results.

[0048] In this step, the similarity search tool Faiss can be used to search in each determined index partition to obtain the final search results. Specifically, for any determined index partition, the feature vector of the query target is input into the similarity search tool Faiss for search to obtain the candidate targets in the index partition and the similarity between each candidate target and the query target. The above-mentioned candidate targets are the preset second number of targets to be matched with the greatest similarity to the query target in the index partition, and the above-mentioned similarity can be expressed by a function of the Euclidean distance (such as the inverse) or a vector inner product. It can be understood that the larger the Euclidean distance, the smaller the similarity; the larger the vector inner product, the greater the similarity (when the vector is normalized).

[0049] After obtaining the candidate targets returned by each index partition, if the same candidate targets exist among the candidate targets returned by different index partitions, multiple candidate target data are first merged into one, and the similarity in the merged data can be the maximum similarity of the multiple data before the merge. After that, these candidate targets can be arranged in descending order according to the similarity, that is, if the Euclidean distance is used to represent the similarity, it is equivalent to arranging the candidate targets in ascending order according to the Euclidean distance; if the vector inner product is used to represent the similarity, it is equivalent to arranging the candidate targets in descending order according to the vector inner product. Finally, the first number of candidate targets arranged in front are determined as the final search results. The above first number may be equal to the second number, or may not be equal.

[0050] Through the above steps, efficient similarity search can be achieved based on the effective combination of similarity search tools and computer clusters, without the need to occupy memory support for a long time, thus meeting similarity search scenarios with billions of targets to be matched and millions of query targets.

[0051] The following will be combined Figure 2 An embodiment of the present invention is described.

[0052] like Figure 2 As shown in the figure, first, the names and descriptions of the target to be matched, the query target, and the training sample target can be segmented, and the Word2vec model can be trained using the segmented training sample target data. Then, the feature vectors of the target to be matched and the query target are generated based on the trained Word2vec and the pooling algorithm. After obtaining the above feature vectors, the feature vector data of the target to be matched can be filtered to remove the relevant data of the inactive target.

[0053] After that, the CPU (central processing unit), memory and other computing resources of the Spark cluster are determined according to the data volume and the number of categories of the target to be matched and the query target, as well as the matching method (exact match or fuzzy match), and the target to be matched and the query target are numbered (that is, the ID of the target to be matched and the query target is determined, and the function number of Faiss can be used, or the number can be customized), and the feature vectors of the target to be matched in each category are allocated to each computing node according to the maximum data processing volume of the computing node and the data volume of each category (the existing load balancing method can be used to perform the allocation), and an index is created for the feature vector data of the target to be matched of each node according to the single index data volume and index type, and finally a routing table is formed that reflects the mapping relationship between each category and the index partition where the feature vector of the target to be matched in this category is located. In the above process, the GroupByKey method of Spark can be used to classify the target to be matched. After that, the routing table can be used to determine the index partition corresponding to the query target, and the feature vector of the query target is input into Faiss for search. After each index partition search is completed, the candidate target data is returned, and finally the candidate target data of each index partition can be combined to obtain the final search result.

[0054] Through the above settings, this embodiment can meet the similarity search needs with billions of targets to be matched and millions of query targets, and at the same time realize the PaaS (Platform as a Service) of similarity search (that is, provide an interface of the similarity search method of the embodiment of the present invention to other application systems); in terms of memory resources, they are applied for when used and released when not used, without the need for additional application; in terms of effect, a default optimal search strategy is given through performance testing. After the user enters parameters such as the number of targets to be matched and the number of query targets, it can be used by configuring according to the default strategy prompted by the system.

[0055] After testing, when the number of targets to be matched is 1.5 billion, the number of query targets is 10,000, and the first number is 10, an accurate match can be completed within half an hour. When the number of targets to be matched is 1.5 billion, the number of query targets is 1 million, and the first number is 10, an accurate match can be completed within 2 hours. It can be seen that the similarity search method of the embodiment of the present invention can meet the similarity search requirements with billions of targets to be matched and millions of query targets, and can also realize the adaptive adjustment of search strategies and Spark cluster resources according to business scenarios.

[0056] In the technical solution of the embodiment of the present invention, before performing the search, the feature vector of the target to be matched is pre-assigned to multiple computing nodes of a computer cluster (such as a Spark cluster) according to the category to which the target to be matched belongs, and an index based on a similarity search tool (such as Faiss) is created for the feature vector of the target to be matched of each computing node, and a mapping relationship between the category to which the target to be matched belongs and the corresponding index partition is established. When performing the search, one or more index partitions are first determined according to the category to which the query target belongs and the above-mentioned mapping relationship, and then a query is performed in each index partition based on the similarity search tool, and finally the query results of each index partition determined are comprehensively determined to obtain the final similarity search results. In this way, the similarity search tool can be effectively combined with the computer cluster to improve the efficiency of the similarity search, and the index partition corresponding to the query target is first located before the search to reduce the search range, and the search is carried out in parallel on multiple computing nodes and multiple index partitions to improve the computing efficiency. In addition, since the embodiment of the present invention is implemented based on a computer cluster such as Spark, memory resources are applied for when they are used and released when they are not used, and no additional application resources are required, thereby solving the problem that the existing similarity search technology occupies a large amount of memory for a long time.

[0057] It should be noted that, for the convenience of description, the aforementioned method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, and some steps can actually be performed in other sequences or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary to implement the present invention.

[0058] In order to better implement the above-mentioned solution of the embodiment of the present invention, relevant devices for implementing the above-mentioned solution are also provided below.

[0059] See also Figure 3 As shown, the similarity search device 300 provided by the embodiment of the present invention may include: an allocation unit 301 , an index creation unit 302 , an index partition determination unit 303 and a search unit 304 .

[0060] The allocation unit 301 can be used to obtain feature vectors of multiple targets to be matched, and allocate the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; the index creation unit 302 can be used to create an index for the feature vectors of the targets to be matched in each category at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; a mapping relationship is established between each category and the index partition where the feature vectors of the targets to be matched in the category are located; the index partition determination unit 303 can be used to obtain the feature vector of the query target, and determine the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; the search unit 304 can be used to search using the feature vector of the query target in the determined index partition, and obtain a preset first number of targets to be matched that are similar to the query target as search results.

[0061] In an embodiment of the present invention, the index is an index based on a preset similarity search tool; and the search unit 304 can be further used to: for any determined index partition, input the feature vector of the query target into the similarity search tool for search, and obtain the candidate targets in the index partition and the similarity between the candidate targets and the query target; wherein the candidate targets are the preset second number of targets to be matched in the index partition with the greatest similarity to the query target.

[0062] As a preferred solution, the search unit 304 may be further configured to: arrange the candidate targets in each determined index partition in descending order according to the similarity, and determine the first number of candidate targets arranged in front as search results.

[0063] In addition, in the embodiment of the present invention, the computer cluster is a Spark cluster, the similarity search tool is Faiss, and the similarity is a similarity based on Euclidean distance or vector inner product.

[0064] In the technical solution of the embodiment of the present invention, before performing the search, the feature vector of the target to be matched is pre-assigned to multiple computing nodes of a computer cluster (such as a Spark cluster) according to the category to which the target to be matched belongs, and an index based on a similarity search tool (such as Faiss) is created for the feature vector of the target to be matched of each computing node, and a mapping relationship between the category to which the target to be matched belongs and the corresponding index partition is established. When performing the search, one or more index partitions are first determined according to the category to which the query target belongs and the above-mentioned mapping relationship, and then a query is performed in each index partition based on the similarity search tool, and finally the query results of each index partition determined are comprehensively determined to obtain the final similarity search results. In this way, the similarity search tool can be effectively combined with the computer cluster to improve the efficiency of the similarity search, and the index partition corresponding to the query target is first located before the search to reduce the search range, and the search is carried out in parallel on multiple computing nodes and multiple index partitions to improve the computing efficiency. In addition, since the embodiment of the present invention is implemented based on a computer cluster such as Spark, memory resources are applied for when they are used and released when they are not used, and no additional application resources are required, thereby solving the problem that the existing similarity search technology occupies a large amount of memory for a long time.

[0065] Figure 4 An exemplary system architecture 400 is shown to which the similarity search method or similarity search device according to the embodiment of the present invention can be applied.

[0066] like Figure 4 As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404 and a server 405 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0067] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as similarity search applications (only as an example).

[0068] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0069] The server 405 may be a server that provides various services, such as a backend server that supports similarity search applications operated by users using the terminal devices 401, 402, and 403 (for example only). The server 405 may process the received query request, etc., and feed back the processing result (for example, the similar target data found in the query - for example only) to the terminal devices 401, 402, and 403.

[0070] It should be noted that the similarity search method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the similarity search device is generally disposed in the server 405 .

[0071] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0072] The present invention also provides an electronic device. The electronic device of the embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the similarity search method provided by the present invention.

[0073] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of an electronic device suitable for implementing an embodiment of the present invention. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0074] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the computer system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0075] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0076] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the main step diagram. In the above embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit 501, the above functions defined in the system of the present invention are executed.

[0077] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0078] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0079] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an allocation unit, an index creation unit, an index partition determination unit, and a search unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the allocation unit may also be described as "a unit that provides feature vectors in computing nodes to the index creation unit".

[0080] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the steps executed by the device include: obtaining feature vectors of multiple targets to be matched, and assigning the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; creating an index for the feature vectors of the targets to be matched in each category at each computing node, and the feature vectors with the same index are in the same index partition corresponding to the index; establishing a mapping relationship between each category and the index partition where the feature vectors of the targets to be matched in the category are located; obtaining the feature vector of the query target, and determining the index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; searching in the determined index partition using the feature vector of the query target to obtain a preset first number of targets to be matched similar to the query target as search results.

[0081] In the technical solution of the embodiment of the present invention, before performing the search, the feature vector of the target to be matched is pre-assigned to multiple computing nodes of a computer cluster (such as a Spark cluster) according to the category to which the target to be matched belongs, and an index based on a similarity search tool (such as Faiss) is created for the feature vector of the target to be matched of each computing node, and a mapping relationship between the category to which the target to be matched belongs and the corresponding index partition is established. When performing the search, one or more index partitions are first determined according to the category to which the query target belongs and the above-mentioned mapping relationship, and then a query is performed in each index partition based on the similarity search tool, and finally the query results of each index partition determined are comprehensively determined to obtain the final similarity search results. In this way, the similarity search tool can be effectively combined with the computer cluster to improve the efficiency of the similarity search, and the index partition corresponding to the query target is first located before the search to reduce the search range, and the search is carried out in parallel on multiple computing nodes and multiple index partitions to improve the computing efficiency. In addition, since the embodiment of the present invention is implemented based on a computer cluster such as Spark, memory resources are applied for when they are used and released when they are not used, and no additional application resources are required, thereby solving the problem that the existing similarity search technology occupies a large amount of memory for a long time.

[0082] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A similarity search method, characterized in that: include: Acquire feature vectors of multiple targets to be matched, and distribute the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; Create an index for the feature vector of each category of the target to be matched at each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; establish a mapping relationship between each category and the index partition where the feature vector of the target to be matched in the category is located; Obtaining a feature vector of the query target, and determining an index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; The feature vector of the query target is used to search in the determined index partition to obtain a preset first number of to-be-matched targets similar to the query target as search results.

2. The method according to claim 1, characterized in that The index is an index based on a preset similarity search tool; and the searching in the determined index partition using the feature vector of the query target includes: For any determined index partition, the feature vector of the query target is input into the similarity search tool for search to obtain the candidate targets in the index partition and the similarity between the candidate targets and the query target; wherein the candidate targets are a preset second number of targets to be matched that have the greatest similarity with the query target in the index partition.

3. The method according to claim 2, characterized in that The step of obtaining a preset first number of to-be-matched targets similar to the query target as search results includes: Arrange the candidate targets in each determined index partition in descending order according to the similarity, The first number of candidate targets ranked at the front are determined as search results.

4. The method according to claim 2 or 3, characterized in that: The computer cluster is a Spark cluster, the similarity search tool is Faiss, and the similarity is a similarity based on Euclidean distance or vector inner product.

5. A similarity search device, characterized in that: include: An allocating unit, used to obtain feature vectors of multiple targets to be matched, and allocate the feature vectors to multiple computing nodes in a preset computer cluster according to the categories to which the targets to be matched belong; wherein each computing node corresponds to one or more categories; An index creation unit is used to create an index for the feature vector of each category of the target to be matched in each computing node, and feature vectors with the same index are in the same index partition corresponding to the index; a mapping relationship is established between each category and the index partition where the feature vector of the target to be matched in the category is located; An index partition determination unit, used to obtain a feature vector of a query target, and determine an index partition corresponding to the query target according to the category to which the query target belongs and the mapping relationship; A search unit is used to search in the determined index partition using the feature vector of the query target to obtain a preset first number of to-be-matched targets similar to the query target as search results.

6. The device according to claim 5, characterized in that The index is an index based on a preset similarity search tool; and the search unit is further used for: For any determined index partition, the feature vector of the query target is input into the similarity search tool for search to obtain the candidate targets in the index partition and the similarity between the candidate targets and the query target; wherein the candidate targets are a preset second number of targets to be matched that have the greatest similarity with the query target in the index partition.

7. The device according to claim 6, characterized in that The search unit is further configured to: The candidate targets in each determined index partition are arranged in descending order according to the similarity, and a first number of candidate targets arranged in front are determined as search results.

8. The device according to claim 6 or 7, characterized in that The computer cluster is a Spark cluster, the similarity search tool is Faiss, and the similarity is a similarity based on Euclidean distance or vector inner product.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.