A high-performance distributed mass vector similarity retrieval system and method

Abstract

The application discloses a kind of high-performance distributed mass vector similarity search system and method, including Faiss Driver, several Faiss Worker and Etcd system;Faiss Driver is used to receive the query request of user end, and the query request is forwarded to the selected Faiss Worker by proxy, and is configured to perform loading configuration Etcd monitoring, exit pre-processing, service discovery and master-slave election task;Faiss Worker is the unit for executing query calculation in search system, and is configured to perform configuration management, shutdown processing, timing task, load index and query calculation task, for calculating similar vector retrieval and updating online or offline index file;Etcd system is used to save Faiss Worker list, master-slave state of Faiss Worker, health status of Faiss Worker, external storage module address of index file and index file list.The application introduces proxy forwarding service mechanism and master-slave node registration mechanism, and can efficiently process vector query calculation request.

Description

Technical Field

[0001] This invention belongs to the field of vector retrieval technology, and specifically relates to a high-performance distributed massive vector similarity retrieval system and method. Background Technology

[0002] With the application of deep learning, the architecture of systems in applications such as image retrieval, face comparison, and navigation recommendation has evolved into architectures such as DSSM (Deep Structured Semantic Model, also known as the dual-tower model). These architectures use DNN (Deep Neural Network) to map user features to user feature vectors. These feature vectors can be generated based on embedding techniques to represent deep feature information. For example, in recommendation systems, users and items are mapped to the same vector space. Then, similarity calculations are used to index and recall the closest vectors, i.e., searching the item vector library for items with the most similar features (embedded) to recommend items that match the user's interests.

[0003] Faiss is an open-source AI similarity search tool that focuses on clustering and similarity search. It provides efficient similarity search and clustering for dense vectors, supporting searches of billions of vectors. It is currently a relatively mature near-neighbor search library, enabling developers to quickly search for similar multimedia files. Chinese invention patent CN109711298A discloses a method and system for efficient face feature value retrieval based on Faiss. Based on the open-source AI similarity search tool Faiss, a face retrieval module is developed. It retrieves the unique identifier and corresponding comparison score of the face image with the highest similarity score to the face to be compared from the face feature value storage module, and then performs a search result judgment to accelerate the face feature value retrieval speed. Simultaneously, during the retrieval process, the face feature value database is continuously optimized to reduce highly repetitive face images and select the most representative face images.

[0004] This method uses the Faiss tool to efficiently calculate the similarity of vector features, thus aiding in face comparison. However, since the Faiss tool is memory-based, this method does not support horizontal scaling. In recommendation scenarios with large amounts of data, the vector library will be massive, and the vector dimension can typically reach 512, 1024, or even larger. The operations involved in this method alone cannot meet the needs of fast similar vector retrieval services. Summary of the Invention

[0005] This invention provides a high-performance distributed massive vector similarity retrieval system and method, aiming to solve the problems of existing technologies not supporting horizontal expansion and insufficient performance in the case of massive vector databases.

[0006] To solve the above-mentioned technical problems, the retrieval system proposed in this invention is based on the Faiss database, including FaissDriver, several Faiss Workers, and the Etcd system;

[0007] The Faiss Driver is used to receive query requests from the user and forward the query requests to the selected Faiss Worker. It is configured to perform tasks such as loading configuration Etcd listener, exiting preprocessing, service discovery, and master-slave election.

[0008] The Faiss Worker is a unit in the retrieval system that performs query calculations. It is configured to perform tasks including configuration management, shutdown handling, scheduled tasks, index loading, and query calculations. It is used to calculate similar vector retrieval and update online or offline index files.

[0009] The Etcd system is used to store the Faiss Worker list, the master-slave status of the Faiss Workers, the health status of the Faiss Workers, the external storage module address of the index files, and the list of index files.

[0010] Preferably, the retrieval system further includes a training module, which is a Faiss Trainer, used to train the vector retrieval model and provide the trained index file to the Faiss Worker.

[0011] Preferably, the loading configuration for Etcd monitoring specifically involves: the Faiss Driver loading the Etcd service address and monitoring the master-slave changes and lifespan status of the Faiss Worker in Etcd;

[0012] The pre-exit processing is as follows: when the service fails and is shut down or restarted due to human factors or server resource issues, the Faiss Driver triggers a notification to the Faiss Worker.

[0013] The service discovery specifically involves: Faiss Driver monitoring the status of Faiss Worker and forwarding retrieval requests;

[0014] The master-slave election operation is as follows: Faiss Driver elects a master node based on the status of the Faiss Worker cluster, registers the master node, and updates the Faiss Worker master-slave node addresses in Etcd.

[0015] Preferably, the master-slave election uses the Raft algorithm.

[0016] Preferably, the Faiss Driver is further configured to perform a log initialization operation, specifically: the FaissDriver initializes a log file when the service starts, which is used to record information and request operations related to Faiss Driver requests.

[0017] Preferably, the configuration management specifically involves: Faiss Worker loading or updating the configuration file to obtain the storage address of the current cluster index file, the list of index files, and the index file itself;

[0018] The shutdown process specifically involves: when the Faiss Worker receives a shutdown signal from the Faiss Driver, it triggers a shutdown process and saves the index file;

[0019] The scheduled task is as follows: Faiss Worker periodically reports its status to Faiss Driver;

[0020] The loading of the index specifically refers to the Faiss Worker loading the index file first when starting to execute the query calculation task.

[0021] Preferably, the configuration files loaded or updated in the configuration management task are stored in the Etcd system.

[0022] Preferably, the heartbeat time of the timed task is set to 3 seconds.

[0023] Accordingly, this invention also proposes a high-performance distributed massive vector similarity retrieval method, which uses the above-mentioned retrieval system and includes the following steps:

[0024] S1: The client specifies the name of the vector library to be retrieved and the vector dimension, and sends a query request to Faiss Driver through the interface provided by Faiss Driver;

[0025] S2: The Faiss Driver sends the query task to the selected Faiss Worker for vector retrieval calculation based on the Faiss Worker status stored in the Etcd system.

[0026] S3: The Faiss Worker receives the task, loads the index file, and returns the results of the vector retrieval calculation to the FaissDriver, which then forwards them to the client.

[0027] Preferably, the factors for selecting the Faiss Worker in step S2 include:

[0028] A service discovery mechanism, based on the Etcd or Consul service registry, enables FaissDriver to dynamically discover currently available Faiss Worker nodes.

[0029] The load balancing strategy involves the Faiss Driver distributing query requests to different Faiss Workers based on the current load and performance metrics of the Faiss Worker nodes.

[0030] Before selecting a Faiss Worker, the Faiss Driver performs a health check on the nodes, including whether the Faiss Worker is online and whether it can respond to query requests normally. Nodes that are malfunctioning are excluded from the load balancer selection range.

[0031] The query routing strategy assigns specific query tasks to specific FaissWorkers based on the set routing strategy.

[0032] Compared with the prior art, the present invention has the following technical effects:

[0033] 1. The massive vector similarity retrieval system proposed in this invention introduces a proxy forwarding service mechanism and a master-slave node registration mechanism. The Faiss Driver acts as a proxy service for the Faiss Server. The Faiss Driver is not responsible for the actual vector similarity calculation or the update and synchronization of online and offline index files. Its primary function is to forward query requests. By forwarding query calculation requests through the Faiss Driver, multiple distributed FaissWorkers execute the specific query calculations, efficiently handling vector query calculation tasks. This system is suitable for massive vector similarity retrieval and efficiently processes query calculation requests based on vector features. It is not limited to business scenarios such as computer vision (CV) image processing, natural language processing (NLP), and recommendation algorithms, and provides horizontal scalability.

[0034] 2. The vector similarity retrieval system proposed in this invention uses Faiss Driver to select Faiss Workers based on factors including service discovery mechanism, load balancing strategy, node health status, and query routing strategy. It can flexibly match suitable Faiss Workers to perform retrieval calculations. Even if individual Faiss Workers fail due to faults, it can still guarantee similar vector retrieval services and improve the reliability of the entire system.

[0035] 3. The vector similarity retrieval system proposed in this invention configures a shutdown handling task on the Faiss Worker to perform processing operations when a Faiss Worker service becomes unavailable or unexpectedly shuts down, ensuring accurate and reliable retrieval services and improving the fault tolerance of the retrieval system. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the retrieval system described in this invention;

[0037] Figure 2 This is a flowchart of the retrieval method described in this invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0039] Example 1

[0040] like Figure 1 The diagram shown is a schematic of the structure of a high-performance distributed massive vector similarity retrieval system proposed in this invention. The retrieval system is based on the Faiss database and includes a Faiss Driver, several Faiss Workers, and an Etcd system.

[0041] The Faiss Driver is used to receive query requests from the user and forward the query requests to the selected Faiss Worker. It is configured to perform tasks such as loading configuration Etcd listener, exiting preprocessing, service discovery, and master-slave election.

[0042] The Faiss Worker is the unit that performs query calculations in the retrieval system. It is a key component responsible for the actual similarity vector retrieval calculations and updating online / offline index files. It is configured to perform tasks including configuration management, shutdown handling, scheduled tasks, index loading, and query calculations. In a single Faiss vector database, multiple Faiss Workers may work collaboratively, processing different data slices or performing different computational tasks. This design helps improve performance and efficiency, especially when dealing with large-scale datasets.

[0043] The Etcd system is used to store the Faiss Worker list, the master-slave status of the Faiss Workers, the health status of the Faiss Workers, the external storage module address of the index files, and the index file list. Etcd is an open-source distributed key-value store designed for reliable storage and retrieval of key / value pairs, supporting highly available and consistent distributed systems. The Etcd system records the Faiss Worker list, the master-slave status of the Faiss Workers, the health status of the Faiss Workers, the external storage module address of the index files, and the index file list. The loading and configuration of Etcd monitoring specifically involves the Faiss Driver loading the Etcd service address and monitoring the master-slave changes and lifespan status of the Faiss Workers in Etcd. Specifically, first, you need to import the Etcd client library. For example, if you are using Go, you can use etcd-client-go. Then, you need to initialize the Etcd client. During the initialization process, you need to create an Etcd client instance and configure the connection parameters, including the address of the Etcd service. Next, you need to use the API provided by the Etcd client library to listen for changes in data state. For example, you can use the Watch function to create a monitor so that you are notified when a specified Etcd key-value pair changes.

[0044] The retrieval system also includes a training module, namely Faiss Trainer, which is used to train the vector retrieval model and provide the trained index file to Faiss Worker.

[0045] For Faiss Driver, the loading configuration for Etcd listening specifically means: Faiss Driver loads the Etcd service address and listens for master-slave changes and lifespan status of Faiss Workers in Etcd.

[0046] The pre-exit processing is as follows: when the service fails and is shut down or restarted due to human factors or server resource issues, the Faiss Driver triggers a notification to the Faiss Worker.

[0047] The service discovery specifically involves the Faiss Driver monitoring the status of Faiss Workers and forwarding retrieval requests. Specifically, the Faiss Driver can monitor the health of Faiss Workers, their ability to provide retrieval computing services, and the status of their related computing resources, allowing the Faiss Driver to make timely adjustments and appropriately forward retrieval query requests.

[0048] The master-slave election operation specifically involves: the Faiss Driver electing a master node based on the state of the Faiss Worker cluster, registering the master node, and updating the Faiss Worker master-slave node addresses in Etcd. The master-slave election uses the Raft algorithm. In Etcd, master-slave election is implemented through the Raft consensus protocol. The Raft protocol defines the communication methods and consensus algorithms between nodes, ensuring system consistency under normal operation and node failure conditions. Each node in the Etcd cluster is in different Raft states, including Follower, Candidate, and Leader. When a Follower node deems the current Leader node unreachable (e.g., in this embodiment, the original Faiss Worker master node is unreachable due to network failure or resource scheduling reasons), it becomes a Candidate and begins a new election; simultaneously, the Candidate node sends voting requests to other nodes. During the election, each node can only vote for one candidate. If a candidate receives votes from a majority of nodes, it becomes the new leader. This process ensures that there is only one leader node in the system. After the election, the new leader node is responsible for accepting write requests from clients and replicating these write operations as log entries to other nodes. Once a majority of nodes have confirmed these log entries, the system achieves consistency.

[0049] The Faiss Driver is also configured to perform a log initialization operation, specifically: the Faiss Driver initializes a log file when the service starts, which is used to record information and request operations related to Faiss Driver requests.

[0050] For Faiss Worker, query computation is the main function for calculating vector feature similarity. Faiss Worker's vector similarity calculation includes two types: one calculates and returns only the similar vector IDs, and the other calculates and returns the similarity index IDs and corresponding similarity scores. It's worth noting that before calculating similar vectors, Faiss Worker checks whether the input vector features and their index names match the vector dimensions stored in the vector library in the index file to avoid calculation errors due to incorrect input parameters.

[0051] The configuration management specifically involves: Faiss Worker loading or updating configuration files, which are stored in the Etcd system; and obtaining the storage address, list, and specific index files of the current cluster index files. Storing configuration files in the Etcd system allows for dynamic updates of configuration information without stopping the system service storing index files.

[0052] The shutdown process is as follows: when the Faiss Worker receives a shutdown signal from the Faiss Driver, it triggers a shutdown process and saves the index file.

[0053] The scheduled task is as follows: the Faiss Worker periodically reports its status to the Faiss Driver. In this embodiment, the heartbeat interval of the scheduled task is set to 3 seconds.

[0054] The loading of the index specifically involves Faiss Worker loading the index file first when it starts executing a query computation task. In other embodiments of the invention, the index file is configured to be stored in a distributed file system. It is understood that the index file includes an offline local index file, an offline distributed index file, and an online index file. The online index file is used by Faiss Worker during query computation and inference; the offline distributed index file is the latest index file retrieved by Faiss Worker from the distributed file system; and the offline local index file is the latest index file updated and stored by Faiss Worker after updating the online index file. The online and offline local index files are used to separately save the index file in memory when Faiss Worker becomes unavailable due to network or other reasons. This is for subsequent synchronization of the index file and avoids the problem of unrecoverable index file corruption due to the unavailability of a particular Faiss Worker service.

[0055] The retrieval system configuration in this embodiment includes a monitoring and data visualization interface for the data visualization platform to provide data. Specifically, the data visualization platform uses Grafana to visualize data, monitoring information, and alarm information for users. Grafana is an open-source monitoring and data visualization platform that allows users to query, visualize, and understand various metrics, logs, and data sources. Grafana provides dashboard functionality, allowing users to create custom dashboards to display the data they care about. This platform is widely used in monitoring, alarm, logging, and visualization.

[0056] Example 2

[0057] like Figure 2As shown, this embodiment is a high-performance distributed massive vector similarity retrieval method. The method uses the retrieval system described in Embodiment 1 and includes the following steps:

[0058] S1: The client specifies the name of the vector library to be searched and the vector dimension, and sends a query request to Faiss Driver through the interface provided by Faiss Driver.

[0059] S2: The Faiss Driver sends the query task to the selected Faiss Worker for vector retrieval calculation based on the Faiss Worker status stored in the Etcd system.

[0060] S3: The Faiss Worker receives the task, loads the index file, and returns the results of the vector retrieval calculation to the FaissDriver, which then forwards them to the client.

[0061] Specifically, the Faiss Worker receives vector retrieval tasks from the Faiss Driver. First, it performs a similarity search on the query vector using the loaded index and returns the similarity matching results. Then, it encapsulates the vector retrieval calculation results, usually in the form of a data structure. In this embodiment, JSON format is used for encapsulation. The encapsulated results include the similarity matching vectors and their related information. The Faiss Driver receives the results from the Faiss Worker and forwards the results returned by the Faiss Worker to the client. The forwarding also uses the gRPC communication protocol to ensure that the client can obtain the results in a timely manner.

[0062] In this embodiment, the communication protocol between the client and the Faiss Driver is gRPC. The gRPC communication protocol specifies the name of the vector library index to be searched, the number of similar vector IDs returned, and the similarity score.

[0063] The factors for selecting Faiss Worker in step S2 of this embodiment include:

[0064] A service discovery mechanism, based on the Etcd or Consul service registry, enables FaissDriver to dynamically discover currently available Faiss Worker nodes.

[0065] The load balancing strategy involves the Faiss Driver distributing query requests to different Faiss Workers based on the current load and performance metrics of the Faiss Worker nodes.

[0066] Before selecting a Faiss Worker, the Faiss Driver performs a health check on the nodes, including whether the Faiss Worker is online and can respond to query requests normally, excluding faulty nodes from the load balancer selection range.

[0067] The query routing strategy assigns specific query tasks to specific FaissWorkers based on the set routing strategy.

[0068] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A high-performance distributed massive vector similarity retrieval system, the retrieval system being based on the Faiss database, characterized in that, This includes Faiss Driver, several Faiss Workers, and the Etcd system; The Faiss Driver is used to receive query requests from the user and forward the query requests to the selected Faiss Worker. It is configured to perform tasks such as loading configuration Etcd listener, exiting preprocessing, service discovery, and master-slave election. The loading configuration for Etcd monitoring specifically involves: Faiss Driver loading the Etcd service address and monitoring the master-slave changes and survival status of FaissWorker in Etcd; The pre-exit processing is as follows: when the service fails and is shut down or restarted due to human factors or server resource issues, the Faiss Driver triggers a notification to the Faiss Worker. The service discovery specifically involves: Faiss Driver monitoring the status of Faiss Worker and forwarding retrieval requests; The master-slave election operation is as follows: Faiss Driver elects a master node based on the status of the Faiss Worker cluster, registers the master node, and updates the Faiss Worker master-slave node addresses to Etcd. The Faiss Worker is a unit in the retrieval system that performs query calculations. It is configured to perform tasks including configuration management, shutdown handling, scheduled tasks, index loading, and query calculations. It is used to calculate similar vector retrieval and update online or offline index files. The configuration management specifically involves: Faiss Worker loading or updating the configuration file to obtain the storage address of the current cluster index file, the list of index files, and the index file itself. The shutdown process specifically involves: when the Faiss Worker receives a shutdown signal from the Faiss Driver, it triggers a shutdown process and saves the index file; The scheduled task is as follows: Faiss Worker periodically reports its status to Faiss Driver; The loading of the index specifically refers to the Faiss Worker loading the index file first when starting to execute the query calculation task; The Etcd system is used to store the Faiss Worker list, the master-slave status of the Faiss Workers, the health status of the Faiss Workers, the external storage module address of the index files, and the list of index files.

2. The high-performance distributed massive vector similarity retrieval system according to claim 1, characterized in that, The retrieval system also includes a training module, namely Faiss Trainer, which is used to train the vector retrieval model and provide the trained index file to Faiss Worker.

3. The high-performance distributed massive vector similarity retrieval system according to claim 1, characterized in that, The master-slave election uses the Raft algorithm.

4. The high-performance distributed massive vector similarity retrieval system according to claim 1, characterized in that, The Faiss Driver is also configured to perform a log initialization operation, specifically: the Faiss Driver initializes a log file when the service starts, which is used to record information and request operations related to Faiss Driver requests.

5. A high-performance distributed massive vector similarity retrieval system according to claim 1, characterized in that, The configuration files loaded or updated in the configuration management task are stored in the Etcd system.

6. The high-performance distributed massive vector similarity retrieval system according to claim 1, characterized in that, The heartbeat interval for the scheduled task is set to 3 seconds.

7. A high-performance distributed massive vector similarity retrieval method, characterized in that, The method uses the retrieval system as described in any one of claims 1-6, and includes the following steps: S1: The client specifies the name of the vector library to be retrieved and the vector dimension, and sends a query request to Faiss Driver through the interface provided by Faiss Driver; S2: The Faiss Driver sends the query task to the selected Faiss Worker for vector retrieval calculation based on the Faiss Worker status stored in the Etcd system. S3: The Faiss Worker receives the task, loads the index file, and returns the results of the vector retrieval calculation to the Faiss Driver, which then forwards them to the client.

8. The high-performance distributed massive vector similarity retrieval method according to claim 7, characterized in that, The factors for selecting Faiss Worker in step S2 include: A service discovery mechanism, based on the Etcd or Consul service registry, enables FaissDriver to dynamically discover currently available Faiss Worker nodes. The load balancing strategy involves the Faiss Driver distributing query requests to different Faiss Workers based on the current load and performance metrics of the Faiss Worker nodes. Before selecting a Faiss Worker, the Faiss Driver performs a health check on the nodes, including whether the Faiss Worker is online and whether it can respond to query requests normally. Nodes that are malfunctioning are excluded from the load balancer selection range. The query routing strategy assigns specific query tasks to specific Faiss Workers based on the set routing strategy.

Images