Data retrieval method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a data retrieval method and device, electronic equipment and a readable storage medium, and the method comprises the steps: constructing target retrieval information based on a feature query vector corresponding to to-be-retrieved data and a target feature library identifier; based on the target feature library identifier, determining a target vector library cluster and a target storage database corresponding to the target feature library identifier; the target vector library cluster comprises N vector databases, and N is greater than or equal to 1; services corresponding to the vector databases are called concurrently, so that the services obtain vector retrieval results corresponding to the feature query vectors from the vector databases on the basis of the target retrieval information; different vector databases correspond to different services; and obtaining a target retrieval result corresponding to the feature query vector from a target storage database based on the vector retrieval result corresponding to each vector database. Through the integrated retrieval system, the retrieval efficiency is improved to a certain extent, and the deployment cost and the later maintenance cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data retrieval method, device, electronic equipment and readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence application technology, vector and structured data fusion retrieval has been widely used, covering almost many fields such as urban governance, campus, transportation, etc.

[0003] However, currently, data retrieval based on a single vector library instance, a single MongoDB database instance, and a single ClickHouse database instance can only solve simple scenario applications. Since the scalability of single library deployment is usually limited, it cannot meet the needs of tens of thousands of vector libraries and multi-tenant high concurrent access. Therefore, in order to meet the retrieval needs of different users, it is necessary to deploy multiple retrieval systems separately according to the customer's performance requirements. However, this deployment method has a high maintenance cost, and for complex retrieval requirements, it is necessary to search in multiple retrieval systems in sequence, resulting in poor retrieval efficiency of data retrieval. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a data retrieval method, device, electronic device and readable storage medium.

[0005] In a first aspect, the present invention provides a data retrieval method, the method comprising:

[0006] Constructing target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier;

[0007] Based on the target feature library identifier, determining a target vector library cluster and a target storage database corresponding to the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1;

[0008] By concurrently calling the services corresponding to the vector databases, each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services;

[0009] Based on the vector retrieval results corresponding to each of the vector databases, a target retrieval result corresponding to the feature query vector is obtained in the target storage database.

[0010] Optionally, the concurrently calling services corresponding to the vector databases so that each service obtains a vector search result corresponding to the feature query vector from each vector database based on the target search information includes:

[0011] Determining a target service corresponding to each of the vector databases;

[0012] For any vector database among the N vector databases, through the target service corresponding to the vector database, based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information, the vector database is searched to obtain a first vector search result corresponding to the vector database;

[0013] Based on the first vector retrieval results corresponding to each of the vector databases, a vector retrieval result corresponding to the feature query vector is determined.

[0014] Optionally, when the type of the vector database is the first type, searching the vector database based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information to obtain a first vector search result corresponding to the vector database includes:

[0015] Obtaining vector library information and search conditions in the target search information;

[0016] Based on the vector library information, the search condition and the feature query vector, obtaining in the vector database vector indexes corresponding to feature vector data matching the feature query vector and similarity scores corresponding to the vector indexes;

[0017] Based on the vector indexes and the similarity scores corresponding to the vector indexes, a first vector search result corresponding to the vector database is determined.

[0018] Optionally, when the type of the vector database is the second type, searching the vector database based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information to obtain a first vector search result corresponding to the vector database includes:

[0019] Loading the feature vector data stored in the vector database into the target memory;

[0020] Obtaining vector library information and search conditions in the target search information;

[0021] Based on the vector library information, the search condition and the feature query vector, a cosine similarity score between the feature query vector and each feature vector data in the target memory is determined, and based on the cosine similarity score, a first vector search result corresponding to the vector database is determined.

[0022] Optionally, acquiring a target retrieval result corresponding to the feature query vector in the target storage database based on the vector retrieval results corresponding to each of the vector databases includes:

[0023] Determining a target search object based on metadata information corresponding to the target storage database;

[0024] Determine a target query statement based on the target search object and the database type of the target storage database;

[0025] Based on the vector index in the vector search result, the target search result corresponding to the feature query vector is obtained in the target storage database through the target query statement.

[0026] Optionally, the method further comprises:

[0027] According to the target performance requirements, the database type of the vector database is determined from the preset vector database types, and the database type of the storage database is determined from the preset storage database types; the preset vector database types include a first type and a second type, and the preset storage database types include a MongoDB database type and a ClickHouse database type.

[0028] Optionally, when the target search result includes associated library information, the method further includes:

[0029] Based on the association library information, determining an associated feature library associated with the feature query vector;

[0030] Based on the feature query vector, a vector search result obtained by searching in a vector database corresponding to the associated feature library and a target search result in a storage database corresponding to the associated feature library are obtained.

[0031] In a second aspect, the present invention provides a data retrieval device, the device comprising:

[0032] A first construction module is used to construct target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier;

[0033] A first determination module is used to determine a target vector library cluster and a target storage database corresponding to the target feature library identifier based on the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1;

[0034] A first acquisition module is used to concurrently call the services corresponding to the vector databases so that each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services;

[0035] The second acquisition module is used to acquire the target retrieval result corresponding to the feature query vector in the target storage database based on the vector retrieval results corresponding to each of the vector databases.

[0036] Optionally, the first acquisition module includes:

[0037] A first determination submodule, used to determine the target service corresponding to each of the vector databases;

[0038] A first search module is used to search any vector database among the N vector databases through a target service corresponding to the vector database, based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information, to obtain a first vector search result corresponding to the vector database;

[0039] The second determining submodule is used to determine the vector retrieval result corresponding to the feature query vector based on the first vector retrieval results corresponding to each of the vector databases.

[0040] Optionally, when the type of the vector database is the first type, the first retrieval module includes:

[0041] A first acquisition submodule is used to acquire the vector library information and the search conditions in the target search information;

[0042] A second acquisition submodule is used to acquire, in the vector database, vector indexes corresponding to feature vector data matching the feature query vector and similarity scores corresponding to each of the vector indexes based on the vector library information, the search condition and the feature query vector;

[0043] The third determining submodule is used to determine the first vector search result corresponding to the vector database based on the vector indexes and the similarity scores corresponding to the vector indexes.

[0044] Optionally, when the type of the vector database is the second type, the first retrieval module includes:

[0045] A first loading module, used for loading the feature vector data stored in the vector database into a target memory;

[0046] A third acquisition submodule is used to acquire the vector library information and the search conditions in the target search information;

[0047] The fourth determination submodule is used to determine the cosine similarity score between the feature query vector and each feature vector data in the target memory based on the vector library information, the search condition and the feature query vector, and determine the first vector search result corresponding to the vector database based on the cosine similarity score.

[0048] Optionally, the second acquisition module includes:

[0049] A fifth determination submodule, configured to determine a target search object based on metadata information corresponding to the target storage database;

[0050] A sixth determination submodule, configured to determine a target query statement based on the target search object and the database type of the target storage database;

[0051] A fourth acquisition submodule is used to acquire, based on the vector index in the vector retrieval result, the target retrieval result corresponding to the feature query vector in the target storage database through the target query statement.

[0052] Optionally, the device further comprises:

[0053] The second determination module is used to determine the database type of the vector database from the preset vector database types according to the target performance requirements, and to determine the database type of the storage database from the preset storage database types; the preset vector database types include the first type and the second type, and the preset storage database types include the MongoDB database type and the ClickHouse database type.

[0054] Optionally, when the target search result includes associated library information, the device further includes:

[0055] A third determining module, configured to determine an associated feature library associated with the feature query vector based on the associated library information;

[0056] The third acquisition module is used to acquire, based on the feature query vector, a vector retrieval result obtained by querying in a vector database corresponding to the associated feature library and a target retrieval result in a storage database corresponding to the associated feature library.

[0057] In a third aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data retrieval method described in any one of the first aspects when executing the program.

[0058] In a fourth aspect, the present invention provides a readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the steps in the data retrieval method in any one of the embodiments of the first aspect above.

[0059] In the embodiment of the present invention, based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier, the target search information is constructed; based on the target feature library identifier, the target vector library cluster and the target storage database corresponding to the target feature library identifier are determined; the target vector library cluster contains N vector databases, N is greater than or equal to 1; by concurrently calling the services corresponding to each vector database, each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services; based on the vector search results corresponding to each of the vector databases, the target search results corresponding to the feature query vector are obtained in the target storage database. In this way, through the target feature library identifier, the target vector library cluster and the target storage database associated with the target feature library identifier can be determined, and then the vector search results and the target search results can be obtained in the target vector library cluster and the target storage database at one time, that is, through one data search, the vector search results and the target search results corresponding to the feature query vector can be obtained, realizing the integrated search of vector search and structured data search, and compared with repeated data search in different search systems, the search efficiency is improved to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 is a flow chart of steps of a data retrieval method provided by an embodiment of the present invention;

[0062] Figure 2 It is a schematic diagram of a database architecture of a retrieval system provided by an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of the architecture of a retrieval system provided by an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of the architecture of another retrieval system provided by an embodiment of the present invention;

[0065] Figure 5is a retrieval timing flow chart of a data retrieval method provided by an embodiment of the present invention;

[0066] Figure 6 is a flowchart of specific steps of a data retrieval method provided by an embodiment of the present invention;

[0067] Figure 7 is a structural diagram of a data retrieval device provided by an embodiment of the present invention;

[0068] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Figure 1 is a flow chart of steps of a data retrieval method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0071] Step 101: construct target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier.

[0072] In an embodiment of the present invention, a feature library for retrieval and matching is constructed based on the data provided by the user, and data can be supplemented regularly to improve the feature library. The information in the data is subjected to feature extraction, and classified into feature libraries corresponding to different feature types according to different feature types. In a possible embodiment, the tenant can extract the features of faces, vehicles, and objects from the snapped pictures of faces, vehicles, and street checkpoints to obtain feature data. The storage interface is called to store the feature data into the corresponding feature libraries such as face, vehicle, and object libraries.

[0073] Different vector databases and storage databases can be bound to different feature libraries. The number of vector databases can be multiple, such as Figure 2As shown in the database architecture of the retrieval system, the retrieval management service can include a feature library instance list, and the feature library instance list can contain multiple feature libraries, namely micro library 1, small library 2, large library 3 and medium library 4, and different vector databases and storage databases are bound to each feature library. Among them, the type of vector database can be Milvus, self-developed vector library, etc., and the type of storage database can be MongoDB, ClickHouse, etc. For example, two vector databases (Milvus-2 and Milvus-3) and one storage database (MongoDB / ClickHouse 3) are bound to large library 3. Correspondingly, when searching based on Milvus, it is necessary to use the retrieval Agent service to call Milvus for retrieval. One retrieval Agent service corresponds to one vector database, that is, one vector database can be called based on one retrieval Agent service, and multiple vector databases can be called through multiple retrieval Agent services. In the configuration file of the retrieval Agent service, set the necessary parameters such as the URI, port, user name and password connected to the vector database to ensure that the retrieval Agent service can successfully connect to the vector database (such as Milvus). Correspondingly, each retrieval agent service mainly manages Milvus and self-developed feature libraries as well as CPU / GPU indexes, and supports expanding the copy capabilities of Milvus or self-developed vector libraries to meet the retrieval capabilities of multiple copy vector libraries, thereby improving overall performance.

[0074] Furthermore, in the retrieval system, each vector database (such as a Milvus cluster instance, etc.) can support physical isolation by tenant (user), for example, independent vector databases are allocated to different tenants. At the same time, tenants also support logical isolation by Collection and Partition in the vector database. For example, in a vector database, different vector data storage spaces are allocated to different tenants according to different Collections and Partitions. Similarly, the storage database can simultaneously support isolation by tenant, and also support isolation of tenants by library and date. It can be understood that the isolation method for the vector database and the storage database can be configured according to actual needs, and the embodiment of the present invention does not limit this.

[0075] In a possible embodiment, the process of constructing a vector database bound to a feature library may include: for any feature library, using an appropriate encoding method to convert feature data of the feature library into feature vector data, storing the feature vector data in the vector database, and setting a corresponding vector index (e.g., vector ID, etc.) for the feature vector data in the vector database.

[0076] In a possible embodiment, the process of constructing a storage database bound to a feature library may include: extracting feature data to be stored from the feature library, including feature data and data-related information (e.g., structured attribute data, which may include feature names, category labels, attribute information, etc.). The vector index of the feature vector data corresponding to the feature data and the data-related information corresponding to the feature data are stored in the storage database. Specifically, the fields of a MongoDB document may be defined according to the structure of the vector index. Typically, the vector index may be stored as an array type, and the data-related information may be stored as a string, an integer, or the like.

[0077] For any feature library, the metadata information corresponding to the feature library may include the vector database bound to the feature library, the storage database, the services corresponding to each vector database, the vector database information, the storage database information, etc. Based on the feature library identifier corresponding to the feature library, the metadata information corresponding to the feature library can be determined.

[0078] For example, the architecture of the retrieval system can be as follows: Figure 3 As shown, the retrieval system may include PaaS components, Agent services, management services, and application layers, as follows:

[0079] 1) PaaS components can include vector databases and storage databases, such as MongoDB, ClickHouse, Milvus, Redis, Kafka, etc.;

[0080] 2) One Agent service instance corresponds to one Milvus instance, which is mainly used to manage Milvus, self-developed feature library, and CPU / GPU index;

[0081] 3) Management services may include metadata management (storage multi-instance management), feature library management, multi-instance vector library management, and SDK management;

[0082] 4) Feature extraction algorithm service: used to extract data features (e.g., feature extraction using a general large model);

[0083] 5) The application layer includes an open platform for providing all-in-one recognition capabilities for different types of retrieval targets.

[0084] Furthermore, according to the design architecture of the retrieval system, the implementation case structure can be as follows: Figure 4As shown, this retrieval system relies on PaaS components including MongoDB, ClickHouse, Milvus, Redis, and Kafka, which need to be deployed on demand. MongoDB, ClickHouse, and Milvus can support connecting multiple instances. This retrieval system relies on the registration center Nacos to ensure that the management service discovers multiple Agent service instances. The management service, algorithm service, Agent service, and Milvus in this retrieval system provide Docker images and helm resource files, which can be directly uploaded to the image warehouse and then started with one click through helminstall.

[0085] In an embodiment of the present invention, when receiving the data to be retrieved, the SDK module can extract the feature query vector corresponding to the data to be retrieved through a feature extraction algorithm. The data to be retrieved may be a picture, text, etc. The feature library ID input by the user is obtained as the target feature library identifier. Based on the target feature library identifier, a unique feature library, a vector database bound to the feature library, and a storage database can be determined. Based on the feature query vector and the target feature library identifier, the target retrieval information is constructed. The target retrieval information may be the necessary information required for the retrieval, and the target retrieval information may include tenant information, target feature library identifier, feature query vector, and retrieval conditions, etc. The retrieval conditions may include retrieval equipment, retrieval time range, and the number of similar vectors returned (top-k), etc. It is understandable that the necessary verification of the input parameter information, the determination of whether the target feature library identifier exists, and the pre-setting of the top-k threshold (for example, setting the dynamic library topK not to exceed 1000) and other operations can be performed in advance, and the embodiment of the present invention does not limit this.

[0086] Step 102: Based on the target feature library identifier, determine a target vector library cluster and a target storage database corresponding to the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1.

[0087] In an embodiment of the present invention, based on the target feature library identifier, a target vector library cluster and a target storage database corresponding to the target feature library identifier are determined. The target vector library cluster includes N vector databases, where N is greater than or equal to 1. The target vector library cluster corresponding to the target feature library identifier includes N vector databases bound to the target feature library corresponding to the target feature library identifier, and the target storage database is a storage database bound to the target feature library corresponding to the target feature library identifier. In other words, based on the target feature library identifier, the vector database and the storage database bound to the target feature library corresponding to the target feature library identifier can be determined.

[0088] Step 103: by concurrently calling the services corresponding to the vector databases, each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services.

[0089] In an embodiment of the present invention, one retrieval agent service corresponds to one vector database, that is, when there are multiple vector databases, it is necessary to call the corresponding vector databases for retrieval based on multiple retrieval agent services. Based on each vector database, the service corresponding to each vector database is determined. The services corresponding to each vector database are called concurrently, and each service is used to retrieve information based on the target, and the vector retrieval results are obtained in the corresponding vector database. Among them, the vector retrieval results may include one or more groups of vector indexes corresponding to the vectors matching the feature query vector and the similarity scores corresponding to the vectors.

[0090] It can be understood that after obtaining the vector retrieval results corresponding to each vector database, the vector retrieval results corresponding to each vector database can be directly used as the retrieval results for the vector database, or, for the sake of data streamlining, based on the similarity score sorting, the vector retrieval results with a similarity score greater than a preset score threshold can be determined from the vector retrieval results corresponding to each vector database, or a preset number of vector retrieval results with similarity scores from high to low can be determined as the retrieval results for the vector database.

[0091] Optionally, step 103 may include the following steps:

[0092] Step 201: Determine the target service corresponding to each of the vector databases.

[0093] In the embodiment of the present invention, for any vector database among the N vector databases, a target service corresponding to the vector database is determined.

[0094] Step 202: For any vector database among the N vector databases, through the target service corresponding to the vector database, based on the vector database information corresponding to the target feature library identifier and the search conditions in the target search information, search the vector database to obtain a first vector search result corresponding to the vector database.

[0095] In an embodiment of the present invention, the target service corresponding to each vector database is concurrently called to perform vector data retrieval. For any vector database, the target service corresponding to the vector database is used to obtain the vector library information and the retrieval condition in the target retrieval information based on the target feature library identifier. The vector library information is pre-stored in the metadata information corresponding to the feature library based on the actual configuration when the feature library and the vector database are pre-built. Among them, the vector library information may include the Collection (table) and Partition (partition) information of the vector database bound to the feature library. Through the target service corresponding to the vector database, based on the vector library information and the retrieval condition, the vector database is retrieved to obtain the first vector retrieval result corresponding to the vector database. Exemplarily, according to the setting of the retrieval condition, the vector database can be targeted for data retrieval. For example, if it is necessary to retrieve data related to a certain camera device, the device identifier corresponding to the camera device can be used as the retrieval condition, and then the retrieval result for the camera device is obtained when the data is retrieved; or, if it is necessary to retrieve the relevant data collected from 7:00 to 20:00, the time range can be used as the retrieval condition, and then the retrieval result for the time range is obtained when the data is retrieved. It is understandable that the embodiment of the present invention does not limit the setting of the search conditions, and does not limit the specific method of searching based on the search conditions.

[0096] Step 203: Determine a vector search result corresponding to the feature query vector based on the first vector search results corresponding to each of the vector databases.

[0097] In the embodiment of the present invention, based on the first vector search results corresponding to each vector database, the vector search results corresponding to the feature query vector are determined. Specifically, each first vector search result can be directly determined as the vector search result, or according to the order of the similarity scores in each first vector search result, the vector search results with a similarity score greater than a preset score threshold, or a preset number of vector search results with similarity scores from high to low are used as the vector search result.

[0098] In an embodiment of the present invention, a target service corresponding to a vector database is utilized to search the vector database based on the vector library information corresponding to the target feature library identifier and the search conditions in the target search information to obtain a first vector search result corresponding to the vector database, so that one target service only searches one corresponding vector database, thereby improving the search speed to a certain extent.

[0099] For example, the target vector library cluster includes two vector databases, namely vector database 1 and vector database 2. Then, service 1 corresponding to vector database 1 can be used to search vector database 1 based on the target search information, and service 2 corresponding to vector database 2 can be used to search vector database 2 based on the target search information. Assuming that the number of similar vectors returned (top-k) included in the target search information is 100, service 1 and service 2 will retrieve 100 search results from vector database 1 and vector database 2 respectively. Furthermore, the top 100 search results with the similarity scores from high to low can be selected from the 200 search results as the final vector search results.

[0100] Step 104: Based on the vector retrieval results corresponding to each of the vector databases, obtain the target retrieval result corresponding to the feature query vector in the target storage database.

[0101] In the embodiment of the present invention, based on the vector search results corresponding to each vector database, the vector index in the vector search result is obtained. Since each pre-built storage database stores vector index and data related information, the structured data corresponding to the vector index stored in the target storage database can be determined based on the vector index as the target search result.

[0102] The target search results and the vector search results are used as the search results for the data to be searched. The SDK module can assemble and return the vectors and structured data returned by the search management service to the application layer.

[0103] For example, Figure 5 A retrieval timing flow chart of a data retrieval method is shown, such as Figure 5 As shown, the application layer sends the data to be retrieved and the feature library ID input by the user to the SDK encapsulation module. The SDK encapsulation module extracts the feature query vector of the data to be retrieved through the algorithm service, and assembles the target retrieval information based on the feature query vector and the feature library ID. The target retrieval information is sent to the retrieval management service. The retrieval management service determines the N vector databases and the target storage database bound to the target feature library according to the feature library ID. Through the services corresponding to each vector database, it searches in each vector database based on the feature query vector, and obtains the retrieval results of each vector database as the vector retrieval results. Based on the vector retrieval results, a structured search is performed in the target storage database to obtain the target retrieval results. The retrieval management service aggregates the vector retrieval results and the target retrieval results and returns them to the SDK encapsulation module, which is then returned to the application layer by the SDK encapsulation module.

[0104] In summary, in the embodiment of the present invention, based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier, the target search information is constructed; based on the target feature library identifier, the target vector library cluster and the target storage database corresponding to the target feature library identifier are determined; the target vector library cluster contains N vector databases, N is greater than or equal to 1; by concurrently calling the services corresponding to each vector database, each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services; based on the vector search results corresponding to each of the vector databases, the target search results corresponding to the feature query vector are obtained in the target storage database. In this way, through the target feature library identifier, the target vector library cluster and the target storage database associated with the target feature library identifier can be determined, and then the vector search results and the target search results can be obtained in the target vector library cluster and the target storage database at one time, that is, through one data search, the vector search results and the target search results corresponding to the feature query vector can be obtained, realizing the integrated search of vector search and structured data search, and compared with repeated data search in different search systems, the search efficiency is improved to a certain extent. Furthermore, there is no need to adapt the storage components of each retrieval system to meet different performance requirements. The embodiment of the present invention reduces the deployment cost and subsequent maintenance cost through an integrated retrieval system.

[0105] Furthermore, in an embodiment of the present invention, different vector databases correspond to different services. Thus, for scenarios requiring retrieval of vector libraries of billions, the number of multi-instance vector databases can be horizontally expanded through the underlying system (i.e., by configuring multiple vector databases and concurrently calling multiple services to perform data retrieval of vector databases) to quickly support performance retrieval of ultra-large capacity libraries, thereby improving the comprehensiveness and flexibility of data retrieval application scenarios.

[0106] Optionally, when the type of the vector database is the first type, the first type may be Milvus, that is, the vector database is a Milvus database. The Milvus database can store vector data by converting unstructured data (such as text, images, etc.) into vector form, and using efficient vector indexing and retrieval algorithms to achieve efficient retrieval of massive data.

[0107] Step 201 may include the following steps:

[0108] Step 301: Obtain vector library information and search conditions in the target search information.

[0109] In an embodiment of the present invention, vector library information and search conditions in the target search information are obtained. The vector library information is pre-stored in metadata information corresponding to the feature library based on actual configuration conditions when the feature library and the vector database are pre-built. The vector library information may include Collection (table) and Partition (partition) information of the vector database bound to the feature library.

[0110] Step 302: Based on the vector library information, the search condition and the feature query vector, obtain in the vector database vector indexes corresponding to the feature vector data matching the feature query vector and similarity scores corresponding to the vector indexes.

[0111] In an embodiment of the present invention, based on the vector library information, the retrieval condition and the feature query vector, the feature vector data matching the feature query vector is determined in the vector database, and the vector index corresponding to the feature vector data matching the feature query vector and the similarity score between the feature query vector and the matching feature vector data are obtained. Accordingly, the similarity score corresponding to the vector index is used to characterize the similarity score between the feature vector data corresponding to the vector index and the feature query vector. Among them, the feature vector data matching the feature query vector can be the feature vector data whose matching degree with the feature query vector satisfies the retrieval condition. Assuming that the retrieval condition includes that the retrieval device is camera device 1, the number of similar vectors returned (top-k) is 100 (the default selection is the first 100 data sorted from high to low in similarity score), then the first 100 feature vector data in the feature vector data corresponding to camera device 1 and the feature query vector are sorted from high to low in similarity score are selected as the query result. The vector index and similarity score corresponding to these 100 feature vector data are obtained.

[0112] Exemplarily, the feature query vector may be a vector field, and the search condition may be a scalar field, that is, the Agent service may call milvus (vector database) for search based on the Collection and Partition information, the scalar field, and the vector field, and return the vector ID and the similarity score.

[0113] Step 303: Determine a first vector search result corresponding to the vector database based on the vector indexes and the similarity scores corresponding to the vector indexes.

[0114] In the embodiment of the present invention, the vector index and the similarity score corresponding to each vector index may be directly determined as the first vector detection result corresponding to the vector database, or the first vector search result may be determined after other processing operations (such as sorting, filtering, displaying to the user, etc.) are performed after obtaining the vector index and the similarity score corresponding to each vector index. It is understandable that the embodiment of the present invention does not limit this.

[0115] In an embodiment of the present invention, by matching feature vector data with a feature query vector to obtain a similarity score, this retrieval method based on vector similarity can quickly find data items that are most similar to the feature vector data, thereby improving the efficiency and accuracy of data retrieval.

[0116] Optionally, in the case where the type of the vector database is the second type, the second type is a self-developed vector database, such as a vector database constructed by a user.

[0117] Step 103 may include the following steps:

[0118] Step 401: Load the feature vector data stored in the vector database into a target memory.

[0119] In the embodiment of the present invention, the feature vector data stored in the vector database is loaded into the target memory, wherein the target memory may be a memory space allocated for the retrieval system.

[0120] Step 402: Obtain vector library information and search conditions in the target search information.

[0121] In an embodiment of the present invention, vector library information and search conditions in the target search information are obtained. The vector library information is pre-stored in metadata information corresponding to the feature library based on actual configuration conditions when the feature library and the vector database are pre-built. The vector library information may include Collection (table) and Partition (partition) information of the vector database bound to the feature library.

[0122] Step 403: Based on the vector library information, the search condition and the feature query vector, determine the cosine similarity score between the feature query vector and each feature vector data in the target memory, and based on the cosine similarity score, determine the first vector search result corresponding to the vector database.

[0123] In an embodiment of the present invention, based on the vector library information, the retrieval conditions and the feature query vector, the cosine similarity scores between the feature query vector and each feature vector data in the target memory are determined, and based on the cosine similarity scores, the first vector retrieval result corresponding to the vector database is determined. Exemplarily, the similarity between the feature query vector and each feature vector data can be measured by calculating the cosine value of the angle between the two vectors. According to the retrieval conditions, the vector index corresponding to the feature vector data whose matching degree with the feature query vector meets the retrieval conditions and the cosine similarity score corresponding to the vector index are determined. Based on the vector index corresponding to the feature vector data whose matching degree with the feature query vector meets the retrieval conditions and the cosine similarity score corresponding to the vector index, the first vector retrieval result is determined.

[0124] In an embodiment of the present invention, for the second type of vector database, the feature vector data can be first loaded into the target memory, and then the first vector retrieval result is determined by calculating the cosine similarity score between the feature query vector and the feature vector data. Different similarity calculation methods are used for different types of vector databases. Such a flexible similarity calculation method enables the vector database to be applicable to a wider range of data retrieval needs.

[0125] Optionally, step 104 may include the following steps:

[0126] Step 501: Determine a target search object based on metadata information corresponding to the target storage database.

[0127] In an embodiment of the present invention, after determining the target storage database based on the target feature library identifier, the target retrieval object can be determined based on the metadata information corresponding to the target storage database. The metadata information corresponding to the target storage database can be the table name and table structure corresponding to the target storage database. In order to facilitate upper-layer development, a unified query can be performed based on the target retrieval object for different types of storage databases. Therefore, when determining the target retrieval object, a unified object, namely the target retrieval object, can be dynamically assembled based on the metadata information. In this way, regardless of the type of the target storage database, a unified target retrieval object can be used for querying.

[0128] Step 502: Determine a target query statement based on the target search object and the database type of the target storage database.

[0129] Step 503: Based on the vector index in the vector search result, obtain the target search result corresponding to the feature query vector in the target storage database through the target query statement.

[0130] In an embodiment of the present invention, after performing vector retrieval on N vector databases to obtain vector retrieval results, the vector retrieval results will include multiple groups of vector indexes matching the feature query vectors and corresponding similarity scores. According to the target retrieval object and the database type of the target storage database, a target query statement for the target storage database is determined. Among them, different database types can correspond to different target query statements, and accordingly, the combination of different target retrieval objects and database types of different target storage databases can correspond to different target query statements. The target query statement can be a query SQL. Based on the vector index in the vector retrieval result, the corresponding instance information is queried in the target storage database through the target query statement to obtain structured data as the target retrieval result. Exemplarily, the vector indexes in the vector retrieval result can be merged and assembled, and the structured data is queried in the target storage database based on the merged and assembled vector index as the target retrieval result. Among them, the target retrieval result can include structured data corresponding to the vector index in each vector retrieval result, and the structured data can include data-related information (such as feature name, category label, attribute information, etc.) of the feature data corresponding to the vector index.

[0131] In an embodiment of the present invention, a target retrieval result corresponding to a feature query vector is obtained in a storage database by using a vector index in a vector retrieval result obtained by retrieving the vector database. Since both the vector database and the storage database store vector indexes, the vector indexes can be used as association items between the vector database and the storage database, and then the vector data and structured data matching the same feature query vector in the vector database and the storage database can be queried accordingly, thereby improving data retrieval efficiency.

[0132] Optionally, the embodiment of the present invention may further include the following steps:

[0133] Step 601: According to the target performance requirements, determine the database type of the vector database from the preset vector database types, and determine the database type of the storage database from the preset storage database types; the preset vector database types include a first type and a second type, and the preset storage database types include a MongoDB database type and a ClickHouse database type.

[0134] In an embodiment of the present invention, different database combinations can be selected according to the actual usage requirements of the user to adapt to different usage requirements. Specifically, the storage method of which storage database + vector database to select can be determined according to the library capacity, performance indicators, and storage space requirements. According to the target performance requirements, the database type corresponding to the vector database is determined from the preset vector database types, and the database type corresponding to the storage database is determined from the preset storage database types. Among them, the preset vector database types include a first type, namely Milvus, and a second type, namely a self-developed vector database, and the preset storage database types include MongoDB database types and ClickHouse database types.

[0135] For example, different storage methods are used to meet different performance indicators. For example, the storage method of MongoDB+self-developed vector library can support high-performance reading and writing of thousands of data, the storage method of MongoDB+Milvus can support high-performance reading and writing of tens of millions of data, and the storage method of ClickHouse+Milvus can support high-performance reading and writing of hundreds of millions of data. Specifically, the correspondence between different library specifications and performance indicators can be as follows:

[0136]

[0137]

[0138] In the embodiment of the present invention, different vector database types and storage database types are selected according to the target performance requirements, which can support tenants to dynamically select a database that matches the target performance requirements for database construction, saving a lot of storage costs and retrieval performance. The compression ratio effect and retrieval performance of different storage databases are inconsistent, thereby realizing the application requirements of different tenants for different data retrieval scenarios.

[0139] Optionally, when the target search result includes associated library information, the embodiment of the present invention may further include the following steps:

[0140] Step 701: Determine an associated feature library associated with the feature query vector based on the associated library information.

[0141] In an embodiment of the present invention, when constructing a storage database bound to a feature library, if the data provided by the user contains at least two different types of feature data, the feature data will be correspondingly summarized into at least two different types of feature libraries. Since at least two different types of feature data come from the same data, there is a certain association relationship between at least two different types of feature data, that is, any feature data can be associated with other types of feature libraries. If there are feature data associated with other types of feature libraries in the feature library, the feature library identifier of other feature libraries associated with the feature data will be added to the data related information stored in the storage database. Therefore, after obtaining the target retrieval result stored in the target storage database based on the vector retrieval result, it is determined whether the target retrieval result contains associated library information. If it contains associated library information, the associated feature library associated with the feature query vector is determined based on the feature library identifier in the associated library information. Among them, the target feature library corresponding to the target feature library identifier is different from the library type of the associated feature library. The inclusion of associated library information in the target retrieval result can represent that the feature query vector contains at least two different types of features.

[0142] Step 702: Based on the feature query vector, obtain the vector search result obtained by searching in the vector database corresponding to the associated feature library and the target search result in the storage database corresponding to the associated feature library.

[0143] In an embodiment of the present invention, after determining the associated feature library, the associated feature library identifier corresponding to the associated feature library is obtained, and the target retrieval information is constructed based on the feature query vector and the associated feature library identifier. Based on the target retrieval information, the vector database and the storage database corresponding to the associated feature library identifier are determined. The vector retrieval results obtained by querying in the vector database corresponding to the associated feature library, and the target retrieval results in the storage database corresponding to the associated feature library. It can be understood that the data retrieval method of step 702 is similar to the data retrieval method of steps 101-104, and can be specifically referred to steps 101-104, 201-203, 301-303, 401-403, 501-502, and the embodiment of the present invention will not be repeated here.

[0144] Exemplarily, when a face and a vehicle exist in a picture, feature information in the picture, namely, face features and vehicle features, is extracted, and the face features are stored in face feature library A, and the vehicle features are stored in vehicle feature library A. A vector database and a storage database are constructed for face feature library A, and when the storage database is constructed, a feature library identifier corresponding to vehicle feature library A is added to the data-related information corresponding to the vector index corresponding to the face feature. Correspondingly, in the storage database corresponding to vehicle feature library A, a feature library identifier corresponding to face feature library A is added to the data-related information corresponding to the vector index corresponding to the vehicle feature. In this way, after the user inputs the feature library identifier corresponding to the face feature library A, the vector database and storage database corresponding to the face feature library A are determined based on the feature library identifier, and the vector retrieval result is retrieved from the vector database, and the target retrieval result is retrieved from the storage database. At this time, the target retrieval result will include the feature library identifier of the vehicle feature library A. Therefore, the vector database and storage database corresponding to the vehicle feature library A can be further determined based on the feature library identifier of the vehicle feature library A, and the vector retrieval result is retrieved from the vector database, and the target retrieval result is retrieved from the storage database, thereby realizing the fusion data retrieval capability of searching for cars by people.

[0145] In an embodiment of the present invention, by judging whether the target search result contains associated library information, other types of associated feature libraries associated with the feature query vector can be quickly determined when the target search result contains associated library information, and then based on the feature query vector, a vector database is searched and a storage database is queried for the associated feature library, thereby realizing multimodal fusion retrieval, that is, by inputting a target feature library identifier, other associated associated feature libraries can be identified and retrieved, and the vector database corresponding to the feature library and the retrieval results in the storage database can be obtained together, thereby improving the fusion scene retrieval capability, such as common scenarios such as searching for people by car, searching for cars by people, and searching for things by all things recognition in specific scenes.

[0146] The embodiments of the present invention can be used for AI applications in multiple scenarios such as urban governance, campus governance, rental housing safety, and kitchen safety. Through the integrated retrieval system of this multi-column storage component (storage database) and Milvus vector library (vector database), the system can be flexibly configured, multiple combinations of retrieval and query methods can be used, the underlying multi-instance and multi-table storage components can be quickly switched, the application access cost can be reduced, and the scalability of multiple storage components can be improved, so that the integrated retrieval performance can be horizontally expanded.

[0147] For example, Figure 6 A flow chart showing the specific steps of a data retrieval method is shown in FIG. Figure 6As shown, the application extracts features from the data to be retrieved to obtain a feature query vector, and the retrieval management service obtains the Agent instance information corresponding to the vector database bound to the target feature library corresponding to the target feature library identifier. Traverse all Agent instance information (instance 0-instance N), and concurrently call each Agent instance information based on the feature query vector. After accumulating multiple data, merge M:N vector retrieval is performed in the Milvus multi-instance vector database to obtain the retrieval results (top-K) obtained by the retrieval of each Agent instance information, aggregate and sort the retrieval results obtained by the retrieval of each Agent instance information, and determine the vector retrieval results. Based on the vector index in the vector retrieval result, the associated structured data is retrieved in MongoDB or ClickHouse to obtain the target retrieval result. The target retrieval result and the vector retrieval result are returned to the application as a composite retrieval result.

[0148] Figure 7 is a schematic diagram of the structure of a data retrieval device provided by an embodiment of the present invention, such as Figure 7 As shown, the device may specifically include:

[0149] The first construction module 801 is used to construct target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier;

[0150] A first determination module 802 is used to determine a target vector library cluster and a target storage database corresponding to the target feature library identifier based on the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1;

[0151] The first acquisition module 803 is used to concurrently call the services corresponding to the vector databases so that each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services;

[0152] The second acquisition module 804 is used to acquire the target retrieval result corresponding to the feature query vector in the target storage database based on the vector retrieval results corresponding to each of the vector databases.

[0153] The embodiment of the present invention provides a data retrieval device, which constructs target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier; determines the target vector library cluster and the target storage database corresponding to the target feature library identifier based on the target feature library identifier; the target vector library cluster contains N vector databases, N is greater than or equal to 1; by concurrently calling the services corresponding to each vector database, each service obtains the vector retrieval results corresponding to the feature query vector from each vector database based on the target retrieval information; different vector databases correspond to different services; based on the vector retrieval results corresponding to each of the vector databases, the target retrieval results corresponding to the feature query vector are obtained in the target storage database. In this way, through the target feature library identifier, the target vector library cluster and the target storage database associated with the target feature library identifier can be determined, and then the vector retrieval results and the target retrieval results can be obtained in the target vector library cluster and the target storage database at one time, that is, through one data retrieval, the vector retrieval results and the target retrieval results corresponding to the feature query vector can be obtained, realizing the integrated retrieval of vector retrieval and structured data retrieval, and compared with repeatedly performing data retrieval multiple times in different retrieval systems, the retrieval efficiency is improved to a certain extent.

[0154] Optionally, the first acquisition module 803 includes:

[0155] A first determination submodule, used to determine the target service corresponding to each of the vector databases;

[0156] A first search module is used to search any vector database among the N vector databases through a target service corresponding to the vector database, based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information, to obtain a first vector search result corresponding to the vector database;

[0157] The second determining submodule is used to determine the vector retrieval result corresponding to the feature query vector based on the first vector retrieval results corresponding to each of the vector databases.

[0158] Optionally, when the type of the vector database is the first type, the first retrieval module includes:

[0159] A first acquisition submodule is used to acquire the vector library information and the search conditions in the target search information;

[0160] A second acquisition submodule is used to acquire, in the vector database, vector indexes corresponding to feature vector data matching the feature query vector and similarity scores corresponding to each of the vector indexes based on the vector library information, the search condition and the feature query vector;

[0161] The third determining submodule is used to determine the first vector search result corresponding to the vector database based on the vector indexes and the similarity scores corresponding to the vector indexes.

[0162] Optionally, when the type of the vector database is the second type, the first retrieval module includes:

[0163] A first loading module, used for loading the feature vector data stored in the vector database into a target memory;

[0164] A third acquisition submodule is used to acquire the vector library information and the search conditions in the target search information;

[0165] The fourth determination submodule is used to determine the cosine similarity score between the feature query vector and each feature vector data in the target memory based on the vector library information, the search condition and the feature query vector, and determine the first vector search result corresponding to the vector database based on the cosine similarity score.

[0166] Optionally, the second obtaining module 804 includes:

[0167] A fifth determination submodule, configured to determine a target search object based on metadata information corresponding to the target storage database;

[0168] A sixth determination submodule, configured to determine a target query statement based on the target search object and the database type of the target storage database;

[0169] A fourth acquisition submodule is used to acquire, based on the vector index in the vector retrieval result, the target retrieval result corresponding to the feature query vector in the target storage database through the target query statement.

[0170] Optionally, the device further comprises:

[0171] The second determination module is used to determine the database type of the vector database from the preset vector database types according to the target performance requirements, and to determine the database type of the storage database from the preset storage database types; the preset vector database types include the first type and the second type, and the preset storage database types include the MongoDB database type and the ClickHouse database type.

[0172] Optionally, when the target search result includes associated library information, the device further includes:

[0173] A third determining module, configured to determine an associated feature library associated with the feature query vector based on the associated library information;

[0174] The third acquisition module is used to acquire, based on the feature query vector, a vector retrieval result obtained by querying in a vector database corresponding to the associated feature library and a target retrieval result in a storage database corresponding to the associated feature library.

[0175] The present invention also provides an electronic device, see Figure 8 , including: a processor 901, a memory 902, and a computer program 9021 stored in the memory and executable on the processor, wherein the processor implements the data retrieval method of the aforementioned embodiment when executing the program.

[0176] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the data retrieval method of the above embodiment.

[0177] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0178] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0179] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0180] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0181] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0182] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0183] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0187] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data retrieval method, characterized in that: The method comprises: Constructing target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier; Based on the target feature library identifier, determining a target vector library cluster and a target storage database corresponding to the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1; By concurrently calling the services corresponding to the vector databases, each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services; Based on the vector retrieval results corresponding to each of the vector databases, a target retrieval result corresponding to the feature query vector is obtained in the target storage database.

2. The method according to claim 1, characterized in that The concurrently calling the services corresponding to the vector databases so that each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information includes: Determining a target service corresponding to each of the vector databases; For any vector database among the N vector databases, through the target service corresponding to the vector database, based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information, the vector database is searched to obtain a first vector search result corresponding to the vector database; Based on the first vector retrieval results corresponding to each of the vector databases, a vector retrieval result corresponding to the feature query vector is determined.

3. The method according to claim 2, characterized in that In the case where the type of the vector database is the first type, searching the vector database based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information to obtain a first vector search result corresponding to the vector database includes: Obtaining vector library information and search conditions in the target search information; Based on the vector library information, the search condition and the feature query vector, obtaining in the vector database vector indexes corresponding to feature vector data matching the feature query vector and similarity scores corresponding to the vector indexes; Based on the vector indexes and the similarity scores corresponding to the vector indexes, a first vector search result corresponding to the vector database is determined.

4. The method according to claim 2, characterized in that: In the case where the type of the vector database is the second type, searching the vector database based on the vector database information corresponding to the target feature library identifier and the search condition in the target search information to obtain a first vector search result corresponding to the vector database includes: Loading the feature vector data stored in the vector database into the target memory; Obtaining vector library information and search conditions in the target search information; Based on the vector library information, the search condition and the feature query vector, a cosine similarity score between the feature query vector and each feature vector data in the target memory is determined, and based on the cosine similarity score, a first vector search result corresponding to the vector database is determined.

5. The method according to claim 1, characterized in that The acquiring, in the target storage database, a target retrieval result corresponding to the feature query vector based on the vector retrieval results corresponding to each of the vector databases comprises: Determining a target search object based on metadata information corresponding to the target storage database; Determine a target query statement based on the target search object and the database type of the target storage database; Based on the vector index in the vector search result, the target search result corresponding to the feature query vector is obtained in the target storage database through the target query statement.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: According to the target performance requirements, the database type of the vector database is determined from the preset vector database types, and the database type of the storage database is determined from the preset storage database types; the preset vector database types include a first type and a second type, and the preset storage database types include a MongoDB database type and a ClickHouse database type.

7. The method according to claim 1, characterized in that In the case where the target search result includes associated library information, the method further includes: Based on the association library information, determining an associated feature library associated with the feature query vector; Based on the feature query vector, a vector search result obtained by searching in a vector database corresponding to the associated feature library and a target search result in a storage database corresponding to the associated feature library are obtained.

8. A data retrieval device, characterized in that: The device comprises: The first construction module is used to construct target retrieval information based on the feature query vector corresponding to the data to be retrieved and the target feature library identifier; A first determination module is used to determine a target vector library cluster and a target storage database corresponding to the target feature library identifier based on the target feature library identifier; the target vector library cluster includes N vector databases, where N is greater than or equal to 1; A first acquisition module is used to concurrently call the services corresponding to the vector databases so that each service obtains the vector search results corresponding to the feature query vector from each vector database based on the target search information; different vector databases correspond to different services; The second acquisition module is used to acquire the target retrieval result corresponding to the feature query vector in the target storage database based on the vector retrieval results corresponding to each of the vector databases.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data retrieval method according to any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data retrieval method described in any one of claims 1 to 7.

Images