Data retrieval method and device

Abstract

The embodiment of the invention provides a data retrieval method and device. The method comprises the following steps: receiving an initiated retrieval task, wherein the retrieval task comprises dimension constraint condition information and a retrieval vector of information to be retrieved; in response to the retrieval task, the retrieval task is executed asynchronously, and the step of asynchronously executing the retrieval task comprises the substeps that specified dimension constraint condition information in the retrieval task is transmitted to a column database, and metadata in the range of the dimension constraint condition information and a corresponding primary key list are screened in the column database; and transmitting the primary key list and a specified retrieval vector in the retrieval task to a vector database, and performing vector similarity retrieval in the vector database to obtain a retrieval result.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data retrieval method and apparatus. Background Technology

[0002] Vector retrieval refers to a data retrieval method that calculates the similarity between vectors of data.

[0003] In related technologies, vector retrieval can generally be divided into two categories. One category uses a single vector database to store all data (including data vectors and metadata); the other category stores vectors and metadata separately, with vectors stored in a separate vector database and metadata stored in a separate relational database, and performs related retrievals using primary keys.

[0004] In the first type of solution, since the single vector database is mainly for vector storage, when facing vector retrieval, the single vector database not only needs to store vectors but also metadata, resulting in low storage space utilization (vector databases lack optimization for metadata storage), inefficient complex queries (vector databases have poor query performance when performing complex queries based on metadata), and limited retrieval scale (vector databases have a limit on the number of queries per time, and the retrieval response time increases exponentially when the number of queries exceeds the limit).

[0005] In the second approach, after a user initiates a query, similar vectors are first retrieved from the vector database to obtain a list of primary keys. Then, the corresponding metadata is queried from the relational database using the primary keys. Finally, the results are integrated and returned. Each pagination requires an independent vector retrieval operation, significantly increasing response time, especially in high-concurrency scenarios, making it unsuitable for real-time recommendations and instant retrieval. Furthermore, when the data volume exceeds the processing capacity of both relational and vector databases, the retrieval response time increases exponentially and cannot overcome the limitation on the number of results retrieved in a single query, thus failing to achieve efficient retrieval of the entire dataset. Summary of the Invention

[0006] In view of the above, one or more embodiments of this specification provide a data retrieval method and apparatus, electronic device, and storage medium to solve the problems existing in the related art.

[0007] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0008] According to a first aspect of the embodiments of this specification, a data retrieval method is provided, applied to a storage system including a columnar database and a vector database; wherein the vector database stores vectors of original data and primary keys of the original data, the columnar database stores metadata of the original data and primary keys of the original data, and the vectors and metadata of the same original data are associated between the vector database and the columnar database through a unique primary key; the method includes:

[0009] Receive an initiated retrieval task, the retrieval task including dimensional constraint information and a retrieval vector of the information to be retrieved;

[0010] In response to the search task, the search task is executed asynchronously, the asynchronous execution of the search task includes:

[0011] The specified dimension constraint information in the retrieval task is transmitted to the columnar database, and the metadata and corresponding primary key list within the scope of the dimension constraint information are filtered in the columnar database.

[0012] The primary key list and the specified search vector in the search task are transmitted to the vector database, and vector similarity search is performed in the vector database to obtain search results; wherein, the search results include target vectors that are similar to the search vectors corresponding to each primary key in the primary key list, as well as the similarity scores between the target vectors and the search vectors.

[0013] Optionally, the step of transmitting the primary key list and the specified search vector in the search task to the vector database, and performing vector similarity search in the vector database, includes:

[0014] According to a preset batch, the primary keys in the primary key list and the specified search vectors in the search task are transmitted to the vector database in batches, and vector similarity search is performed in the vector database in batches.

[0015] Optionally, the batch size of the preset batch is calculated based on the number of primary keys in the primary key list and the maximum number of searches per time in the vector database.

[0016] Optional, also includes:

[0017] In response to the initiated query request for the retrieval task, the retrieval results obtained by performing vector similarity retrieval on the vector database are obtained;

[0018] Based on the target primary key corresponding to each target vector in the search results, query the target metadata corresponding to the target primary key from the columnar database;

[0019] The search results are sorted according to the similarity scores between each target vector and the search vector, and the final search results, including the target vector, target metadata, and similarity scores, are displayed.

[0020] Optional, also includes:

[0021] In response to a secondary search task initiated for a local search result selected from the final search results, the vector corresponding to the local search result is queried in the vector database, and the average vector is calculated.

[0022] Based on the average vector, a vector similarity search is performed in the vector database to obtain secondary search results; wherein, the secondary search results include target vectors in the vector database that are similar to the average vector, and similarity scores between the target vectors and the search vector.

[0023] Optionally, after the vector database performs similarity retrieval and obtains the retrieval results, it further includes:

[0024] The search results are associated with the task identifier of the search task and then stored in the task table of the columnar database;

[0025] The step of obtaining the search results from the vector database by performing vector similarity retrieval includes:

[0026] Based on the task identifier specified in the query request, retrieve the search results associated with the task identifier from the task table of the columnar database.

[0027] Optionally, the task table also includes the task status of the retrieval task, and the task status is updated to "completed" after all primary keys in the primary key list have completed similarity retrieval.

[0028] The step of retrieving search results associated with the task identifier from the task table of the columnar database based on the task identifier specified in the query request includes:

[0029] Based on the task identifier specified in the query request, query the task status of the task identifier from the task table of the columnar database;

[0030] If the task status is "completed", then further query the search results associated with the task identifier.

[0031] Optionally, the dimensional constraint information includes at least one of the following: time range, label filtering.

[0032] Optionally, the columnar database includes ClickHouse.

[0033] Optionally, the vector database includes Milvus.

[0034] According to a second aspect of the embodiments of this specification, a data retrieval apparatus is provided, applied to a storage system including a columnar database and a vector database; wherein the vector database stores vectors of raw data and primary keys of the raw data, the columnar database stores metadata of the raw data and primary keys of the raw data, and the vectors and metadata of the same raw data are associated between the vector database and the columnar database through a unique primary key; the apparatus includes:

[0035] The receiving unit receives the initiated retrieval task, which includes dimensional constraint information and a retrieval vector of the information to be retrieved.

[0036] An execution unit, in response to the retrieval task, asynchronously executes the retrieval task, the execution unit further comprising:

[0037] The first retrieval subunit transmits the specified dimension constraint information in the retrieval task to the columnar database, and filters the metadata and corresponding primary key list within the scope of the dimension constraint information in the columnar database.

[0038] The second retrieval subunit transmits the primary key list and the specified retrieval vector in the retrieval task to the vector database, performs vector similarity retrieval in the vector database, and obtains retrieval results; wherein, the retrieval results include target vectors that are similar to the retrieval vector and the vector corresponding to each primary key in the primary key list, as well as the similarity scores between the target vectors and the retrieval vector.

[0039] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0040] The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0041] According to a fourth aspect of the embodiments of this specification, a machine-readable storage medium is provided, the machine-readable storage medium storing machine-readable instructions, which, when invoked and executed by a processor, implement the above-described method.

[0042] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0043] By organically combining columnar and vector databases, a hybrid storage system with complementary advantages is formed. The columnar database focuses on lower-cost metadata storage and efficient complex query processing, while the vector database handles efficient vector retrieval, avoiding performance bottlenecks and resource waste caused by the mixed functionality of a single database. This simultaneously meets the diverse retrieval needs of massive data scenarios. Attached Figure Description

[0044] Figure 1 A schematic diagram illustrating data retrieval from a single-vector database using existing technology;

[0045] Figure 2 This is a schematic diagram illustrating the cross-database retrieval capabilities of existing technologies.

[0046] Figure 3 A flowchart of a data retrieval method provided as an exemplary embodiment of this specification;

[0047] Figure 4 A flowchart of asynchronous task execution provided as an exemplary embodiment of this specification;

[0048] Figure 5 A query diagram of the results page provided for an exemplary embodiment of this specification;

[0049] Figure 6 A schematic diagram of the structure of an electronic device in which a data retrieval device is located, provided as an exemplary embodiment of this specification;

[0050] Figure 7 This is a block diagram of a data retrieval device provided for an exemplary embodiment of this specification. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0052] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0053] As mentioned earlier, vector retrieval can generally be divided into two types of schemes. One type uses a single vector database to store all data (including data vectors and metadata); the other type stores vectors and metadata separately, with vectors stored in a separate vector database and metadata stored in a separate relational database, and performs related retrievals through primary keys.

[0054] Please refer to the following. Figure 1 The diagram illustrates data retrieval in a single vector database. When a user initiates a similarity search request, the retrieval module directly performs similarity calculations in the vector database, filters out similar vectors based on a distance metric algorithm, and returns their corresponding metadata. For example, in an image retrieval scenario, image feature vectors and metadata such as time and labels are all stored in the same vector data. During a query, results can only be returned based on vector similarity sorting, making it impossible to quickly filter based on time range or labels.

[0055] In the first type of solution, since the single vector database is mainly for vector storage, when facing vector retrieval, the single vector database not only needs to store vectors but also metadata, resulting in low storage space utilization (vector databases lack optimization for metadata storage), inefficient complex queries (vector databases have poor query performance when performing complex queries based on metadata), and limited retrieval scale (vector databases have a limit on the number of queries per time, and the retrieval response time increases exponentially when the number of queries exceeds the limit).

[0056] Please refer to the following. Figure 2 The diagram illustrates a multi-database linked search. When a user initiates a similarity search request, the search module directly performs similarity calculations in the vector database, filters out similar vectors based on a distance metric algorithm, and returns their corresponding metadata. For example, in an image retrieval scenario, image feature vectors and metadata such as time and labels are all stored in the same vector data. During a query, results can only be returned based on vector similarity sorting, making it impossible to quickly filter based on time range or labels.

[0057] In the second approach, after a user initiates a query, similar vectors are first retrieved from the vector database to obtain a list of primary keys. Then, the corresponding metadata is queried from the relational database using the primary keys. Finally, the results are integrated and returned. Each pagination requires an independent vector retrieval operation, significantly increasing response time, especially in high-concurrency scenarios, making it unsuitable for real-time recommendations and instant retrieval. Furthermore, when the data volume exceeds the processing capacity of both relational and vector databases, the retrieval response time increases exponentially and cannot overcome the limitation on the number of results retrieved in a single query, thus failing to achieve efficient retrieval of the entire dataset.

[0058] To address the technical issues of wasted storage resources, inefficient complex queries, limited retrieval scale, and high latency in paginated queries inherent in the first and second types of solutions mentioned above, this specification aims to provide a data retrieval solution that organically combines columnar databases and vector databases to form a hybrid storage system with complementary advantages. The columnar database focuses on lower-cost metadata storage and efficient complex query processing, while the vector database handles efficient vector retrieval, avoiding performance bottlenecks and resource waste caused by the mixed functions of a single database. This simultaneously meets the diverse retrieval needs of massive data scenarios.

[0059] The following is in conjunction with the appendix Figures 3 to 5 This manual provides a detailed explanation of the data retrieval methods.

[0060] Please refer to Figure 3 , Figure 3 This is a flowchart of a data retrieval method provided as an exemplary embodiment of this specification.

[0061] First, this method can be applied to a storage system including columnar databases and vector databases; wherein, the vector database stores vectors of the original data and the primary key of the original data, and the columnar database stores metadata of the original data and the primary key of the original data, and the vectors and metadata of the same original data are associated between the vector database and the columnar database through a unique primary key; the method includes:

[0062] Step 310: Receive the initiated retrieval task, which includes dimensional constraint information and a retrieval vector of the information to be retrieved;

[0063] Step 320: In response to the retrieval task, the retrieval task is executed asynchronously, wherein the asynchronous execution of the retrieval task includes:

[0064] Step 322: Transmit the specified dimension constraint information in the retrieval task to the columnar database, and filter the metadata and corresponding primary key list within the scope of the dimension constraint information in the columnar database;

[0065] Step 324: Transmit the primary key list and the specified search vector in the search task to the vector database, perform vector similarity search in the vector database, and obtain search results; wherein, the search results include target vectors that are similar to the search vectors corresponding to each primary key in the primary key list, as well as similarity scores between the target vectors and the search vectors.

[0066] In this embodiment, the original data is split into metadata and vectors. The metadata is stored in a columnar database, leveraging its columnar storage and high compression characteristics for efficient storage. Furthermore, because the columnar database supports complex filtering queries, metadata queries under complex dimensional constraints can be implemented. Simultaneously, the vectors are stored in a vector database, utilizing its vector indexing and fast retrieval capabilities for efficient vector retrieval. Finally, a connection is established between the columnar database and the vector database using a common primary key between the metadata and vectors to achieve efficient collaborative retrieval between the two databases.

[0067] In this specification, the columnar databases mentioned include, but are not limited to, ClickHouse, Apache HBase, Vertica, Greenplum, SQLite, etc.

[0068] Taking ClickHouse as an example, ClickHouse boasts high-performance querying, high-concurrency processing capabilities, and efficient storage and compression mechanisms. Its distributed architecture and efficient querying capabilities make it ideal for handling large-scale real-time retrieval tasks. Furthermore, leveraging ClickHouse's efficient data compression (a natural advantage of columnar databases), it can support massive metadata storage at a lower hardware cost, reducing hardware resource investment in storage systems while meeting diverse retrieval needs in massive data scenarios.

[0069] In this specification, the vector databases include, but are not limited to, Milvus, FAISS, Pinecone, Weaviate, etc.

[0070] Taking Milvus as an example, Milvus boasts high performance, scalability, flexibility, and compatibility with modern machine learning tools, excelling particularly in large-scale, high-dimensional data retrieval scenarios. Whether in real-time data insertion and query speed, or in supporting multiple indexes and data types, Milvus demonstrates powerful capabilities, making it ideal for large-scale vector data storage and similarity search tasks.

[0071] The following will use ClickHouse and Milvus as examples to illustrate the implementation of the data retrieval method in this specification.

[0072] In this specification, the search task can be created by the user of the search party.

[0073] For example, creating a retrieval task may include the following steps:

[0074] A1. Click to create a search task.

[0075] A2. Select the dimensional constraints for retrieval, the retrieval vector for the information to be retrieved, and click "Execute Task".

[0076] A3. Generate task information in ClickHouse.

[0077] A4. The task creation page displays a message indicating that the task creation is complete, and the created search task begins to be executed asynchronously.

[0078] In this specification, as shown in step 320 above, the asynchronous execution of the retrieval task may include:

[0079] Step 322: Transmit the specified dimension constraint information in the retrieval task to the columnar database, and filter the metadata and corresponding primary key list within the scope of the dimension constraint information in the columnar database.

[0080] Step 324: Transmit the primary key list and the specified search vector in the search task to the vector database, perform vector similarity search in the vector database, and obtain search results; wherein, the search results include target vectors that are similar to the search vectors corresponding to each primary key in the primary key list, as well as similarity scores between the target vectors and the search vectors.

[0081] Please refer to the following. Figure 4 The flowchart shown illustrates the execution of asynchronous tasks. Figure 4 As shown:

[0082] B1. Task begins. First, determine whether it is a secondary retrieval task.

[0083] The secondary retrieval task refers to the task of selecting local retrieval results based on the initial retrieval results after the initial retrieval task. The secondary retrieval task can be used to deeply mine data vectors related to the user's retrieval needs, thereby enriching the retrieval results and improving retrieval accuracy. The secondary retrieval process will be described in later embodiments.

[0084] For the retrieval tasks created through A1-A4 above, they are not secondary retrieval tasks, so B2 is executed.

[0085] B2. If not, retrieve the primary key in ClickHouse based on the dimension constraint information (i.e., step 322 above), and then proceed to B3.

[0086] The dimensional constraint information may include, but is not limited to, time range, label filtering, etc.

[0087] The time range can be limited to the time range within which metadata is written to the columnar database; tag filtering can filter out the metadata of which tags.

[0088] Because columnar databases support complex filtering queries, they can perform metadata queries under complex dimensional constraints. For example, assuming filtering is performed by time range and tag, the final filtered metadata will be the metadata written to the columnar database within that time range, and will not belong to the tags specified by the tag filter.

[0089] B3. Perform similarity retrieval in Milvus using the retrieval vector and the batch primary key list (i.e., step 324 above).

[0090] In one exemplary embodiment, because the vector database has a limit on the number of searches per query, the search response time increases exponentially when the number of searches exceeds the limit.

[0091] Therefore, in step 324, transmitting the primary key list and the specified retrieval vector in the retrieval task to the vector database, and performing vector similarity retrieval in the vector database, may further include:

[0092] According to a preset batch, the primary keys in the primary key list and the specified search vectors in the search task are transmitted to the vector database in batches, and vector similarity search is performed in the vector database in batches.

[0093] Through the above embodiments, the batch retrieval strategy can effectively avoid the single retrieval limit of vector databases. Even when performing a full retrieval of hundreds of millions of data, it can avoid the exponential increase in retrieval response time and improve the user's retrieval experience.

[0094] The number of batches in the preset batch is calculated based on the number of primary keys in the primary key list and the maximum number of searches in a single instance in the vector database.

[0095] Taking Milvus as an example, its maximum number of searches in a single search is 16384. When the number of primary keys does not exceed 16384, there is no need to search in batches. However, when the number of primary keys exceeds 16384, the number of primary keys can be divided by the maximum number of searches in a single search to obtain the required number of batches.

[0096] B4: When performing batch searches, store the search results (such as task ID, primary key, and similarity score) in the task table in ClickHouse.

[0097] B5: Has the batch retrieval been completed? If yes, proceed to B6. If no, repeat B3 and B4 until all primary keys in the primary key list have been retrieved, indicating that the retrieval task is complete.

[0098] B6: Update the task status to "Completed" (ClickHouse inserts a record with the same task ID primary key into the task table; ClickHouse automatically merges records with the same primary key according to their write time, keeping only the most recently written data). End the task.

[0099] In practical applications, when a user initiates a query request for a search task, the search results for that task need to be displayed to the user.

[0100] The embodiments in this specification separate the creation of search tasks from the query of search results. Instead of waiting for the search results to be displayed after the search task is created, the user actively initiates a query request. This supports asynchronous processing of search tasks, reduces the user's query waiting time, and the task status is updated in real time, making it easy for the user to track the search progress.

[0101] Based on this, in an exemplary embodiment, after step 324 above, the following may also be included:

[0102] In response to the initiated query request for the retrieval task, the retrieval results obtained by performing vector similarity retrieval on the vector database are obtained;

[0103] Based on the target primary key corresponding to each target vector in the search results, query the target metadata corresponding to the target primary key from the columnar database;

[0104] The search results are sorted according to the similarity scores between each target vector and the search vector, and the final search results, including the target vector, target metadata, and similarity scores, are displayed.

[0105] In this embodiment, since the retrieval results of the vector database include the target vectors that are similar to the retrieval vector and the similarity score between the target vector and the retrieval vector, and the query results also need to include the metadata of the original data, it is also necessary to query the target metadata from the columnar database, and finally display the final retrieval results containing the target vector, target metadata and similarity score in sorted order by similarity score.

[0106] In an exemplary embodiment, if the vector database obtains search results after performing similarity retrieval, and as shown in B4-B6 above, the search results are associated with the task identifier of the search task and stored in the task table of the columnar database;

[0107] Therefore, the search results obtained by performing vector similarity retrieval on the aforementioned vector database can include:

[0108] Based on the task identifier specified in the query request, retrieve the search results associated with the task identifier from the task table of the columnar database.

[0109] For example, such as Figure 5 The results page shown is a query illustration, as follows: Figure 5 Assuming a user chooses to query search results, the following steps can be performed:

[0110] D1. Select the search task and trigger the query request.

[0111] D2. Obtain target metadata from ClickHouse based on the target primary key (i.e., query the target metadata corresponding to the target primary key from the columnar database according to the target primary key corresponding to each target vector in the search results).

[0112] D4. The final search results are displayed after being sorted according to the similarity scores.

[0113] In addition, after completing the similarity retrieval of all primary keys in the primary key list, the aforementioned vector database further initiates a task status update for the retrieval task to the task table in the columnar database, setting the task status to completed.

[0114] Accordingly, the above-mentioned querying of the task table in the columnar database for search results associated with the task identifier based on the task identifier specified in the query request may further include:

[0115] Based on the task identifier specified in the query request, query the task status of the task identifier from the task table of the columnar database;

[0116] If the task status is "completed", then further query the search results associated with the task identifier.

[0117] In this embodiment, the task status is used to determine whether the search is complete, and then the search results are displayed to the user after the search is completed.

[0118] In addition to displaying the final search results, this manual also aims to provide users with secondary search functions, such as... Figure 5 As shown: D5. If a secondary search is required, select any local search result; D6. Click "Create Secondary Search Task" to execute the secondary search task asynchronously in the background.

[0119] Specifically, in an exemplary embodiment, it may further include:

[0120] In response to a secondary search task initiated for a local search result selected from the final search results, the vector corresponding to the local search result is queried in the vector database, and the average vector is calculated.

[0121] Based on the average vector, a vector similarity search is performed in the vector database to obtain secondary search results; wherein, the secondary search results include target vectors in the vector database that are similar to the average vector, and similarity scores between the target vectors and the search vector.

[0122] Please refer to the following. Figure 4 The flowchart shown illustrates the execution of asynchronous tasks. Figure 4 As shown:

[0123] At the start of a secondary retrieval task, B1 determines whether it is a secondary retrieval task. Since it is a secondary retrieval task, C1 is executed.

[0124] C1. Obtain the vector of local search results in Milvus.

[0125] C2. Calculate the average vector. (For example, if there are three two-dimensional vectors, [1,2], [3,4] and [5,6], then their average vector is calculated as [(1+3+5) / 3, (2+4+6) / 3], and the calculated average vector is [3,4].

[0126] C3. Use the average vector to perform similarity retrieval in Milvus.

[0127] C4. Store the secondary search results (which may include the task ID, the retrieved vector, the vector's primary key, similarity score, etc.) into the task table in ClickHouse.

[0128] C5. Update the status of the task in ClickHouse to "completed" (i.e., insert a record with the same task ID primary key; ClickHouse will automatically merge records with the same primary key according to the write time, keeping only the most recently written data).

[0129] The above embodiments allow users to perform secondary search tasks based on the final search results, in order to further explore the search results, thereby enriching the search results, improving search accuracy, and meeting users' refined search needs.

[0130] The data retrieval solution provided in this manual has high versatility and can be applied to a variety of different application scenarios, such as, but not limited to, recommendation systems for e-commerce scenarios, recommendation systems for music platforms, and retrieval systems for academic literature.

[0131] In e-commerce scenarios, user behavior vectors can be stored in Milvus, while metadata such as product attributes and sales data can be stored in ClickHouse. When a user searches for a product, the system recommends similar products based on vector similarity and filters them using metadata conditions (such as price range and sales volume). Users can select products of interest on the recommendation results page, and the system generates a more accurate recommendation list through secondary queries.

[0132] In academic literature retrieval, for research databases, document feature vectors are stored in Milvus, while metadata such as document title, author, and publication date are stored in ClickHouse. After a user enters keywords, the system first retrieves relevant literature based on vector similarity and then filters it using metadata conditions (such as journal level and year). Users can select a subset of documents on the results page, and the system will recommend other literature with similar research directions through a secondary query, assisting academic research. It is worth noting that the metadata tagging system and vector feature extraction algorithm can be optimized based on the characteristics of the document data, thereby further improving retrieval accuracy.

[0133] In an exemplary embodiment of this specification, an apparatus capable of implementing the above-described method is also provided.

[0134] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 6 At the hardware level, the device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor reading the corresponding computer program from non-volatile memory into memory and then running it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0135] Please refer to Figure 7 In one software implementation, a data retrieval device is provided, applied to a storage system including a columnar database and a vector database; wherein, the vector database stores vectors of the original data and the primary key of the original data, the columnar database stores metadata of the original data and the primary key of the original data, and the vectors and metadata of the same original data are associated between the vector database and the columnar database through a unique primary key; the device includes:

[0136] The receiving unit 710 receives the initiated retrieval task, which includes dimensional constraint information and a retrieval vector of the information to be retrieved.

[0137] Execution unit 720, in response to the retrieval task, asynchronously executes the retrieval task, and the execution unit 720 further includes:

[0138] The first retrieval subunit 722 transmits the specified dimension constraint information in the retrieval task to the columnar database, and filters the metadata and corresponding primary key list within the scope of the dimension constraint information in the columnar database.

[0139] The second retrieval subunit 724 transmits the primary key list and the specified retrieval vector in the retrieval task to the vector database, performs vector similarity retrieval in the vector database, and obtains retrieval results; wherein, the retrieval results include target vectors that are similar to the retrieval vectors corresponding to each primary key in the primary key list, as well as similarity scores between the target vectors and the retrieval vectors.

[0140] Optionally, the second retrieval subunit 724 includes:

[0141] The batch retrieval subunit transmits the primary keys in the primary key list and the specified retrieval vectors in the retrieval task to the vector database in batches according to a preset batch, and performs vector similarity retrieval in the vector database in batches.

[0142] Optionally, the batch size of the preset batch is calculated based on the number of primary keys in the primary key list and the maximum number of searches per time in the vector database.

[0143] Optional, also includes:

[0144] The query unit, in response to an initiated query request for a retrieval task, obtains the retrieval results obtained by performing vector similarity retrieval on the vector database; queries the target metadata corresponding to the target primary key from the columnar database based on the target primary key corresponding to each target vector in the retrieval results; sorts the target vectors in the retrieval results according to the similarity scores between each target vector and the retrieval vector, and displays the final retrieval results after sorting, including the target vectors, target metadata, and similarity scores.

[0145] Optional, also includes:

[0146] The secondary retrieval unit, in response to a secondary retrieval task initiated for a local retrieval result selected from the final retrieval results, queries the vector database for the vector corresponding to the local retrieval result and calculates the average vector; based on the average vector, it performs vector similarity retrieval in the vector database to obtain the secondary retrieval result; wherein, the secondary retrieval result includes target vectors in the vector database that are similar to the average vector and the similarity score between the target vectors and the retrieval vector.

[0147] Optionally, after the second retrieval subunit 724, the system further includes:

[0148] The storage subunit associates the search results with the task identifier of the search task and stores them in the task table of the columnar database;

[0149] The query unit, when obtaining the retrieval results obtained by vector similarity retrieval from the vector database, includes: querying the retrieval results associated with the task identifier from the task table of the columnar database according to the task identifier specified in the query request.

[0150] Optionally, the task table also includes the task status of the retrieval task, and the task status is updated to "completed" after all primary keys in the primary key list have completed similarity retrieval.

[0151] The query unit, when querying the search results associated with the task identifier from the task table of the columnar database according to the task identifier specified in the query request, includes: querying the task status of the task identifier from the task table of the columnar database according to the task identifier specified in the query request; if the task status is completed, then further querying the search results associated with the task identifier.

[0152] Optionally, the dimensional constraint information includes at least one of the following: time range, label filtering.

[0153] Optionally, the columnar database includes ClickHouse.

[0154] Optionally, the vector database includes Milvus.

[0155] For details on the implementation process of the functions and roles of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned data retrieval method. For relevant parts, please refer to the description of the method implementation method. It will not be repeated here.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of the solution described in this specification, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0157] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0158] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0160] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0163] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0164] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0165] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data retrieval method, characterized in that, An application to a storage system comprising columnar databases and vector databases; wherein the vector database stores vectors of the original data and the primary key of the original data, and the columnar database stores metadata of the original data and the primary key of the original data, and the vectors and metadata of the same original data are associated between the vector database and the columnar database through a unique primary key; the method includes: Receive an initiated retrieval task, the retrieval task including dimensional constraint information and a retrieval vector of the information to be retrieved; In response to the search task, the search task is executed asynchronously, the asynchronous execution of the search task includes: The specified dimension constraint information in the retrieval task is transmitted to the columnar database, and the metadata and corresponding primary key list within the scope of the dimension constraint information are filtered in the columnar database. The primary key list and the specified search vector in the search task are transmitted to the vector database, and vector similarity search is performed in the vector database to obtain search results; wherein, the search results include target vectors that are similar to the search vectors corresponding to each primary key in the primary key list, as well as the similarity scores between the target vectors and the search vectors.

2. The method according to claim 1, characterized in that, The step of transmitting the primary key list and the specified search vector in the search task to the vector database, and performing vector similarity search in the vector database, includes: According to a preset batch, the primary keys in the primary key list and the specified search vectors in the search task are transmitted to the vector database in batches, and vector similarity search is performed in the vector database in batches.

3. The method according to claim 2, characterized in that, The batch size of the preset batch is calculated based on the number of primary keys in the primary key list and the maximum number of searches in a single instance in the vector database.

4. The method according to claim 1, characterized in that, Also includes: In response to the initiated query request for the retrieval task, the retrieval results obtained by performing vector similarity retrieval on the vector database are obtained; Based on the target primary key corresponding to each target vector in the search results, query the target metadata corresponding to the target primary key from the columnar database; The search results are sorted according to the similarity scores between each target vector and the search vector, and the final search results, including the target vector, target metadata, and similarity scores, are displayed.

5. The method according to claim 4, characterized in that, Also includes: In response to a secondary search task initiated for a local search result selected from the final search results, the vector corresponding to the local search result is queried in the vector database, and the average vector is calculated. Based on the average vector, a vector similarity search is performed in the vector database to obtain secondary search results; wherein, the secondary search results include target vectors in the vector database that are similar to the average vector, and similarity scores between the target vectors and the search vector.

6. The method according to claim 4, characterized in that, After performing similarity retrieval on the vector database and obtaining the retrieval results, the method further includes: The search results are associated with the task identifier of the search task and then stored in the task table of the columnar database; The step of obtaining the search results from the vector database by performing vector similarity retrieval includes: Based on the task identifier specified in the query request, retrieve the search results associated with the task identifier from the task table of the columnar database.

7. The method according to claim 6, characterized in that, The task table also includes the task status of the retrieval task, and after all primary keys in the primary key list have completed similarity retrieval, the task status is updated to "completed". The step of retrieving search results associated with the task identifier from the task table of the columnar database based on the task identifier specified in the query request includes: Based on the task identifier specified in the query request, query the task status of the task identifier from the task table of the columnar database; If the task status is "completed", then further query the search results associated with the task identifier.

8. The method according to claim 1, characterized in that, The dimensional constraint information includes at least one of the following: time range and label filtering.

9. The method according to claim 1, characterized in that, The columnar database includes ClickHouse.

10. The method according to claim 1, characterized in that, The vector database includes Milvus.

11. A data retrieval device, applied to a storage system including columnar databases and vector databases; wherein, The vector database stores vectors of the original data and the primary key of the original data, and the columnar database stores metadata of the original data and the primary key of the original data. Vectors and metadata of the same original data are associated between the vector database and the columnar database through a unique primary key. The device includes: The receiving unit receives the initiated retrieval task, which includes dimensional constraint information and a retrieval vector of the information to be retrieved. An execution unit, in response to the retrieval task, asynchronously executes the retrieval task, the execution unit further comprising: The first retrieval subunit transmits the specified dimension constraint information in the retrieval task to the columnar database, and filters the metadata and corresponding primary key list within the scope of the dimension constraint information in the columnar database. The second retrieval subunit transmits the primary key list and the specified retrieval vector in the retrieval task to the vector database, performs vector similarity retrieval in the vector database, and obtains retrieval results; wherein, the retrieval results include target vectors that are similar to the retrieval vector and the vector corresponding to each primary key in the primary key list, as well as the similarity scores between the target vectors and the retrieval vector.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-10 by executing the executable instructions.

13. A machine-readable storage medium having stored thereon machine-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-10.

Images