Query method and system based on data lake house

Abstract

The application provides a data lake house-based query method and system, wherein the data lake house-based query method comprises the following steps: obtaining scalar query information and vector query information in a query request; performing multi-layer pruning processing on files of a data lake house based on the scalar query information and metadata, to determine a target file set; obtaining a data identifier set corresponding to the target file set; obtaining a candidate vector set corresponding to the data identifier set; and feeding back a query result based on the candidate vector set.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a query method and system based on a data lake. Background Technology

[0002] With the widespread application of artificial intelligence, the need for hybrid queries that combine content similarity (vectors) and precise attributes (scalars) across massive datasets is becoming increasingly urgent. Data lakehouse architecture, with its open and low-cost characteristics, has become the preferred platform for storing such multimodal data. However, achieving efficient hybrid queries on this architecture is not easy. It often requires reading the entire data file before filtering and computation, resulting in significant I / O and network bottlenecks and low efficiency. Alternatively, it may require loading data from the data lakehouse into a proprietary system, introducing data redundancy, consistency issues, and additional architectural complexity, also leading to low efficiency. Summary of the Invention

[0003] To address the existing technical problems, this application provides a query method and system based on a data lake that can improve data query efficiency.

[0004] In a first aspect, embodiments of this application provide a query method based on a data lakehouse, comprising: obtaining scalar query information and vector query information from a query request; performing multi-level pruning on the files of the data lakehouse based on the scalar query information and metadata to determine a target file set; obtaining a data identifier set corresponding to the target file set; obtaining a corresponding candidate vector set based on the data identifier set; and providing query results based on the candidate vector set.

[0005] Secondly, embodiments of this application provide a query system based on a data lakehouse, comprising: a query module, a pre-filtering module, and a fine-ranking module; the query module is used to obtain scalar query information and vector query information from a query request; the pre-filtering module is used to perform multi-level pruning on the files in the data lakehouse based on the scalar query information and metadata to determine a target file set; the query module is also used to obtain a data identifier set corresponding to the target file set; the fine-ranking module is used to obtain a corresponding candidate vector set based on the data identifier set; the query module is also used to provide query results based on the candidate vector set.

[0006] The data lake-based query method provided in the above embodiments involves: obtaining scalar query information and vector query information from the query request; performing multi-level pruning on the files in the data lake based on the scalar query information and metadata to determine the target file set; obtaining the data identifier set corresponding to the target file set; obtaining the corresponding candidate vector set based on the data identifier set; and providing query results based on the candidate vector set. Thus, by prioritizing scalar filtering and metadata-based pruning, most irrelevant data can be filtered out at a lower cost without loading actual data rows. It eliminates the need to export the entire data file for processing, ultimately requiring only vector ranking calculation based on a very small set of data identifiers to obtain the query results, thereby effectively reducing computational costs and improving computational efficiency.

[0007] The query system based on data lakehouses provided in the above embodiments belongs to the same concept as the corresponding query method embodiments based on data lakehouses, and thus has the same technical effect as the corresponding query method embodiments based on data lakehouses, which will not be repeated here. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of a query method based on a data lakehouse in one embodiment of this application;

[0009] Figure 2 This is a schematic diagram of the interaction process in a query method based on a data lakehouse according to another embodiment of this application;

[0010] Figure 3 This is a schematic diagram of a query system based on a data lakehouse in one embodiment of this application.

[0011] Explanation of reference numerals in the attached figures

[0012] 1. Query module; 2. Pre-filtering module; 3. Fine ranking module; 4. Cache module. Detailed Implementation

[0013] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0016] In the following description, the terms "first," "second," and "third" are used merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0017] In their research on data query solutions for Data Lakehouse, the inventors of this application have conducted the following research and analysis:

[0018] Scalar data and queries: Scalar data refers to structured data with well-defined attributes, such as price, category, and date. Queries on scalar data are traditional relational queries, filtered by precise scalar conditions (e.g., price < 100).

[0019] Vector Data and Queries: Vector data refers to high-dimensional mathematical representations derived from non-relational / unstructured data. This transformation process, known as vectorization, is crucial for content understanding. Specifically:

[0020] For text data: a pre-trained language model can be used to encode a sentence or paragraph into a fixed-length vector that captures its semantic information.

[0021] For image data: computer vision models can be used to extract features from the image and generate a feature vector that can represent its visual content.

[0022] These generated vectors form the basis for semantic recognition queries on non-relational data. Vector queries find the most similar data items by calculating the distance between vectors (such as cosine similarity).

[0023] Existing technical solutions mainly fall into two categories, both of which have limitations:

[0024] The first type is the "full load followed by computation" mode, which involves reading a large number of data files from the underlying object storage to the computation engine, and then filtering and computing them. This mode suffers from huge I / O and network bottlenecks, resulting in high latency, high cost, and difficulty in meeting real-time requirements.

[0025] The second type uses a dedicated vector database. This model requires data to be extracted, transformed, and loaded from the data lakehouse into its proprietary system through ETL (Extract, Transform, Load), which violates the single data source principle of the data lakehouse, introduces data redundancy, consistency issues, and additional architectural complexity, and creates "data silos".

[0026] Meanwhile, existing pre-filtering solutions typically employ fixed query flows, such as simply performing scalar filtering followed by vector search, lacking awareness and adaptability to the query content itself. When the filtering effect (i.e., selectivity) of scalar conditions is poor, this rigid flow is not optimal and cannot handle the problem of repetitive computation in high-concurrency scenarios. Therefore, there is an urgent need for a hybrid query solution that can intelligently, efficiently, and "on-site" apply to the data lakehouse.

[0027] In view of this, such as Figure 1 As shown, one embodiment of this application provides a query method based on a data lakehouse, which may include:

[0028] S10: Obtain scalar query information and vector query information from the query request;

[0029] S20: Based on the scalar query information and metadata, perform multi-level pruning on the files of the data lake house to determine the target file set;

[0030] S30: Obtain the data identifier set corresponding to the target file set;

[0031] S40: Obtain the corresponding candidate vector set based on the data identifier set;

[0032] S50: Feedback the query results based on the candidate vector set.

[0033] In this embodiment, a data lakehouse is a data management architecture, also known as a data warehouse, where data files can be stored. A query request can be a request to retrieve a portion of the data files stored in the data lakehouse.

[0034] In one embodiment, step S10 may include: in response to receiving a query request, verifying the sender's identity information of the query request; and in response to successful verification, executing step S10. Verifying the sender's identity information of the query request may include: obtaining the identity ID from the query request and matching it against a preset whitelist based on the identity ID. Executing step S10 in response to successful verification may include: if a match is successful, determining that the verification is successful, confirming that the sender's identity information is valid, and executing step S10. Alternatively, it may also include: if a match fails, returning an illegitimate identity message.

[0035] In one embodiment, a query request may include scalar query information and vector query information. The scalar query information may include scalar conditions, such as structured scalar filtering conditions, which constitute the filtering logic for relational data, i.e., scalar data. The vector query information may be a vector query request, vector search-related instructions, etc., and may include at least one of the following: the target query vector, the number of most similar results to be returned (Top-K), and a similarity metric function (e.g., cosine similarity). The vector query information defines the search target for non-relational data, i.e., vector data.

[0036] In one embodiment, metadata can be metadata corresponding to the data lakehouse file (i.e., data file), such as metadata in open table formats (e.g., Iceberg) or in columnar storage formats (e.g., Parquet).

[0037] In one embodiment, pruning can refer to filtering based on certain conditions. The target file set is a set of data files that match the scalar query information, which includes the data files in the data lake that match the scalar query information.

[0038] In one embodiment, the data identifier set can also be called the candidate ID set, which contains the data identifier (ID) corresponding to each data file in the target file set. The data identifier can be obtained from the data identifier column (e.g., the ID column) of the data files in the target file set.

[0039] In one embodiment, obtaining scalar query information and vector query information from the query request can be performed by a query module, such as a query optimizer (CBO). Based on the scalar query information and metadata, performing multi-level pruning on the files in the data lakehouse to determine the target file set and obtaining the data identifier set corresponding to the target file set can be performed by a pre-filtering module. Before step S20, the process may further include sending the scalar query information to the pre-filtering module via the query module.

[0040] In one embodiment, sending scalar query information to the pre-filtering module via the query module may include: querying the metadata corresponding to the files of the data lake house based on the scalar query information to determine the scalar selectivity; wherein the scalar selectivity represents the amount of data that the scalar query information is expected to filter; and if the scalar selectivity is greater than a preset upper limit, sending the scalar query information to the pre-filtering module via the query module.

[0041] In one embodiment, when the scalar selectivity is greater than a preset upper limit, sending scalar query information to the pre-filtering module through the query module may include: when the scalar selectivity is greater than the preset upper limit, generating a cache key based on the scalar query information through the query module; sending a retrieval request to the cache module based on the cache key; and in response to receiving a feedback message from the cache module indicating that the retrieval request was not hit, sending the scalar query information to the pre-filtering module through the query module.

[0042] In one embodiment, the above steps may further include: in response to receiving the data identifier set associated with the cache key, sending the data identifier set to the fine ranking module through the query module; obtaining the corresponding candidate vector set through the fine ranking module based on the data identifier set; and providing query results based on the candidate vector set.

[0043] In one embodiment, the above steps may further include: when the scalar selectivity is less than a preset upper limit, performing clustering and locating of the files in the data lake house based on the vector query information by the query module; sending the scalar query information to the pre-filtering module by the query module; and performing multi-level pruning on the clustered and located files by the pre-filtering module based on the scalar query information and the metadata to determine the target file set.

[0044] In one embodiment, obtaining the corresponding candidate vector set based on the data identifier set can be performed by a fine-ranking module. Before step S40, the process may further include: sending the data identifier set to the query module via a pre-filtering module; and sending the data identifier set to the fine-ranking module via the query module.

[0045] In one embodiment, before sending the data identifier set to the fine-ranking module through the query module, the method may further include: associating the data identifier set with the cache key and then writing it into the cache module.

[0046] In one embodiment, providing query results based on the candidate vector set may include: calculating the similarity between the candidate vectors in the candidate vector set and the vector query information through a fine-ranking module, and determining a preset number of candidate vectors with the highest similarity; sending the data identifiers and corresponding similarities of the preset number of candidate vectors to the query module through the fine-ranking module; obtaining the complete data rows corresponding to the preset number of candidate vectors through the query module, and providing query results based on the complete data rows and corresponding similarities.

[0047] In one embodiment, the candidate vector set is obtained by constraint search based on the data identifier set, and the candidate vector set contains the candidate vectors corresponding to the nodes and neighboring nodes of the data identifier set.

[0048] In one embodiment, the query results may include complete data rows corresponding to a preset number of candidate vectors that are closest to the vector query information (i.e., have the highest similarity), and the similarity scores corresponding to each of the preset number of candidate vectors. The complete data rows may contain all scalar attributes, the similarity scores may be similarity scores, and the preset number may be represented by K, etc.

[0049] In this way, by prioritizing scalar filtering and performing metadata pruning, most irrelevant data can be filtered out at a lower cost without loading actual data rows. There is no need to export the full data file for processing. Ultimately, only vector ranking calculation is needed based on a very small set of data identifiers to obtain the query results, thereby effectively reducing computational costs and improving computational efficiency.

[0050] In related technologies, many commercial vector libraries (such as those that write vectors and metadata into the index together) typically adopt a static or index-first execution path. That is, they prioritize performing Approximate Nearest Neighbor (ANN) retrieval at the index layer before attribute filtering, or rely on the filtering capabilities of the index itself. These solutions often require copying data to the vector indexing system, leading to data silos and increased governance complexity.

[0051] In view of this, the embodiments of this application incorporate CBO into hybrid decision-making during the query inbound stage, enabling the selection of the most resource-efficient execution order driven by statistical information, thereby avoiding the inefficiency of fixed processes in low selectivity or high-concurrency scenarios. Therefore, in step S20 above, the multi-level pruning of the data lake house files based on the scalar query information and metadata may include:

[0052] Based on the scalar query information, the metadata corresponding to the files in the data lake house is queried to determine the scalar selectivity; wherein, the scalar selectivity represents the amount of data that the scalar query information is expected to filter.

[0053] When the scalar selectivity is greater than a preset upper limit, the files in the data lake are subjected to multi-level pruning based on the scalar query information and the metadata.

[0054] In one embodiment, determining the scalar selectivity by querying the metadata corresponding to the files in the data lake based on the scalar query information can be performed by a query module. Step S22 may include: if the scalar selectivity is greater than a preset upper limit, sending the scalar query information to a pre-filtering module via the query module, and then performing multi-level pruning on the files in the data lake based on the scalar query information and the metadata.

[0055] In one embodiment, scalar selectivity can characterize the estimated amount of data that the scalar query information can filter, i.e., the expected amount of data to be filtered, or the amount of data that satisfies the scalar query information, i.e., the amount of data remaining after filtering. Here, the expected amount of data to be filtered can be expressed as the proportion of the expected amount of data to be filtered out of the total amount of data in the data lake, and the amount of data that satisfies the scalar query information can be expressed as the proportion of the amount of data that satisfies the scalar query information of the total amount of data in the data lake. The total amount of data in the data lake can refer to the amount of metadata corresponding to all files in the data lake.

[0056] Among them, the higher the scalar selectivity, the larger the amount of data that the scalar query information can filter (for example, the larger the proportion of the amount of data that the scalar query information can filter in the total amount of data in Data Lake House), and the smaller the amount of data that satisfies the scalar query information (for example, the smaller the proportion of the amount of data that satisfies the scalar query information in the total amount of data in Data Lake House).

[0057] In one embodiment, if the scalar selectivity is greater than a preset upper limit, it indicates that the scalar query information can filter a large amount of data, meaning that the amount of data satisfying the scalar query information is relatively small. For example, this could be represented by the fact that the amount of data satisfying the scalar query information accounts for a small percentage of the total data in the data lake. In this case, the pre-filtering module performs pruning based on the scalar query information.

[0058] In one embodiment, when the scalar selectivity is less than a preset upper limit, the amount of data that satisfies the scalar query information is relatively large, for example, the amount of data satisfying the scalar query information accounts for a large proportion of all data in the data lake. In this case, clustering and localization are performed on the files in the data lake based on the vector query information; based on the scalar query information and the metadata, multi-level pruning is performed on the clustered and localized files to determine the target file set.

[0059] In one embodiment, clustering and locating the files in the data lake based on the vector query information can be performed by the query module, or the vector query information can be sent to the fine-ranking module by the query module and then performed by the fine-ranking module.

[0060] In one embodiment, multi-level pruning processing of the clustered and located files based on the scalar query information and the metadata can be performed by a pre-filtering module. For example, the scalar query information can be sent to the pre-filtering module by the query module and then the pre-filtering module can perform the pruning.

[0061] In one embodiment, cluster localization is coarse-grained. By first filtering out some irrelevant data through cluster localization, and then performing scalar filtering and vector search, efficiency can be improved.

[0062] In one embodiment, in step S21, the query module may use a cost model to determine the scalar selectivity.

[0063] Thus, if scalar selectivity is high (e.g., estimated to filter out 99% of the data), a "scalar first, vector later" plan is generated, prioritizing an efficient multi-level metadata pruning operation that only involves relational data. If scalar selectivity is low (e.g., estimated to filter out only 5% of the data), an alternative plan may be generated, such as first performing coarse-grained clustering using vector indexes, and then performing scalar filtering. This achieves dynamic adjustment of the hybrid query execution strategy based on the characteristics of relational data, while considering both scalar selectivity and vector computation costs. This allows for adaptive selection of the execution path, reducing unnecessary vector retrieval or large-scale I / O, reducing latency, and improving stability.

[0064] In some embodiments, the step described above, where the scalar selectivity is greater than a preset upper limit, involves performing multi-level pruning on the data lake's files based on the scalar query information and the metadata, which may include:

[0065] If the scalar selectivity is greater than a preset upper limit, a cache key is generated based on the scalar query information;

[0066] A request to retrieve the cache key is sent to the cache module.

[0067] In response to receiving a feedback message indicating that the caching module failed to retrieve the request, the files in the data lake are subjected to multi-level pruning based on the scalar query information and the metadata.

[0068] In one embodiment, the cache key corresponds to scalar query information. For example, generating the cache key based on the scalar query information may include generating the cache key based on deterministic scalar conditions in the scalar query information. The cache key fully represents the relational filtering portion of the query.

[0069] In one embodiment, generating a cache key based on the scalar query information can be performed by the query module. Sending a retrieval request to the cache module based on the cache key can refer to the query module sending a retrieval request carrying the cache key to the cache module.

[0070] In one embodiment, before receiving the feedback message indicating that the retrieval request failed from the caching module, the process may further include: receiving the retrieval request through the caching module, and querying whether there is an associated set of data identifiers based on the cache key in the retrieval request; if it exists, returning the set of data identifiers; if it does not exist, returning the feedback message indicating that the retrieval request failed.

[0071] In one embodiment, a retrieval request miss can indicate that the caching module does not have a set of data identifiers associated with the cache key in the retrieval request.

[0072] In one embodiment, in response to receiving the set of data identifiers associated with the cache key, a corresponding set of candidate vectors is obtained based on the set of data identifiers; and a query result is returned based on the set of candidate vectors.

[0073] Here, in response to receiving the feedback message indicating that the caching module failed to retrieve the request, based on the scalar query information and the metadata, multi-level pruning is performed on the files in the data lake to determine the target file set, and steps S30~S50 are executed. In response to receiving the data identifier set associated with the cache key, the corresponding candidate vector set is obtained based on the data identifier set; the query result is fed back based on the candidate vector set.

[0074] In one embodiment, in response to receiving the set of data identifiers associated with the cache key, obtaining a corresponding candidate vector set based on the set of data identifiers includes: in response to receiving the set of data identifiers associated with the cache key, performing a constraint search based on the set of data identifiers, traversing the nodes corresponding to the set of data identifiers and their adjacent nodes to obtain the corresponding candidate vector set.

[0075] In one embodiment, the step of providing query results based on the candidate vector set includes: calculating the similarity between the candidate vectors in the candidate vector set and the vector query information; determining a preset number of candidate vectors with the highest similarity; obtaining the complete data rows corresponding to the preset number of candidate vectors; and providing query results based on the complete data rows and the corresponding similarity.

[0076] Thus, when a cache key-based retrieval request is hit, it indicates that an identical scalar query has been executed recently, allowing the corresponding data identifier set to be returned directly, significantly shortening the query path. Existing technologies typically focus on caching index structures or hot vectors, but rarely materialize the "data identifier set obtained from scalar filtering" as a reusable intermediate result. In contrast, this embodiment treats the data identifier set as a cache asset, providing a "fast track" for frequently repeated queries and reducing repetitive input / output (I / O) and computational resource overhead.

[0077] In related technologies, existing solutions mostly rely on copying metadata to the index or filtering it within the index. There are also "full load" solutions that directly read and filter data rows. These methods are efficient for filtering within the index, but they usually require modifications to the index structure or the creation of dedicated indexes for different attributes, resulting in additional costs and complexity.

[0078] In some embodiments, in response to receiving a feedback message indicating that the caching module has not found a retrieved request, the data lake's files are subjected to multi-level pruning based on the scalar query information and the metadata, including:

[0079] In response to receiving a feedback message from the cache module indicating that the retrieval request was not hit, the data lake house files are pruned for the first time based on a comparison between the scalar query information and the table-level partition range corresponding to the metadata.

[0080] Based on the comparison between the scalar query information and the footer metadata of the remaining files after the first pruning, a second pruning is performed on the remaining files after the first pruning.

[0081] Based on the scalar query information and the row group metadata corresponding to the remaining files after the second pruning, a third pruning is performed on the remaining files after the second pruning.

[0082] Based on the comparison between the scalar query information and the page-level metadata corresponding to the remaining files after the third pruning, a fourth pruning is performed on the remaining files after the third pruning.

[0083] In one embodiment, the metadata may include open table format metadata, columnar storage format metadata, and columnar storage format metadata may include footer metadata, row group metadata, and page-level metadata, etc.

[0084] In one embodiment, in response to receiving a feedback message indicating that the caching module indicated a missed request, the data lake's files are pruned for the first time based on a comparison between the scalar query information and the table-level partition range corresponding to the metadata. This includes: in response to receiving a feedback message indicating that the caching module indicated a missed request, sending the scalar query information to the pre-filtering module through the query module; and performing the first pruning of the data lake's files through the pre-filtering module based on a comparison between the scalar query information and the table-level partition range corresponding to the metadata.

[0085] In one embodiment, the second, third, and fourth pruning are all performed through a pre-filtering module.

[0086] In one embodiment, the first pruning is partition pruning, specifically table-level partition pruning. This involves comparing the partition columns representing the table-level partition range of the metadata with the scalar conditions indicated by the scalar query information, thus skipping entire irrelevant table partitions directly at the metadata level.

[0087] In one embodiment, the initial pruning is achieved through a pre-filtering module interacting with the metadata service of an open table format (such as Iceberg). For example, an open table format (such as the manifest file in Iceberg) records the range of table-level partitions covered by each data file, i.e., the range of partition values. The pre-filtering module reads these lightweight metadata files, comparing scalar conditions with the partition ranges, thereby excluding all data files under a large number of irrelevant partitions. This step can reduce the number of files to be scanned by several orders of magnitude.

[0088] In one embodiment, the second pruning is file-level pruning, where the footer metadata may include statistical metadata from the footer. For the remaining files after the first pruning, only the statistical metadata in their footers is read, and scalar conditions are compared with this statistical metadata, thereby skipping irrelevant parts of the file.

[0089] In one embodiment, during the second pruning process, for the remaining files that passed the first pruning, the pre-filtering module does not immediately read their contents, but instead reads the footer metadata of each file. The footer contains statistical information for all columns within that file.

[0090] For example, by comparing a scalar condition (e.g., price < 100) with metadata in the price column of a file (e.g., the minimum value min_price), it is possible to safely filter out data in the entire file that does not contain the condition, thereby removing it from the list to be scanned.

[0091] In one embodiment, the third pruning is row-group-level pruning. Within the file, the filtering logic of the second pruning is repeated for each row group, using more granular statistical information for filtering.

[0092] In one embodiment, during the third pruning process, for surviving files, the pre-filtering module continues to delve deeper, reading the metadata of the row groups corresponding to the remaining files—that is, the metadata of each row group within the remaining files. For example, a columnar storage format (Parquet) file typically contains multiple row groups, each with independent metadata, i.e., column statistics. This step enables finer-grained pruning within a single file, further reducing the amount of data that needs to be actually decompressed and read.

[0093] In one embodiment, the fourth pruning is page-level pruning, or page-level metadata pruning. It uses page-level metadata such as Bloom filters or dictionary encoding to further determine whether a data page may contain a value that satisfies the scalar condition, thereby avoiding reading that page.

[0094] In one embodiment, at the finest level during the fourth pruning process, page-level metadata can include Bloom filters and dictionary encodings. For example, if the Parquet file was built with a Bloom filter for a specific column (such as a high-cardinality ID column) or used dictionary encoding on a low-cardinality column, the pre-filtering module can utilize this information. For instance, a Bloom filter can quickly determine whether a value indicated by a scalar condition "exists" in a data page, thus determining whether to filter that page.

[0095] In this way, the responsibility of "coarse screening" is transferred to the metadata layer of the data lakehouse. Based on the four-level pruning and screening, the number of candidates is greatly reduced at the storage layer without copying the master data, which greatly reduces the computing cost and improves the computing efficiency.

[0096] In some embodiments, after obtaining the data identifier set corresponding to the target file set, the method further includes:

[0097] The data identifier set is associated with the cache key and then written into the cache module.

[0098] In one embodiment, in response to receiving a feedback message indicating that the caching module failed to retrieve the request, the files in the data lake are subjected to multi-level pruning based on the scalar query information and the metadata to determine the target file set; the data identifier set corresponding to the target file set is obtained; and the data identifier set is associated with the cache key and written to the caching module.

[0099] In one embodiment, the process of associating the data identifier set with the cache key and writing it into the cache module is performed by the query module.

[0100] In one embodiment, obtaining the data identifier set corresponding to the target file set includes: generating the data identifier set corresponding to the target file set through a pre-filtering module, and sending the data identifier set to a query module; the query module receiving the data identifier set corresponding to the target file set.

[0101] Thus, when the data identifier set corresponding to the scalar query information is obtained for the first time, the cache key corresponding to the scalar query information is promptly associated with the data identifier set and stored in the cache module. This allows the data identifier set to be quickly retrieved based on the cache key the next time the same scalar query information needs to be queried, without having to execute steps S20 and S30, thereby improving efficiency.

[0102] In some embodiments, obtaining the corresponding candidate vector set based on the data identifier set includes:

[0103] Based on the data identifier set, a constraint search is performed, traversing the nodes corresponding to the data identifier set and their adjacent nodes to obtain the corresponding candidate vector set.

[0104] In one embodiment, obtaining the corresponding candidate vector set based on the data identifier set includes: sending the data identifier set to the fine ranking module through the query module, performing a constraint search based on the data identifier set through the fine ranking module, traversing the nodes and adjacent nodes corresponding to the data identifier set, and obtaining the corresponding candidate vector set.

[0105] In one embodiment, the fine-ranking module may include one or more vector retrieval engines. These vector retrieval engines can perform constrained searches based on the data identifier set, traversing the nodes corresponding to the data identifier set and their adjacent nodes to obtain a corresponding candidate vector set.

[0106] In one embodiment, if vector data and scalar data are stored separately in the above steps, the vector retrieval engine will load the vector data corresponding to the data identifiers into memory as needed based on the data identifier set, avoiding the full loading of all vectors.

[0107] In one embodiment, the search algorithm of the vector retrieval engine is strictly limited to the subset of data represented by the data identifier. For example, the algorithm's traversal will only be performed between the node corresponding to the data identifier and its neighboring nodes. This reduces a global, high-cost approximate nearest neighbor search problem to a local, low-cost exact or approximate search problem.

[0108] In one embodiment, an index layer ANN, such as a filtered ANN, can be introduced as a secondary accelerator to work with the vector search engine to perform the above steps, thereby complementing each other and further improving efficiency.

[0109] In this way, pruning based on metadata ensures that the set entering vector computation is very small, and then fine sorting is performed on this small set, which can greatly improve the efficiency of the expensive vector fine sorting stage.

[0110] In some embodiments, the step of returning the query result based on the candidate vector set includes:

[0111] Calculate the similarity between the candidate vectors in the candidate vector set and the vector query information;

[0112] Determine the preset number of candidate vectors with the highest similarity;

[0113] Obtain the complete data rows corresponding to the preset number of candidate vectors;

[0114] Based on the complete data row and the corresponding similarity, the query results are fed back.

[0115] In one embodiment, a fine-ranking module calculates the similarity between candidate vectors in the candidate vector set and the vector query information, and determines a preset number of candidate vectors with the highest similarity. The calculated similarity is referred to as a similarity score.

[0116] After determining the preset number of candidate vectors with the highest similarity, the algorithm may further include: sending the preset number of candidate vectors with the highest similarity and their corresponding similarities to the query module through a fine-ranking module.

[0117] In one embodiment, the query module receives a preset number of candidate vectors with the highest similarity and their corresponding similarities, and obtains the complete data rows corresponding to the preset number of candidate vectors; the query module then returns the query results based on the complete data rows and their corresponding similarities.

[0118] In one embodiment, the complete data row can be obtained from the data lakehouse, which contains all scalar attributes.

[0119] In one embodiment, providing query results based on the complete data row and its corresponding similarity can include: associating the complete data row and its corresponding similarity to generate query results, and then providing the query results. The query results can be provided via an application programming interface (API).

[0120] In this way, a readable result set is ultimately formed that includes both similarity scores (non-relational query results) and complete product information (relational data), resulting in accurate and comprehensive feedback.

[0121] As one possible implementation, the query method based on the data lakehouse can include, for example: Figure 2 The specific process is shown below.

[0122] This application also provides a query system based on a data lakehouse, used to execute the query method based on a data lakehouse described in any one or more of the foregoing embodiments. The system may include: a query module 1, a pre-filtering module 2, and a fine-ranking module 3;

[0123] The query module 1 is used to obtain scalar query information and vector query information from the query request;

[0124] The pre-filtering module 2 is used to perform multi-level pruning on the files of the data lake house based on the scalar query information and metadata to determine the target file set;

[0125] The query module 1 is also used to obtain the data identifier set corresponding to the target file set;

[0126] The fine-ranking module 3 is used to obtain the corresponding candidate vector set based on the data identifier set;

[0127] The query module 1 is also used to provide query results based on the candidate vector set.

[0128] In one embodiment, query module 1 may include a query optimizer, which may contain a decision engine based on a cost-based optimizer (CBO). It is not only responsible for parsing query requests, but also for collecting or utilizing pre-existing statistical information to estimate the costs of different execution paths and select the lowest-cost execution plan.

[0129] In one embodiment, the pre-filtering module 2 (Pre-Filter Engine) is the physical execution layer responsible for executing the "coarse-screening" tasks issued by the query module. It is typically implemented by a lightweight Structured Query Language (SQL) engine capable of directly reading metadata. Its core design principle is to minimize I / O and avoid reading any unnecessary data.

[0130] In one embodiment, the Vector Refinement Engine (Module 3) is responsible for performing the expensive vector similarity calculations. This module may contain one or more vector retrieval engines. Its key feature is support for constrained search, which allows it to limit its search scope to a given list of IDs.

[0131] In one embodiment, such as Figure 3 As shown, the system may further include: a cache module 4;

[0132] The query module 1 is also used to send a retrieval request to the cache module 4 based on the cache key. For example, the query module 1 is also used to generate a cache key based on the scalar query information when the scalar selectivity is greater than a preset upper limit value; and to send a retrieval request to the cache module 4 based on the cache key.

[0133] The caching module 4 is used to receive the acquisition request and query whether there is an associated data identifier set based on the cache key in the acquisition request; if it exists, the data identifier set is returned; if it does not exist, a feedback message indicating that the acquisition request was not hit is returned.

[0134] The query module 1 is also used to send scalar query information to the pre-filtering module 2 in response to receiving a feedback message from the cache module 4 indicating that the request was not hit;

[0135] The pre-filtering module 2 is used to receive the scalar query information and perform multi-level pruning on the files of the data lake house based on the scalar query information and the metadata.

[0136] In one embodiment, the Cache Module 4 can be a high-speed, distributed key-value store. It can be used to materialize and cache the set of candidate data identifiers generated by the pre-filtering module, providing a "shortcut" for high-frequency, repetitive query patterns.

[0137] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0139] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A query method based on a data lakehouse, characterized in that, include: Retrieve scalar query information and vector query information from the query request; Based on the scalar query information, the metadata corresponding to the files in the data lake house is queried to determine the scalar selectivity; wherein, the scalar selectivity represents the amount of data that the scalar query information is expected to filter. If the scalar selectivity is greater than a preset upper limit, a cache key is generated based on the scalar query information; A request to retrieve the cache key is sent to the cache module. In response to receiving the set of data identifiers associated with the cache key, a corresponding set of candidate vectors is obtained based on the set of data identifiers; and a query result is returned based on the set of candidate vectors. In response to receiving a feedback message indicating that the caching module failed to retrieve the request, the files in the data lake are subjected to multi-level pruning based on the scalar query information and the metadata to determine the target file set; wherein, the metadata includes at least one of open table format metadata and columnar storage format metadata; the multi-level pruning includes: pruning based on table-level partition range, pruning based on footer metadata, pruning based on row group metadata, and pruning based on page-level metadata; Obtain the data identifier set corresponding to the target file set; Based on the data identifier set, a constraint search is performed, traversing the nodes corresponding to the data identifier set and their adjacent nodes to obtain the corresponding candidate vector set; The query results are fed back based on the set of candidate vectors; the query results include the complete data rows corresponding to a preset number of candidate vectors with the highest similarity to the vector query information, and the similarity of each of the preset number of candidate vectors.

2. The method according to claim 1, characterized in that, The method further includes: If the scalar selectivity is less than a preset upper limit, clustering and localization are performed on the files of the data lake house based on the vector query information; Based on the scalar query information and the metadata, multi-level pruning is performed on the clustered and located files to determine the target file set.

3. The method according to claim 1, characterized in that, In response to receiving a feedback message from the caching module indicating that the request was not found, the system performs multi-level pruning on the files in the data lake based on the scalar query information and the metadata, including: In response to receiving a feedback message from the cache module indicating that the retrieval request was not hit, the data lake house files are pruned for the first time based on a comparison between the scalar query information and the table-level partition range corresponding to the metadata. Based on the comparison between the scalar query information and the footer metadata of the remaining files after the first pruning, a second pruning is performed on the remaining files after the first pruning. Based on the scalar query information and the row group metadata corresponding to the remaining files after the second pruning, a third pruning is performed on the remaining files after the second pruning. Based on the comparison between the scalar query information and the page-level metadata corresponding to the remaining files after the third pruning, a fourth pruning is performed on the remaining files after the third pruning.

4. The method according to claim 1, characterized in that, After obtaining the data identifier set corresponding to the target file set, the method further includes: The data identifier set is associated with the cache key and then written into the cache module.

5. The method according to claim 1, characterized in that, The feedback of query results based on the candidate vector set includes: Calculate the similarity between the candidate vectors in the candidate vector set and the vector query information; Determine the preset number of candidate vectors with the highest similarity; Obtain the complete data rows corresponding to the preset number of candidate vectors; Based on the complete data row and the corresponding similarity, the query results are fed back.

6. A query system based on a data lakehouse, used to execute the query method based on a data lakehouse as described in any one of claims 1 to 5, characterized in that, The system includes: a query module, a pre-filtering module, and a fine-ranking module; The query module is used to obtain scalar query information and vector query information from the query request; and to send a request to the cache module based on the cache key. The caching module is configured to receive the retrieval request and query whether an associated data identifier set exists based on the cache key in the retrieval request; if it exists, the data identifier set is returned; if it does not exist, a feedback message indicating that the retrieval request was not found is returned. The query module is further configured to, in response to receiving a feedback message from the caching module indicating that the request was not found, send scalar query information to the pre-filtering module; in response to receiving the data identifier set associated with the cache key, obtain the corresponding candidate vector set based on the data identifier set; and provide query results based on the candidate vector set. The pre-filtering module is used to receive the scalar query information and perform multi-level pruning on the files of the data lake based on the scalar query information and the metadata. The pre-filtering module is used to query the metadata corresponding to the files in the data lake based on the scalar query information to determine the scalar selectivity; wherein, the scalar selectivity represents the amount of data that the scalar query information is expected to filter; when the scalar selectivity is greater than a preset upper limit, the files in the data lake are subjected to multi-level pruning based on the scalar query information and the metadata to determine the target file set; wherein, the metadata includes at least one of open table format metadata and columnar storage format metadata; the multi-level pruning includes: pruning based on table-level partition range, pruning based on footer metadata, pruning based on row group metadata, and pruning based on page-level metadata; The query module is also used to obtain the data identifier set corresponding to the target file set; The fine-ranking module is used to perform constraint search based on the data identifier set, traversing the nodes and adjacent nodes corresponding to the data identifier set to obtain the corresponding candidate vector set. The query module is also used to provide query results based on the candidate vector set; the query results include complete data rows corresponding to a preset number of candidate vectors with the highest similarity to the vector query information, and the similarity of each of the preset number of candidate vectors.

Images