Data query method and device, electronic equipment and storage medium

Abstract

The invention provides a data query method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving a query request which carries a query condition and a target field needing to be queried; scanning data table metadata based on a partition condition and a field value range condition in the query conditions, and performing file-level filtering on data files stored in a bottom layer to obtain a candidate file set; based on an accurate matching condition and a range query condition in the query conditions and a pre-established row group level index, performing row group level filtering on each data file in the candidate file set to obtain a data row group set meeting the conditions; and reading data of the corresponding column from the data row group set according to the target field to obtain a query result set. The defect that the traditional file-level filtering granularity is insufficient is overcome, the problem that the adaptability of a single index is poor is solved through intelligent index combination, and meanwhile the heterogeneous characteristic of multi-modal data is perfectly adapted.

Description

Technical Field

[0001] This application relates to the field of data query technology, and in particular to a data query method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal data processing has become a core requirement for current AI applications. Various systems need to efficiently manage heterogeneous data such as image features, text descriptions, and audio vectors simultaneously. Apache Iceberg, as a next-generation data lake table format, is widely used in multimodal data storage scenarios due to its excellent metadata management and query optimization capabilities. However, when faced with the small-batch data query requirements unique to deep learning training and online inference scenarios, existing storage architectures face severe challenges in query efficiency, seriously affecting the training speed and real-time inference performance of AI models.

[0003] Current mainstream technical solutions mainly revolve around three directions: optimizing file-level indexes and partitioning strategies based on Iceberg's native functionality to reduce the data scanning range through coarse-grained filtering; introducing row-group-level fine-grained indexes at the storage engine level to improve the filtering accuracy of equality and range queries; and employing pre-computation and materialized views to attempt to pre-process high-frequency query requests. In addition, some research has begun to explore novel architectures that optimize computation and storage in tandem, aiming to better adapt to the needs of multimodal data processing.

[0004] However, the granularity of file-level filtering in existing solutions is still too coarse, which cannot effectively cope with small-batch query scenarios and severely restricts the performance and scalability of AI multimodal applications. Summary of the Invention

[0005] This application provides a data query method, apparatus, electronic device, and storage medium to solve the problem that the granularity of file-level filtering in the prior art is still too coarse and cannot effectively cope with small-batch query scenarios.

[0006] Firstly, this application provides a data query method, including:

[0007] Receive a query request, wherein the query request carries query conditions and the target field to be queried;

[0008] Based on the partitioning conditions and field value range conditions in the query conditions, the data table metadata is scanned, and file-level filtering is performed on the underlying stored data files to obtain a candidate file set.

[0009] Based on the exact match condition and range query condition in the query conditions, as well as the pre-established row group level index, each data file in the candidate file set is filtered at the row group level to obtain a set of data row groups that meet the conditions.

[0010] The data of the corresponding column is read from the data row group set according to the target field to obtain the query result set.

[0011] In one possible implementation, the step of scanning the data table metadata based on the partitioning conditions and field value range conditions in the query conditions, and performing file-level filtering on the underlying stored data files, includes:

[0012] Determine the corresponding partition storage path based on the partitioning conditions;

[0013] Obtain the field value range information of each data file under the partition storage path;

[0014] The candidate file set is obtained by filtering out data files whose field value range information does not overlap with the field value range conditions in the query conditions from the underlying data files.

[0015] In one possible implementation, the row-group-level filtering of each data file in the candidate file set based on the exact match condition and range query condition in the query conditions, as well as the pre-established row-group-level index, includes:

[0016] Parse the exact matching conditions and extract the first field to be matched and the target value;

[0017] For each candidate file, load the probabilistic filtering index of the first field in the candidate file, and determine whether each row group in the candidate file contains the target value through the probabilistic filtering index, and exclude the row group that does not contain the target value;

[0018] Parse the range query conditions and extract the second field to be matched and its value range;

[0019] For each candidate file, the segmented value range index of the second field in the candidate file is loaded, and the segmented value range index is used to determine whether each row group in the candidate file contains data within the value range, and row groups that do not contain data within the value range are excluded.

[0020] In one possible implementation, the method further includes:

[0021] The distribution of the first field's values ​​in each row group is statistically analyzed to obtain the characteristics of the row group data.

[0022] Set the bit array length according to the number of values ​​in the row group data characteristics;

[0023] The number of hash functions is determined based on a preset false positive rate requirement;

[0024] Construct a probabilistic filtering index based on the length of the bit array and the number of hash functions.

[0025] In one possible implementation, the method further includes:

[0026] Analyze the value distribution of the second field across all row groups to obtain the field value distribution characteristics;

[0027] When the field value distribution characteristics indicate that the field is a continuous field, the interval division granularity is determined to obtain the interval division scheme. The field values ​​are then binned according to the interval division scheme to construct a continuous value domain index. The continuous value domain index is then used as the segmented value domain index.

[0028] When the field value distribution characteristics indicate that the field is a discrete field, the frequency of discrete values ​​is counted to obtain the enumerated value distribution. Based on the enumerated value distribution, a discrete value mapping relationship is established, an enumerated mapping index is constructed, and the enumerated mapping index is used as the segmented value domain index.

[0029] In one possible implementation, the process of receiving the query request further includes:

[0030] For each field to be queried in the data table, determine the field characteristics of that field;

[0031] When the field feature indicates that it is a time-series field, the field value of the time-series field is compressed and stored using difference encoding;

[0032] When the field feature indicates that it is a text field and the query frequency meets the preset conditions, a dictionary mapping is constructed and stored for the field value of the text field;

[0033] When the field characteristics indicate that the field size exceeds a preset threshold, the field value is stored in blocks and a block index is created.

[0034] In one possible implementation, the process of receiving the query request further includes:

[0035] Statistically analyze the frequency of each field in the data table in historical query conditions, and determine the field combinations with a frequency exceeding a set threshold as optimized field combinations;

[0036] The data files corresponding to the data table are sorted and stored according to the space filling curve of the optimized field combination.

[0037] Secondly, this application provides a data query device, comprising:

[0038] A receiving module is used to receive query requests, wherein the query request carries query conditions and the target field to be queried;

[0039] The first filtering module is used to scan the metadata of the data table based on the partitioning conditions and field value range conditions in the query conditions, perform file-level filtering on the underlying stored data files, and obtain a candidate file set.

[0040] The second filtering module is used to perform row group-level filtering on each data file in the candidate file set based on the exact matching conditions and range query conditions in the query conditions and the pre-established row group-level index, so as to obtain a set of data row groups that meet the conditions.

[0041] The reading module is used to read data of the corresponding column from the data row group set according to the target field to obtain a query result set.

[0042] In one possible implementation, the first filtering module is specifically used for:

[0043] Determine the corresponding partition storage path based on the partitioning conditions;

[0044] Obtain the field value range information of each data file under the partition storage path;

[0045] The candidate file set is obtained by filtering out data files whose field value range information does not overlap with the field value range conditions in the query conditions from the underlying data files.

[0046] In one possible implementation, the second filtering module is specifically used for:

[0047] Parse the exact matching conditions and extract the first field to be matched and the target value;

[0048] For each candidate file, load the probabilistic filtering index of the first field in the candidate file, and determine whether each row group in the candidate file contains the target value through the probabilistic filtering index, and exclude the row group that does not contain the target value;

[0049] Parse the range query conditions and extract the second field to be matched and its value range;

[0050] For each candidate file, the segmented value range index of the second field in the candidate file is loaded, and the segmented value range index is used to determine whether each row group in the candidate file contains data within the value range, and row groups that do not contain data within the value range are excluded.

[0051] In one possible implementation, the second filtering module is further configured to:

[0052] The distribution of the first field's values ​​in each row group is statistically analyzed to obtain the characteristics of the row group data.

[0053] Set the bit array length according to the number of values ​​in the row group data characteristics;

[0054] The number of hash functions is determined based on a preset false positive rate requirement;

[0055] Construct a probabilistic filtering index based on the length of the bit array and the number of hash functions.

[0056] In one possible implementation, the second filtering module is further configured to:

[0057] Analyze the value distribution of the second field across all row groups to obtain the field value distribution characteristics;

[0058] When the field value distribution characteristics indicate that the field is a continuous field, the interval division granularity is determined to obtain the interval division scheme. The field values ​​are then binned according to the interval division scheme to construct a continuous value domain index. The continuous value domain index is then used as the segmented value domain index.

[0059] When the field value distribution characteristics indicate that the field is a discrete field, the frequency of discrete values ​​is counted to obtain the enumerated value distribution. Based on the enumerated value distribution, a discrete value mapping relationship is established, an enumerated mapping index is constructed, and the enumerated mapping index is used as the segmented value domain index.

[0060] In one possible implementation, the device further includes a storage optimization module for:

[0061] For each field to be queried in the data table, determine the field characteristics of that field;

[0062] When the field feature indicates that it is a time-series field, the field value of the time-series field is compressed and stored using difference encoding;

[0063] When the field feature indicates that it is a text field and the query frequency meets the preset conditions, a dictionary mapping is constructed and stored for the field value of the text field;

[0064] When the field characteristics indicate that the field size exceeds a preset threshold, the field value is stored in blocks and a block index is created.

[0065] In one possible implementation, the storage optimization module is further configured to:

[0066] Statistically analyze the frequency of each field in the data table in historical query conditions, and determine the field combinations with a frequency exceeding a set threshold as optimized field combinations;

[0067] The data files corresponding to the data table are sorted and stored according to the space filling curve of the optimized field combination.

[0068] Thirdly, this application provides an apparatus comprising: a processor and a memory, wherein the processor is configured to execute a data query program stored in the memory to implement the data query method described in any one of the first aspects.

[0069] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the data query method described in any one aspect.

[0070] Compared with the prior art, the technical solution provided in this application has the following advantages: First, the method provided in this application performs file-level coarse screening based on partition and field range information in the query conditions, significantly reducing the initial data scanning range. Then, combining exact matching and range query conditions, a second fine screening is implemented using a pre-built row group-level hybrid index, greatly improving filtering accuracy. Finally, efficient data extraction is achieved through columnar reading of the target field. This hierarchical and progressive query optimization strategy overcomes the shortcomings of insufficient granularity in traditional file-level filtering, solves the problem of poor adaptability of a single index through intelligent index combination, and perfectly adapts to the heterogeneous characteristics of multimodal data, forming a complete and efficient data supply chain from the storage layer to the computing layer, enabling the system to achieve significant performance improvements while maintaining versatility. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0074] Figure 1 A flowchart illustrating an embodiment of a data query method provided in this application;

[0075] Figure 2 A flowchart illustrating an embodiment of another data query method provided in this application;

[0076] Figure 3A data query system architecture diagram provided in this application embodiment;

[0077] Figure 4 A block diagram illustrating an embodiment of a data query device provided in this application;

[0078] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0081] To address the technical problem that existing file-level filtering granularity is still too coarse and cannot effectively handle small-batch query scenarios, this application provides a data query method that overcomes the shortcomings of insufficient granularity in traditional file-level filtering, solves the problem of poor adaptability of a single index through intelligent index combination, and perfectly adapts to the heterogeneous characteristics of multimodal data, forming a complete and efficient data supply chain from the storage layer to the computing layer, enabling the system to achieve significant performance improvement while maintaining versatility.

[0082] Figure 1 This is a flowchart illustrating an embodiment of a data query method provided in this application. Figure 1 As shown, the method includes the following steps:

[0083] Step 101: Receive a query request, wherein the query request carries query conditions and the target field to be queried.

[0084] A query request refers to an instruction initiated by a user to a data system that includes a data retrieval requirement; in this application, it specifically refers to a retrieval operation targeting a multimodal data lake. The query condition is a logical expression that limits the data range (e.g., "time range AND image category"), and the target field is the specified data column to be returned (e.g., image feature vectors, text descriptions, etc.).

[0085] The specific processing steps in this step are as follows: The system obtains the structured query representation through the API interface, parses it to generate a query execution plan containing partition keys, filtering predicates, and a list of field projections. The query conditions are converted into a predicate tree suitable for Iceberg table format processing, and the target fields are mapped to the underlying column storage structure. This step establishes the basic query context for subsequent multi-level filtering, especially providing the necessary input parameters for file-level pruning and row group-level index filtering.

[0086] Step 102: Based on the partitioning conditions and field value range conditions in the query conditions, scan the metadata of the data table, perform file-level filtering on the underlying stored data files, and obtain a set of candidate files.

[0087] Partitioning conditions refer to the filtering expressions in the query conditions that involve the partition key of the data table (such as dt = '2023-01-01'). Field value range conditions are the numerical range restrictions for non-partitioned fields in the query conditions (such as 100). <price<500)。

[0088] In this embodiment of the application, step 102 may specifically include the following steps: determining the corresponding partition storage path according to the partition conditions; obtaining the field value range information of each data file under the partition storage path; filtering out data files whose field value range information does not overlap with the field value range conditions in the query conditions from the underlying stored data files to obtain a candidate file set.

[0089] The partition storage path refers to the physical storage location of the data table in the distributed file system, organized by partition key values ​​(e.g., / table / dt=20230101 / ). Field value range information refers to the value boundaries (including minimum, maximum, and null value statistics) of fields within each data file recorded in the Iceberg metadata.

[0090] The specific implementation process of this solution is as follows: First, the system parses the partition conditions to determine the corresponding physical storage path (i.e., the partition storage path); then, it reads the metadata of all data files under this path (including the Manifest list and Parquet file metadata) and extracts the statistical information of the field value range of the records; finally, it quickly compares the field value range predicate in the query conditions with the Min-Max values ​​of the files to exclude data files whose value ranges do not overlap with the query conditions at all (such as the Max value of the price field being less than the minimum price requirement in the query conditions).

[0091] This solution leverages Iceberg's multi-level metadata indexing capabilities to perform efficient pruning without reading actual data files, laying the foundation for subsequent row group-level filtering and addressing the issue of insufficient file-level filtering granularity in the background technologies.

[0092] Step 103: Based on the exact match condition and range query condition in the query conditions and the pre-established row group level index, perform row group level filtering on each data file in the candidate file set to obtain a set of data row groups that meet the conditions.

[0093] Exact match conditions refer to equal-valued expressions in the query conditions (such as user_id = 12345).

[0094] Range query conditions refer to the interval limit expression (such as 100≤price≤500).

[0095] Row group level indexes are fine-grained index structures built within the Parquet file for each row group.

[0096] The execution process of this step is as follows: The system first loads the pre-built Bloom filter (for exact matches) and range-encoded bitmap (for range queries) in the candidate files. Then, it scans the row group index information of each candidate file in parallel, quickly eliminating row groups that are unlikely to contain the target data. Next, it performs metadata verification on the potentially matching row groups, and finally filters out the set of data row groups that fully meet the query conditions. The specific filtering process will be explained through the following examples, and will not be detailed here.

[0097] Step 104: Read the data of the corresponding column from the data row group set according to the target field to obtain the query result set.

[0098] The execution process of this step is as follows: The system first parses the target field to generate a column projection scheme, and then, based on the columnar storage characteristics, reads the compressed data block of the corresponding column only from the data row group set obtained by row group filtering. The record is quickly reconstructed through parallel decoding and memory mapping technology, and finally assembled into a complete result set that meets the query requirements.

[0099] This step avoids the I / O overhead of reading redundant fields in traditional row-based storage schemes by optimizing columnar reading. Combined with the compression encoding methods unique to multimodal data (such as time-series fields with difference encoding and text fields with dictionary encoding), it improves the effective data throughput and significantly improves the data supply efficiency in deep learning training scenarios.

[0100] In another embodiment of this application, after obtaining the query result set, the following steps may be included: establishing a data supply channel: inputting the query result set into the data loading module of the deep learning framework according to training batches to form a pipeline transmission channel from the storage system to the computing device; configuring a multi-level caching system: analyzing the access characteristics of the data in the data row group set obtained by row group-level filtering, and establishing a multi-level caching system including high-speed cache and low-speed cache; performing data prefetching operation: combining the model training stage information and historical query records, predicting the data required for the next training batch and loading it into the caching system in advance.

[0101] Data supply channels refer to efficient data transmission links connecting storage systems and deep learning frameworks. Multi-level caching systems are hierarchical caching architectures that include different storage media (such as memory and SSDs). Access characteristics include data access frequency, temporal locality, and other pattern characteristics.

[0102] This embodiment adds a training and optimization phase after the query process, which specifically includes the following steps:

[0103] Establishing a data supply channel: This step constructs an efficient data transfer mechanism between the storage system and the deep learning framework. The system uses the deep learning framework's native data loading interface (such as PyTorch's DataLoader) to divide the query result set into batches according to the batch size required for training and establishes an asynchronous transmission channel. This channel employs double buffering and a zero-copy mechanism to achieve pipelined transfer from storage devices to GPU memory, ensuring that the training process will not be halted due to insufficient data supply.

[0104] Configure a multi-level caching system: Based on the data access characteristics (including access frequency, temporal locality, etc.) recorded during the row group-level filtering process, the system dynamically constructs a hierarchical storage system that includes high-speed memory cache and low-speed SSD cache. Among them, frequently accessed row group data (such as feature vectors) are preferentially cached in memory, while medium and low-frequency data are stored on SSD. The system also uses intelligent eviction algorithms such as LRU-K to automatically maintain cache content and improve the cache hit rate of hot data.

[0105] Perform data prefetching: The system monitors the model training progress (current epoch, batch, etc.) and analyzes historical query patterns to train a specialized prediction model to predict the data required for the next training batch. The prefetching module loads the predicted data from distributed storage to the local cache level in advance based on row group index and storage location information, so that the data readiness time completely overlaps with the training computation time, eliminating I / O wait latency.

[0106] This solution enables the storage system to proactively adapt to the data consumption rhythm of the training framework, reducing GPU waiting time in batch processing scenarios and forming an end-to-end optimization solution from the data lake to the training framework.

[0107] In another embodiment of this application, the following steps may be included before receiving a query request: for each field to be queried in the data table, determine the field characteristics of the field; when the field characteristics indicate that the field is a time-series field, compress and store the field value of the time-series field using difference encoding; when the field characteristics indicate that the field is a text field and the query frequency meets a preset condition, construct a dictionary mapping storage for the field value of the text field; when the field characteristics indicate that the field size exceeds a preset threshold, store the field value of the field in blocks and establish a block index.

[0108] Field characteristics refer to the set of metadata attributes for each field in a data table, including data type (such as time series or text), access mode, and storage characteristics.

[0109] This embodiment adds a data preprocessing stage before the query process: The system first analyzes the characteristic attributes of the field to be queried, and uses differential encoding to store time-series fields (such as timestamps, sensor readings and other data with continuous characteristics), recording only the incremental changes of adjacent data; for text fields that meet the preset query frequency conditions (such as frequently searched product category labels), a dictionary mapping is constructed to replace the original text with compact numerical identifiers; for field values ​​(such as image features) in a single record that exceed the size threshold (such as 1MB), they are stored in blocks of a fixed size (such as 4KB) and block index metadata is attached to support on-demand reading.

[0110] This preprocessing stage significantly improves the access efficiency of multimodal data in subsequent queries through differentiated storage optimization strategies, laying an efficient storage foundation for subsequent multi-level filtering mechanisms.

[0111] In another embodiment of this application, the following steps may be included before receiving a query request: statistically analyzing the frequency of occurrence of each field in the data table in historical query conditions, identifying field combinations with a frequency exceeding a set threshold as optimized field combinations; and sorting and storing the data file corresponding to the data table according to the space filling curve of the optimized field combinations.

[0112] Frequency of occurrence refers to the statistical frequency with which a specific field is referenced in the filtering conditions of historical query requests, reflecting the query popularity of the field. Optimized field combination refers to a set of frequently queried fields that, after analysis, need to be prioritized for optimization. Space-filling curves are multidimensional data sorting algorithms (such as Z-order curves) that can map multidimensional data to a one-dimensional space to maintain data locality.

[0113] This embodiment adds a data distribution optimization stage before the query process: the system analyzes historical query logs, counts the frequency of use of each field in the filtering conditions, and filters out high-frequency field combinations (such as user_id and timestamp); then, through the ORDER BY clause of the Spark engine, the data is physically sorted according to the Z-Order encoding values ​​of these fields and written to the Parquet file, so that data with similar query conditions are continuously distributed at the storage level.

[0114] This approach can improve the pruning efficiency of subsequent file-level filtering, reduce I / O access by enhancing data locality, and provide an efficient data physical layout foundation for multimodal queries.

[0115] In another embodiment of this application, the method may further include the following steps: monitoring the number of partition files corresponding to each partition in the data table, and triggering a file merging operation for the corresponding partition when the number of partition files in any partition exceeds a preset threshold.

[0116] The number of partition files refers to the number of physical files contained in a single partition in a data table. This metric directly affects metadata management overhead and query efficiency.

[0117] This embodiment tracks the changes in the number of files in each partition in real time through a background monitoring process. When the number of files in a specific partition exceeds a preset threshold (which can be dynamically adjusted based on storage engine performance and query load), a file merging operation is automatically triggered: the system selects small files with overlapping data ranges for merging and reorganization to generate larger files with fewer files and more compact data distribution, while maintaining consistent metadata versions before and after the merge.

[0118] This approach can solve the metadata bloat problem faced by file-level filtering, reduce the scheduling overhead of file-level filtering by reducing the number of files in a single partition, and improve the data locality of subsequent row group-level filtering, thereby improving the overall processing efficiency of multimodal queries.

[0119] The technical solution provided in this application firstly performs file-level coarse screening based on partition and field range information in the query conditions, significantly reducing the initial data scanning range; then, combining exact matching and range query conditions, it implements secondary fine screening using a pre-built row group-level hybrid index, greatly improving filtering accuracy; finally, it achieves efficient data extraction through columnar reading of the target field. This hierarchical and progressive query optimization strategy overcomes the shortcomings of insufficient granularity in traditional file-level filtering, solves the problem of poor adaptability of a single index through intelligent index combination, and perfectly adapts to the heterogeneous characteristics of multimodal data, forming a complete and efficient data supply chain from the storage layer to the computing layer, enabling the system to achieve significant performance improvements while maintaining versatility.

[0120] Figure 2 A flowchart illustrating an embodiment of another data query method provided in this application. Figure 2 The process shown is in Figure 1 Based on the illustrated process, the following steps are included:

[0121] Step 201: Parse the exact matching conditions and extract the first field to be matched and the target value.

[0122] Exact match conditions refer to the filtering expressions in a query statement that use equality operators (such as = and IN).

[0123] This step parses the query conditions using a syntax tree, extracting the field names requiring exact matching (e.g., user_id) and the set of target values ​​to be matched (e.g., [12345, 67890]). The system verifies whether the fields have a probabilistic filtering index and generates corresponding predicate pushdown conditions, providing input parameters for subsequent row group-level filtering. This parsing process supports handling compound queries containing multiple exact matching conditions and automatically optimizes the condition expressions to conjunctive normal form.

[0124] Step 202: For each candidate file, load the probabilistic filtering index of the first field in the candidate file, and determine whether each row group in the candidate file contains the target value through the probabilistic filtering index, and exclude the row groups that do not contain the target value.

[0125] A probabilistic filtering index is a row group-level index structure built for the first field (i.e., the target field in the exact matching condition).

[0126] The execution process of this step is as follows: The system loads the probabilistic filtering index corresponding to the first field of each candidate file in parallel. This index records the feature information of the data within the row group through a specific data structure. The extracted target value is input into the index for fast comparison, and a probabilistic judgment mechanism determines whether each row group may contain the target value. Row groups that are determined by the index to be unlikely to contain the target value are directly excluded, and only row groups that may match are retained for subsequent processing. This process makes full use of the fast exclusion feature of the index, significantly reducing the amount of data that needs to be precisely verified.

[0127] Step 203: Parse the range query conditions and extract the second field to be matched and the value range.

[0128] Range query conditions include a continuous range of values ​​defined by comparison operators (such as >, BETWEEN).

[0129] This step uses a range tree data structure to parse the query conditions, extracting the field name (i.e., the second field, such as price) and its valid value range (i.e., the value range, such as [100, 500]). The system will check whether the range boundaries can be closed and optimized (e.g., converting price>100 to price≥101), and automatically perform range merging for queries involving multiple range conditions (e.g., [100, 300]∪[400, 500]), finally generating a normalized range descriptor for index query use.

[0130] Step 204: For each candidate file, load the segmented value range index of the second field in the candidate file, and determine whether each row group in the candidate file contains data within the value range through the segmented value range index, and exclude row groups that do not contain data within the value range.

[0131] A segmented range index is a row group-level range index created for the second field (i.e., the target field in the range query condition).

[0132] The execution process of this step is as follows: The system loads the segmented value range index of the second field in each candidate file. This index divides the field value range into several intervals and records the data existence status of each interval. By comparing the value range in the query conditions with the interval distribution of the index records, the system quickly locates the intervals where matching data may exist. Based on the status information of the index markers, the system excludes and determines the row groups that do not contain matching data, effectively narrowing the precise scan range. This index structure is particularly suitable for handling range queries of high cardinality segments, and can complete efficient filtering without reading the actual data.

[0133] In one embodiment, a probabilistic filtering index can be constructed through the following steps: statistically analyzing the value distribution of the first field in each row group to obtain row group data features; setting the bit array length according to the number of values ​​in the row group data features; determining the number of hash functions based on a preset false positive rate requirement; and constructing a probabilistic filtering index based on the bit array length and the number of hash functions.

[0134] Row group data characteristics refer to the numerical distribution characteristics of the first field within a single row group, including statistical information such as the number of unique values ​​and the range of values.

[0135] The index construction process in this embodiment includes four key stages: First, the distribution pattern of the first field in each row group is analyzed to obtain statistical characteristics including the number of unique values; then, the required bit array storage space is calculated based on the scale of the number of unique values; next, the optimal number of hash functions is derived based on the maximum allowable false positive rate preset by the system; finally, the probabilistic filtering index structure of the row group is instantiated by combining the determined bit array length and the number of hash functions.

[0136] Furthermore, the construction of the probabilistic filtering index is implemented for the high-radix number segment in the first field; the bit array generated for each row group is stored independently; the preset false positive rate requirement is achieved by: calculating the minimum length of the bit array based on the number of unique values ​​of the first field in the row group; and determining the optimal combination of hash functions according to the logarithmic relationship.

[0137] This solution optimizes the probabilistic filtering index for high-cardinality numeric fields (such as user IDs, product numbers, and other fields containing a large number of unique values) that require precise matching in the query conditions. During the construction process, the system independently creates a bit array structure for each row group. This independent storage design ensures the parallel processing capability of the filtering operation. The index parameters are determined through precise mathematical calculations: first, the number of unique values ​​contained in the field within the row group is counted, and the minimum length required for the bit array is calculated based on this value; then, the optimal combination of hash functions is derived based on the logarithmic function relationship. This parameterized construction method can automatically adapt to data characteristics of different scales, achieving optimal storage space utilization while strictly ensuring the preset false positive rate requirements.

[0138] This dynamic parameterization construction method can effectively control the index misjudgment rate and adaptively adjust the storage overhead according to the data distribution characteristics of different fields, so that the index size of high cardinality segments is always kept within a reasonable proportion of the original data volume of the row group, achieving the optimal balance between query performance and storage efficiency.

[0139] In one embodiment, a segmented value domain index can be constructed through the following steps: analyzing the value distribution of the second field in all row groups to obtain the field value distribution characteristics; if the field value distribution characteristics indicate that the field is a continuous field, determining the interval division granularity to obtain an interval division scheme, performing equal-frequency binning on the field values ​​according to the interval division scheme, constructing a continuous value domain index, and using the continuous value domain index as the segmented value domain index; if the field value distribution characteristics indicate that the field is a discrete field, statistically analyzing the frequency of discrete values ​​to obtain an enumerated value distribution, establishing a discrete value mapping relationship based on the enumerated value distribution, constructing an enumerated mapping index, and using the enumerated mapping index as the segmented value domain index.

[0140] The field value distribution characteristics refer to the overall numerical distribution pattern of the second field in all row groups, including statistical characteristics such as numerical continuity and frequency of discrete values.

[0141] The index construction process in this embodiment adopts a differentiated strategy: For continuous fields (such as price, temperature and other numerical data), firstly, the distribution characteristics of their value range are analyzed to determine the optimal interval division granularity, and the value range is divided into several uniformly distributed intervals using the equal frequency binning algorithm to construct a continuous value range index based on the interval boundaries; for discrete fields (such as category, region and other enumerated data), the frequency of occurrence of each discrete value is counted, and after sorting by frequency, an efficient search structure based on the mapping of enumerated values ​​is constructed.

[0142] Furthermore, the intervals generated by the equal-frequency binning process are left-closed and right-open intervals; the segmented value domain index maintained by each row group includes: an ordered set of interval boundary values, and a bitmap marking the data existence status of each interval within the row group.

[0143] In this scheme, the segmented value range index uses an equal-frequency binning algorithm to divide field values ​​into intervals. The generated intervals strictly adhere to the left-closed, right-open mathematical representation, ensuring clear interval boundaries and accurate query processing. Each row group maintains an index containing two key data structures: first, an ordered set that fully stores all interval boundary values ​​for quick location of the query range; and second, a compact bitmap structure that records whether valid data exists within each interval. This standardized construction method ensures accurate determination of the matching relationship between numerical intervals and query conditions during range queries, while also controlling the index's space overhead within a reasonable range through the efficient compression storage characteristics of bitmaps, achieving a balance between query performance and storage efficiency.

[0144] This dynamic and adaptive index building method can intelligently select the optimal index form based on the field data type, which not only ensures the efficiency of range queries for continuous values, but also optimizes the efficient matching of discrete values, significantly improving the accuracy of row group-level filtering, while keeping the index storage overhead at a reasonable level.

[0145] Figure 2 The process shown achieves efficient row-group-level precise filtering through a multi-level index collaborative filtering mechanism: First, it parses and extracts the target fields and values ​​of exact matching conditions and range query conditions. Then, it uses a probabilistic filtering index to quickly exclude row groups that do not contain the target values. Finally, it uses a segmented value range index to precisely locate row groups that meet the range requirements. This phased, progressive filtering strategy leverages the advantages of efficient exclusion by probabilistic indexes while combining the precise location characteristics of value range indexes. It significantly reduces the amount of data requiring precise scanning while maintaining query accuracy, effectively solving the performance bottleneck problem of high-cardinality, multimodal data queries.

[0146] Figure 3 This is a data query system architecture diagram provided for an embodiment of this application. For example... Figure 3 As shown, the system includes:

[0147] Storage optimization engine: responsible for compact coding and intelligent sorting during data writing.

[0148] Multi-level index manager: Builds hybrid indexes at the file and row group levels and integrates with Iceberg metadata.

[0149] Dynamic pruning executor: Parses query conditions and combines indexes and metadata to achieve multi-level filtering.

[0150] Training framework adaptation layer: Deeply integrated with PyTorch / TensorFlow's DataLoader to achieve pipelined loading.

[0151] Strategy recommendation system: a dynamic optimization module based on historical query feedback.

[0152] The system achieves efficient data reading through four core modules: storage structure optimization, multi-level indexing and dynamic pruning, deep integration of training framework, and automated strategy recommendation.

[0153] Figure 4 This is a block diagram illustrating an embodiment of a data query device provided in this application. Figure 4 As shown, the device includes:

[0154] The receiving module 41 is used to receive a query request, wherein the query request carries query conditions and the target field to be queried;

[0155] The first filtering module 42 is used to scan the metadata of the data table based on the partition conditions and field value range conditions in the query conditions, perform file-level filtering on the underlying stored data files, and obtain a candidate file set.

[0156] The second filtering module 43 is used to perform row group-level filtering on each data file in the candidate file set based on the exact matching conditions and range query conditions in the query conditions and the pre-established row group-level index, so as to obtain a set of data row groups that meet the conditions.

[0157] The reading module 44 is used to read data of the corresponding column from the data row group set according to the target field to obtain a query result set.

[0158] In one possible implementation, the first filtering module is specifically used for:

[0159] Determine the corresponding partition storage path based on the partitioning conditions;

[0160] Obtain the field value range information of each data file under the partition storage path;

[0161] The candidate file set is obtained by filtering out data files whose field value range information does not overlap with the field value range conditions in the query conditions from the underlying data files.

[0162] In one possible implementation, the second filtering module is specifically used for:

[0163] Parse the exact matching conditions and extract the first field to be matched and the target value;

[0164] For each candidate file, load the probabilistic filtering index of the first field in the candidate file, and determine whether each row group in the candidate file contains the target value through the probabilistic filtering index, and exclude the row group that does not contain the target value;

[0165] Parse the range query conditions and extract the second field to be matched and its value range;

[0166] For each candidate file, the segmented value range index of the second field in the candidate file is loaded, and the segmented value range index is used to determine whether each row group in the candidate file contains data within the value range, and row groups that do not contain data within the value range are excluded.

[0167] In one possible implementation, the second filtering module is further configured to:

[0168] The distribution of the first field's values ​​in each row group is statistically analyzed to obtain the characteristics of the row group data.

[0169] Set the bit array length according to the number of values ​​in the row group data characteristics;

[0170] The number of hash functions is determined based on a preset false positive rate requirement;

[0171] Construct a probabilistic filtering index based on the length of the bit array and the number of hash functions.

[0172] In one possible implementation, the second filtering module is further configured to:

[0173] Analyze the value distribution of the second field across all row groups to obtain the field value distribution characteristics;

[0174] When the field value distribution characteristics indicate that the field is a continuous field, the interval division granularity is determined to obtain the interval division scheme. The field values ​​are then binned according to the interval division scheme to construct a continuous value domain index. The continuous value domain index is then used as the segmented value domain index.

[0175] When the field value distribution characteristics indicate that the field is a discrete field, the frequency of discrete values ​​is counted to obtain the enumerated value distribution. Based on the enumerated value distribution, a discrete value mapping relationship is established, an enumerated mapping index is constructed, and the enumerated mapping index is used as the segmented value domain index.

[0176] In one possible implementation, the device further includes a storage optimization module for:

[0177] For each field to be queried in the data table, determine the field characteristics of that field;

[0178] When the field feature indicates that it is a time-series field, the field value of the time-series field is compressed and stored using difference encoding;

[0179] When the field feature indicates that it is a text field and the query frequency meets the preset conditions, a dictionary mapping is constructed and stored for the field value of the text field;

[0180] When the field characteristics indicate that the field size exceeds a preset threshold, the field value is stored in blocks and a block index is created.

[0181] In one possible implementation, the storage optimization module is further configured to:

[0182] Statistically analyze the frequency of each field in the data table in historical query conditions, and determine the field combinations with a frequency exceeding a set threshold as optimized field combinations;

[0183] The data files corresponding to the data table are sorted and stored according to the space filling curve of the optimized field combination.

[0184] like Figure 5As shown in the figure, this application provides a device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0185] Memory 113 is used to store computer programs;

[0186] In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the data query method provided in any of the foregoing method embodiments, including:

[0187] Receive a query request, wherein the query request carries query conditions and the target field to be queried;

[0188] Based on the partitioning conditions and field value range conditions in the query conditions, the data table metadata is scanned, and file-level filtering is performed on the underlying stored data files to obtain a candidate file set.

[0189] Based on the exact match condition and range query condition in the query conditions, as well as the pre-established row group level index, each data file in the candidate file set is filtered at the row group level to obtain a set of data row groups that meet the conditions.

[0190] The data of the corresponding column is read from the data row group set according to the target field to obtain the query result set.

[0191] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data query method provided in any of the foregoing method embodiments.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0194] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0195] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data query method, characterized in that, The method includes: Receive a query request, wherein the query request carries query conditions and the target field to be queried; Based on the partitioning conditions and field value range conditions in the query conditions, the data table metadata is scanned, and file-level filtering is performed on the underlying stored data files to obtain a candidate file set. Based on the exact match condition and range query condition in the query conditions, as well as the pre-established row group level index, each data file in the candidate file set is filtered at the row group level to obtain a set of data row groups that meet the conditions. The data of the corresponding column is read from the data row group set according to the target field to obtain the query result set.

2. The method according to claim 1, characterized in that, The step of scanning the data table metadata based on the partitioning conditions and field value range conditions in the query conditions, and performing file-level filtering on the underlying stored data files, includes: Determine the corresponding partition storage path based on the partitioning conditions; Obtain the field value range information of each data file under the partition storage path; The candidate file set is obtained by filtering out data files whose field value range information does not overlap with the field value range conditions in the query conditions from the underlying data files.

3. The method according to claim 1, characterized in that, The process of performing row-group-level filtering on each data file in the candidate file set based on the exact match conditions and range query conditions in the query conditions, as well as the pre-established row group-level index, includes: Parse the exact matching conditions and extract the first field to be matched and the target value; For each candidate file, load the probabilistic filtering index of the first field in the candidate file, and determine whether each row group in the candidate file contains the target value through the probabilistic filtering index, and exclude the row group that does not contain the target value; Parse the range query conditions and extract the second field to be matched and its value range; For each candidate file, the segmented value range index of the second field in the candidate file is loaded, and the segmented value range index is used to determine whether each row group in the candidate file contains data within the value range, and row groups that do not contain data within the value range are excluded.

4. The method according to claim 3, characterized in that, The method further includes: The distribution of the first field's values ​​in each row group is statistically analyzed to obtain the characteristics of the row group data. Set the bit array length according to the number of values ​​in the row group data characteristics; The number of hash functions is determined based on a preset false positive rate requirement; Construct a probabilistic filtering index based on the length of the bit array and the number of hash functions.

5. The method according to claim 3, characterized in that, The method further includes: Analyze the value distribution of the second field across all row groups to obtain the field value distribution characteristics; When the field value distribution characteristics indicate that the field is a continuous field, the interval division granularity is determined to obtain the interval division scheme. The field values ​​are then binned according to the interval division scheme to construct a continuous value domain index. The continuous value domain index is then used as the segmented value domain index. When the field value distribution characteristics indicate that the field is a discrete field, the frequency of discrete values ​​is counted to obtain the enumerated value distribution. Based on the enumerated value distribution, a discrete value mapping relationship is established, an enumerated mapping index is constructed, and the enumerated mapping index is used as the segmented value domain index.

6. The method according to claim 1, characterized in that, The process of receiving a query request includes: For each field to be queried in the data table, determine the field characteristics of that field; When the field feature indicates that it is a time-series field, the field value of the time-series field is compressed and stored using difference encoding; When the field feature indicates that it is a text field and the query frequency meets the preset conditions, a dictionary mapping is constructed and stored for the field value of the text field; When the field characteristics indicate that the field size exceeds a preset threshold, the field value is stored in blocks and a block index is created.

7. The method according to claim 1, characterized in that, The process of receiving a query request includes: Statistically analyze the frequency of each field in the data table in historical query conditions, and determine the field combinations with a frequency exceeding a set threshold as optimized field combinations; The data files corresponding to the data table are sorted and stored according to the space filling curve of the optimized field combination.

8. A data query device, characterized in that, The device includes: A receiving module is used to receive query requests, wherein the query request carries query conditions and the target field to be queried; The first filtering module is used to scan the metadata of the data table based on the partitioning conditions and field value range conditions in the query conditions, perform file-level filtering on the underlying stored data files, and obtain a candidate file set. The second filtering module is used to perform row group-level filtering on each data file in the candidate file set based on the exact matching conditions and range query conditions in the query conditions and the pre-established row group-level index, so as to obtain a set of data row groups that meet the conditions. The reading module is used to read data of the corresponding column from the data row group set according to the target field to obtain a query result set.

9. A device, characterized in that, include: A processor and a memory, the processor being configured to execute a data query program stored in the memory to implement the data query method according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the data query method according to any one of claims 1-7.

Images