Data query method, device, storage medium and program

Abstract

Embodiments of the present application provide a data query method, a device, a storage medium and a program. The method comprises: in response to a query request for a target database, determining a target column to be queried and a data screening condition; on the basis of index information of a plurality of row groups comprised in a columnar storage file corresponding to the target database, selecting, from among the plurality of row groups, at least one row group matching the target column and the data screening condition, wherein different row groups correspond to different rows of a plurality of columns in the target database; and on the basis of address information of the at least one row group, sequentially querying multiple levels of cache spaces according to a set query order of the multiple levels of cache spaces, and if data corresponding to a target row group among the at least one row group is obtained from a target cache space, filtering, from among the data corresponding to the target row group, for target data satisfying the data screening condition. By means of multiple levels of caches, the data query efficiency in a columnar storage query scenario is improved.

Description

[0001] This disclosure relates to the field of database technology, and more particularly to a data query method, device, storage medium, and program. Background Art: Columnar storage query refers to a query operation performed in a columnar storage database. Unlike row-oriented storage, columnar storage stores data from the same column together, making it particularly suitable for aggregation and statistical calculations in many big data analysis scenarios. A columnar database often contains many data tables, and these data tables can be stored using a certain columnar storage format file, such as an ORC (Optimized Row Columnar) file. A data table typically corresponds to at least one ORC file. Each ORC file can be set to a preset size. These ORC files are often centrally stored at the user's remote location (e.g., cloud storage services). When performing data queries, a significant amount of time is spent querying the ORC files, resulting in low data query efficiency. Summary of the Invention: Embodiments of this disclosure provide a data query method, device, storage medium, and program to improve data query efficiency. In a first aspect, embodiments of this disclosure provide a data query method, the method comprising: responding to a query request for a target database, determining a target column and data filtering conditions to be queried; filtering at least one row group matching the target column and the data filtering conditions from the multiple row groups according to index information of multiple row groups contained in a columnar storage file corresponding to the target database, wherein different row groups correspond to different rows of multiple columns in the target database; querying the multi-level cache space sequentially according to a set query order based on the address information of the at least one row group; if data corresponding to the target row group in the at least one row group is obtained from the target cache space, then filtering out target data that satisfies the data filtering conditions from the data corresponding to the target row group, wherein the multi-level cache space has different data storage formats and different addressing modes, the multi-level cache space includes different cache spaces located in memory, the target row group is any one of the at least one row group, and the target cache space is any one of the multi-level cache spaces.Secondly, embodiments of this disclosure provide a data query apparatus, the apparatus comprising: a determining module, configured to determine a target column and data filtering conditions to be queried in response to a query request for a target database; a filtering module, configured to filter at least one row group matching the target column and the data filtering conditions from the multiple row groups based on index information of multiple row groups contained in a columnar storage file corresponding to the target database, wherein different row groups correspond to different rows of multiple columns in the target database; and a query module, configured to query the multi-level cache space sequentially according to a set query order based on the address information of the at least one row group, and if data corresponding to the target row group in the at least one row group is obtained from the target cache space, then target data satisfying the data filtering conditions is filtered out from the data corresponding to the target row group, wherein the multi-level cache space has different data storage formats and different addressing modes, the multi-level cache space includes different cache spaces located in memory, the target row group is any one of the at least one row group, and the target cache space is any one of the multi-level cache spaces. Thirdly, embodiments of this disclosure provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data query method as described in the first aspect. Fourthly, embodiments of this disclosure provide a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor can at least implement the data query method as described in the first aspect. Fifthly, embodiments of this disclosure provide a computer program product, wherein the computer program product includes a computer program, and when executed by the processor of the electronic device, the processor can at least implement the data query method as described in the first aspect. The data query method provided by embodiments of this disclosure is applicable to data querying in columnar databases, and multi-level caching can be set in a cloud server used to perform data query tasks, including different cache spaces located in memory. The multi-level cache spaces have different data storage formats and different addressing modes, and the data query efficiency in columnar database query scenarios is improved by setting up multi-level caching.Specifically, upon receiving a query request for the target database and determining the target columns and data filtering conditions, the system uses the index information of multiple row groups contained in the column-oriented storage file (e.g., an ORC file) corresponding to the target database to filter out at least one row group that matches the target columns and data filtering conditions. Then, based on the address information of this at least one row group, the system sequentially queries the multi-level cache spaces according to the set query order. For the target row group within this at least one row group, if the corresponding data is obtained from the target cache space, the system directly filters out the target data that meets the data filtering conditions from the obtained data corresponding to the target row group. By operating on a row group basis and following the query order of the multi-level cache, if the data corresponding to a row group can be obtained from a certain cache space, there is no need to query the remote target database's storage space, thereby improving data query efficiency in column-oriented storage scenarios. Brief Description of the Drawings To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 is a schematic diagram of the composition of an ORC file provided in an embodiment of this disclosure; Figure 2 is a schematic diagram of the ORC file usage process provided in an embodiment of this disclosure; Figure 3 is a flowchart of a data query method provided in an embodiment of this disclosure; Figure 4 is an application schematic diagram of a data query process provided in an embodiment of this disclosure; Figure 5 is a flowchart of a data query method provided in an embodiment of this disclosure; Figure 6 is a schematic diagram of a process for downloading file fragments from cloud storage services provided in an embodiment of this disclosure; Figure 7 is a structural schematic diagram of a data query device provided in an embodiment of this disclosure; Figure 8 is a structural schematic diagram of an electronic device provided in an embodiment of this disclosure. Detailed Description of the Embodiments To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions in the embodiments of this disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some embodiments of this disclosure, but not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this disclosure.It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. The following describes some embodiments of this disclosure in detail with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features in the embodiments can be combined with each other. In addition, the timing of the steps in the following method embodiments is only an example and not a strict limitation. First, the terms involved in the embodiments of this disclosure are explained. Columnar query: refers to a query operation performed in a columnar database. Unlike row-oriented databases, column-oriented databases store data from the same column together, thereby optimizing the performance of certain types of queries, especially in big data analytics scenarios such as Online Analytical Processing (OLAP) and Hybrid Transactional Analytical Processing (HTAP) for aggregation and statistical computation. Column-oriented queries can efficiently read the data in the required columns, reduce I / O operations, provide faster data retrieval speeds and higher compression ratios, and are particularly suitable for analyzing and processing large datasets.

[0002] ORC (Optimized Row Columnar): A self-describing, type-aware columnar storage format used for storing data in columnar databases. It organizes data by columns and applies efficient compression and encoding techniques to each column, supporting complex nested data structures. ORC optimizes data read performance, reduces storage space requirements, and provides efficient data lookup and serialization mechanisms, making it widely used in big data ecosystems such as Hadoop and Hive. Scan Operator: In the database field, the Scan operator refers to the operation of performing a full table scan on a database table or index, often accompanied by certain filtering conditions (also known as predicate conditions). It is a linear traversal method, reading each row or column of data in the table one by one. Object Storage Service (OSS): A cloud storage service used to store and access any type of data. It provides stable, secure, efficient, and easy-to-use data storage capabilities. Users can upload and download data anytime, anywhere, in any application, and on any device via the network, and manage data accordingly. OSS is widely used in website content, mobile applications, big data analytics, and other scenarios, and is particularly suitable for handling large-scale data storage needs. In OLAP, HTAP, and other big data analytics scenarios, there is often a need to query specific types of data (such as product sales). Columnar databases are well-suited to meet these needs, so they are used in many big data storage scenarios. Columnar databases store data from the same column together, allowing for lower-cost queries on specific columns to perform aggregation and statistical calculations (such as averages). A columnar database is a database management system that stores and manages data in units of columns. A columnar database can contain multiple tables, each with one or more ORC files (column storage files) to record data, related index information, metadata, etc. In practical applications, in big data analytics scenarios, due to the massive amount of data that needs to be stored, the ORC files (including the ORC files of each table) corresponding to the user's columnar database are typically stored and maintained using cloud storage services provided by cloud vendors, such as OSS. Thus, when a user needs to query data in certain columns of a table that meets a specific filter condition for statistical calculations, a request must be made from the remote cloud storage service, resulting in high latency and low efficiency.However, if a columnar database storing massive amounts of data is deployed on the host where the user's query processing "executor" resides, such as on the host's disk, the storage capacity requirement for the disk will be enormous, resulting in high storage costs. Especially during data queries, the significant performance overhead can impact the normal execution of other local tasks or applications. This host could be a user-rented cloud server, virtual machine, etc. Therefore, considering the user's storage costs and the access speed of different storage media, this disclosure introduces a multi-level caching mechanism to improve the efficiency of columnar queries on large-scale datasets. This addresses the slow query response speed in large-scale columnar databases, such as those in ORC format, achieving a balance between query efficiency and storage costs. In summary, the solution provided in this disclosure employs a tiered architecture of different levels of caching and remote storage (such as OSS storage) for different data frequencies. Through the principle of locality, frequently accessed data is stored in high-speed storage media (such as memory), optimizing the query path. Simultaneously, overflow management of the cache space is implemented to improve the resource utilization efficiency and performance of the entire caching system. The data query method provided in this disclosure can be executed by a data query service. This data query service can run on a server or server cluster in the cloud, or it can be located on a host on the user's local machine. The user can use this data query service to query data in their columnar database. To facilitate understanding of the data query method provided in this disclosure, the structure of an ORC file is first briefly introduced. It should be noted that this disclosure only uses ORC as an example to introduce one storage format of columnar databases; other columnar database storage formats are also applicable. Figure 1 is a schematic diagram of the composition of an ORC file provided in this disclosure. As shown in Figure 1, an ORC file consists of three main parts: multiple stripes, a file footer, and a postscript. For ease of description, assume that a columnar database contains only one data table, which corresponds to only one ORC file, and assume that the data table is an N-row, M-column table. Then, a stripe can store N1 rows and M columns of data, where N1 is less than N. In practice, the size of the stripes can be determined by a set amount of data, such as 64MB. Therefore, a stripe can be used to store data from a portion of all rows in all columns.As shown in Figure 1, each stripe consists of three parts: index data, row data, and stripe footer. Within a stripe, each column is stored in a contiguous file area—a row data area. That is, the N1 rows and M columns of data corresponding to a stripe (e.g., column 1 - column M in Figure 1) are stored in the row data of that stripe. Specifically, the data is stored column-wise; the first N1 rows of data are written first, followed by the second N1 rows of data. Furthermore, within a stripe, data is grouped and stored in rows. A row group (or row group) refers to a single column of data stored within a stripe (actually storing N1 rows of that column). For example, a row group can be divided into groups of 10,000 rows. It's understood that if the total number of rows N1 in the column is not a multiple of 10,000, then the last row group will have fewer than 10,000 rows. Row groups are the basic unit for compression, decompression, encoding, and decoding in ORC files. Furthermore, a row group can be further divided into blocks; for example, 1,000 rows in a row group can form a block. A block is the basic unit for performing computational operations on the database. The 10,000-row and 1,000-row examples mentioned above are merely illustrations. In practice, when storing a column of data within a stripe in the row data section, the data content itself can be considered as constituting a data stream, while information such as data structures and data types, in addition to the data content itself, constitutes the corresponding metadata stream. Therefore, as shown in Figure 1, the row data can contain both of these data streams. In fact, besides the file areas used for data storage—the row data areas in each stripe—other file areas in an ORC file are primarily used to store the ORC file's metadata and index information. For example, the information stored in the file footer, postscript, and stripe footer can all be considered metadata information, and the index information can include the index information located in the index data section of each stripe.The postscript, located at the very end of the ORC file, is used to locate the byte positions corresponding to the main areas within the file. It can include the entire ORC file size, the length and starting byte position of the file footer, stripe sizes, and version information. The file footer stores file-level statistics, such as maximum and minimum values ​​for columns, and the starting byte positions of each stripe (as shown by the dashed arrows in Figure 1). Stripes can also store compression, encoding, and encryption algorithm information. The stripe footer stores the starting position and length of index data and row data within the stripe, as well as metadata for column and row group structures, and stripe-level statistical information. The index information in the stripe (index data): Stores the index information of each column, including statistical information such as the maximum value (max), minimum value (min), sum, and count for each column. It can also store information such as the starting byte position of each row group corresponding to each column of data, and the byte size before and after compression. For example, the index information of columns M (column 1 - column M) shown in Figure 1 includes the index information of each row group (e.g., the index information of K row groups in Figure 1: row index 1 - row index K), such as the starting byte position. The ORC file format described above is only one example. Other similar ORC file formats or other columnar storage formats have similar columnar storage principles and are equally applicable to the data query scheme provided in this disclosure embodiment. Based on the ORC file format shown in Figure 1, and combined with Figure 2, a brief introduction to the general usage process of this ORC file during data query is given. In general, it is parsed and used from the bottom up. The data query service can include two main functional modules: optimizer and executor. Upon receiving a query request from a user containing the column names and data filtering conditions of a specific data table, the optimizer generates a corresponding execution plan. The execution plan includes the aforementioned data filtering conditions, the target columns to be queried, the table identifier, and the query steps to be performed, as well as the operators used in these steps, such as scan and join operators. Additionally, the execution plan specifies the ORC file corresponding to the data table that needs to be accessed.The `scan` operator describes where to query data and what the filtering conditions are. Then, the execution plan is provided to the executor, which performs the query operation based on this plan, obtains the final target data, and returns it to the user. During the executor's execution of the query based on the query plan, it needs to access the index information (index data) of the target column in the ORC file. Based on this index information (index data), it filters out row groups that meet the data filtering conditions from the row groups corresponding to the target column, assuming at least one row group meets the data filtering conditions. Then, it reads the data corresponding to each of these at least one row group, that is, each row of data in each data block, to filter out the target data that meets the data filtering conditions. In short, it first narrows the search scope to the row group level, and then filters out the target data from the row groups, that is, the rows of data in the target column that meet the data filtering conditions. The process of obtaining the index information (index data) of the target column can be as follows: Since the length of the ORC file is known, first read the postscript area at the end of the ORC file, parse out the starting byte position and length of the file footer, and then read the starting byte position of each stripe stored in the file footer. This allows the index information (index data) of each stripe to be read. oSince each strip's index data contains the index information for each column, and each column's index information contains the index information for its corresponding row groups, the index information for each row group of the target column can be read from it. Then, based on the index information of the row groups corresponding to the target column, row groups that meet the data filtering criteria can be selected from these row groups. For example, suppose the data filtering criteria are to filter data in the target column that falls within the value range [x1, x2]. If the index information of a row group corresponding to the target column in a certain strip is: the minimum value of the data in this row group is x3, the maximum value is x4, and x3 is greater than x1, and x4 is less than x2, then it means that the data in this row group all fall within this value range, and therefore this row group is considered to meet the data filtering criteria. Conversely, if x3 is less than x1, and x4 is between x1 and x2, it means that the data in this row group intersects with this value range, and in this case, the row group is also considered to meet the data filtering criteria. Therefore, if the index information of a row group corresponding to the target column determines that the data in that row group "intersects" with the data filtering criteria—that is, at least some data in that row group satisfies the data filtering criteria—then that row group can be considered a row group that satisfies the data filtering criteria. Then, the various data blocks contained in that row group can be read from the row data area of ​​the corresponding strip. Each row in each data block is then checked against the data filtering criteria to determine if it satisfies them. Data that meets the criteria is then selected and used as the target data, which is output to the user. Alternatively, if the user instructs the user to perform some operation on this target data (such as displaying it as a list, a bar chart, or performing aggregation calculations), the calculation results are returned to the user. In practice, the reading of the aforementioned data blocks is based on the logical address corresponding to the already formed row group. This logical address is: (stripeid, columnid, row-groupid}, where stripeid represents the stripe number, columnid represents the column number within the stripe, and row-groupid represents the row group number within the column, i.e., a specific row group corresponding to a specific column in a stripe. In fact, based on this logical address, the known data block size, and the number of data blocks contained in a row group, multiple data blocks within a row group can be read.For a single data block, its location address can further include `startPosition` and `positionCount`. Assuming a row group has a maximum of 10,000 rows and a data block has 1,000 rows, a row group can store multiple data blocks. Therefore, within a row group, the specific location of a data block can be identified based on the row group's starting byte position (`startPosition`) and the row number (`positionCount`). Based on the above, during data querying according to data filtering conditions and the index information corresponding to each row group of the target column, it is possible to efficiently determine whether a specific row group contains data that meets the data filtering conditions without reading all the data, reducing the workload of data loading and scanning. As described above, while columnar queries offer higher efficiency than searching for data in a row-based database that meets specific filtering criteria, the massive amounts of data in the ORC files often reside in lower-cost cloud storage services (such as OSS). The cloud storage service is hosted on a cloud server, while the device executing the query might be a user's device or a remote service cluster in the cloud, resulting in significant query latency and requiring further optimization. The data query method provided in this embodiment can further improve columnar query efficiency. Figure 3 is a flowchart of a data query method provided in this embodiment. As shown in Figure 3, the method may include the following steps:

[0003] 301. In response to a query request for the target database, determine the target columns and data filtering conditions to be queried.

[0004] 302. Based on the index information of multiple row groups contained in the columnar storage file corresponding to the target database, at least one row group that matches the target column and data filtering conditions is selected from the multiple row groups, wherein different row groups correspond to different rows of multiple columns in the target database.

[0005] 303. Based on the address information of at least one row group, the multi-level cache spaces are queried sequentially according to the set query order. If data corresponding to the target row group in at least one row group is obtained from the target cache space, the target data that meets the data filtering conditions is filtered out from the data corresponding to the target row group. The multi-level cache spaces have different data storage formats and different addressing methods, and include different cache spaces located in memory. In this embodiment, the target database is a columnar database. When a user triggers a query request for the target database, it may be to query target data in a target column of a certain data table that meets a certain data filtering condition. Therefore, in practical applications, the query request may be in the following form: `select * from table 1 where a < 0 and a > 1`. Here, 'a' represents the target column, the 'WHERE' clause represents the data filtering conditions, and the queried data table is table 1 » As mentioned above, after receiving the above query request, as shown in Figure 4, the optimizer can generate a corresponding execution plan. The execution plan includes the above data filtering conditions, the target column, the ORC file that needs to be accessed corresponding to the data table, and some operators that need to be executed during the query process, such as scan operators and join operators. Then, the executor completes the query operation based on the execution plan, obtains the final target data, and feeds it back to the user. As mentioned above, during the query operation based on the execution plan, the executor can first filter out at least one row group that matches the target column and data filtering conditions from multiple row groups based on the index information of multiple row groups corresponding to each column in each strip. Then, it loads the data corresponding to each of the at least one row group and determines the target data that meets the data filtering conditions. Here, as mentioned above, matching means that the index information of a row group corresponding to the target column has an intersection range with the data filtering suite. To accelerate the loading speed of data corresponding to at least one row group and thus improve the filtering speed of the target data, this embodiment of the disclosure designs a multi-level cache, which may include cache space located in the memory of the host (such as a cloud server) where the data query service is located and cache space located on the local disk. In the multi-level cache space, the cache space closer to the executor has a faster access speed; that is, the access speed of the cache space in memory will be higher than the access speed of the cache space on the disk. Therefore, the access order of the multi-level cache is as follows: first, access the cache space in memory; if the required data is not found in the cache space in memory, then access the cache space on the disk. If the required data is not found in any of the cache spaces, then access the remote storage space, such as OSS.Therefore, for any target row group in at least one of the above row groups, based on its address information, the multi-level cache spaces are queried sequentially according to the set query order. If data corresponding to the target row group is obtained from the target cache space, the target data that meets the data filtering conditions is filtered out from the data corresponding to that target row group, and other row groups are processed in the same way. If the data corresponding to the target row group is not obtained from the multi-level cache space, the file fragment (or file area) corresponding to the target row group is downloaded from the cloud storage service storing the complete ORC file according to the address information of the target row group, and the target data that meets the data filtering conditions is filtered out from the file fragment. At this time, according to the data storage requirements of the multi-level cache space, the file fragment corresponding to the target row group obtained from the cloud storage service is also stored in the multi-level cache space. In fact, because the storage media of the above different cache spaces are different, the access speed is different. If the data corresponding to the target row group can be found in the cache space set in memory closer to the executor, then there is no need to query other cache spaces, let alone access the remote cloud storage service, and the data query efficiency will be significantly improved. Furthermore, in this embodiment, even within the same storage medium, more than one cache space can be provided. For example, two cache spaces can be set up in memory, but the format and addressing method of the data stored in these two cache spaces can differ. In summary, in this embodiment, taking a target row group as an example, the storage format of the data corresponding to the target row group in the multi-level cache spaces can be different, thus the addressing method will also be different. The difference in access speed of the storage media where different cache spaces are located, combined with the design of the data storage format in different cache spaces, can help further improve columnar storage query efficiency. In this embodiment, including multi-level cache spaces and cloud storage services, there are two ways to find the data of the target row group: one is to use the logical address of the target row group for addressing, and the other is to use the physical address of the target row group for addressing. The addressing methods used by different cache spaces and cloud storage services can be pre-set.Based on this, the address information of the target row group can include the logical address mentioned above, that is, the logical address (stripeid, columnid, row-groupid) corresponding to the target row group obtained based on the stripe, target column, and identifier of the target row group. » The address information of the target row group can also include the physical address, that is, the physical address (i.e., physical byte address) of the data corresponding to the target row group inside the ORC file, which can be represented as {filePath, offset, len), where filePath is the storage path of the ORC file in the cloud storage service (such as OSS), offset represents the distance from the starting byte position in the ORC file, and len refers to the byte length from the offset position. The addressing method used in cloud storage services is physical address. Therefore, as mentioned above, downloading the file fragment corresponding to the target line group from the cloud storage service based on the address information of the target line group means sending a data retrieval request containing the physical address of the target line group to the cloud storage service. The cloud storage service determines the 0RC file from several files stored locally based on the physical address, and locates the portion between the offset position and the offset+len position in the 0RC file as the file fragment corresponding to the target line group, which is then fed back to the executor in the form of a file. It should be noted that the physical address of the target line group is actually obtained by mapping based on the logical address of the target line group. The process of determining the logical address of the target line group is as described above, and this mapping relationship can be pre-established and stored in the 0RC file, for example, it can be stored in any of the following locations: file footer, postscript, footer of each strip, index information of each strip. In an optional embodiment, as shown in Figure 4, the multi-level cache space includes a first cache space and a second cache space located in memory, and a third cache space located on the local disk. The data stored in the first cache space is uncompressed, while the data stored in the second and third cache spaces is compressed. Furthermore, taking a target row group as an example, as shown in Figure 4, the data stored in the first cache space corresponding to the target row group can be a list of decompressed data blocks (each rectangular block in Figure 4 represents a block), the data stored in the second cache space corresponding to the target row group can be a compressed byte stream, i.e., a byte array (each rectangular block in Figure 4 represents a byte), and the data stored in the third cache space corresponding to the target row group can be compressed file fragments (each rectangular block in Figure 4 represents a file fragment).The following section will describe the specific implementation process of these storage formats by writing the data corresponding to the target row group to each level of the cache space. In fact, the original ORC file stored in the cloud storage service is a compressed file, so the file fragment obtained from this compressed ORC file based on the physical address is also a compressed file. Furthermore, in practical applications, in addition to compression, some content in the ORC file may also be encoded. Accordingly, the first cache space stores the decoded and decompressed data, while the other cache spaces store compressed and encoded data. Corresponding to the data storage format of the three-level cache spaces, the addressing method of the first cache space is logical address, while the addressing methods of the second, third, and cloud storage services are physical address. Although the second cache space and the first cache space are both located in memory, they use different addressing methods due to the different data storage formats. This allows for easier use of the host's existing basic interface for reading, decompressing, and decoding byte streams; this interface is `file.read(path, offset, len)`. Since the second cache space stores byte streams that need to be decompressed and decoded, it uses the physical address of the target row group for addressing, which aligns with this interface. The third cache space is similar. Due to the limited storage capacity of memory, in practical applications, the storage capacity of the first cache space can be configured to not exceed, for example, 20% of the memory capacity, the storage capacity of the second cache space can be configured to not exceed, for example, 30% of the memory capacity, and the third cache space, located on disk, can have a larger storage capacity, such as 500GB - 1TB. OSS storage space, however, has no upper limit. The above three-level cache space sizes are merely examples, and there are no specific size restrictions between the first and second cache spaces. It can be understood that the access priority of various storage spaces, including OSS, from highest to lowest is: first cache space, second cache space, third cache space, OSS. From a compression ratio perspective, the first cache space stores decompressed data, while the second, third, and OSS cache spaces store compressed data. The compression ratio can be 2-4 times, and these storage spaces can use the same compression ratio. From the perspective of access speed, the access speed of the first cache space = memory access latency, the access speed of the second cache space = memory access latency + decompression time, the access speed of the third cache space = disk access latency + decompression time, and the access speed of OSS = network access latency + decompression time.Therefore, retrieving data from another cache level will have a shorter latency compared to retrieving the target row group from OSS. Furthermore, this embodiment also utilizes the principle of locality, storing frequently accessed data closer to the executor. To achieve this, a cache data eviction mechanism is provided for multi-level cache spaces, such as the Least Frequently Used (LFU) mechanism, but not limited to this. Taking the case where the data corresponding to the target row group is retrieved from the target cache space as an example, the access count of the data corresponding to the target row group in the target cache space is updated. Thus, when the eviction conditions corresponding to the target cache space are met, the data corresponding to a set number of row groups with low access counts is deleted from the target cache space based on the access counts of different row groups in the target cache space. First, it's understandable that if the target cache space is the first cache space, the data corresponding to the target row group obtained from the first cache space is the individual blocks contained in the target row group. In this case, you can directly read each row of data in each block and determine whether it matches the data filtering conditions. If the target cache space is the second cache space, what you obtain is the compressed byte stream corresponding to the target row group. In this case, you need to decompress and decode to obtain each row of data before determining whether it matches the data filtering conditions. If the target cache space is the third cache space, what you obtain is the compressed file fragment corresponding to the target row group. In this case, you need to decompress and decode to obtain each row of data before determining whether it matches the data filtering conditions. Second, the above-mentioned eviction condition can be that the remaining storage capacity of the target cache space is less than a set threshold. The number of row groups evicted each time can be one row group or multiple row groups. Understandably, assuming that the target row group's data is stored in multiple levels of cache, and multiple data query requests, during processing, all determine that they need to retrieve the data corresponding to the target row group, the executor will query the target row group's data in the first cache space using its logical address, update the access count of the target row group's data in the first cache space, and will not continue querying subsequent cache spaces or OSS. Therefore, the data corresponding to the target row group may have a high access count in the first cache space but a low access count in other cache spaces. This achieves the goal of storing the target row group's data with a high access count in the first cache space, which is closer to the executor, to accelerate the retrieval of the target row group's data.It is understood that the cache space closer to the executor mentioned in this embodiment does not refer to physical proximity, but rather to the query order of the multi-level cache space; the cache space queried more frequently is closer to the executor. In summary, this embodiment designs a hierarchical architecture of multi-level cache space and cloud storage service as the underlying storage, considering the data frequency of different row groups. Through the principle of locality, data from frequently accessed row groups is stored in memory, optimizing the query path. Simultaneously, algorithms such as LFU are used to dynamically manage the cache, improving the overall system's resource utilization efficiency and performance. Specifically, by establishing a multi-level cache space between memory and local disk, the speed of data querying is greatly accelerated, especially for rapid access to hot data, thereby improving query performance. The first and second cache spaces are directly stored in runtime memory, enabling fast query response at memory speed and significantly reducing query latency. By leveraging the characteristics of different levels of storage media (such as memory, local disk, and remote cloud storage services) and combining them with data compression methods, limited storage space can be effectively utilized. Furthermore, a multi-level caching mechanism reduces access to the underlying cloud storage services, optimizing storage space usage and achieving a balance between storage cost and query efficiency. Through cache eviction algorithms such as LFU, the system can automatically adapt to changes in data access patterns, ensuring that the most frequently accessed data remains in the cache. Figure 5 is a flowchart of a data query method provided in an embodiment of this disclosure. As shown in Figure 5, the method may include the following steps:

[0006] 501. In response to a query request for the target database, determine the target columns and data filtering conditions to be queried.

[0007] 502. Based on the index information of multiple row groups contained in the columnar storage file corresponding to the target database, at least one row group that matches the target column and data filtering conditions is selected from the multiple row groups, wherein different row groups correspond to different rows of multiple columns in the target database.

[0008] 503. Based on the address information of at least one row group, query the multi-level cache space sequentially according to the set query order. If the data corresponding to the target row group in at least one row group is not obtained from the multi-level cache space, download the file fragment corresponding to the target row group from the cloud storage service, and filter out the target data that meets the data filtering conditions from the file fragment corresponding to the target row group.

[0009] 504. Based on the data storage requirements of the multi-level cache space, the file fragment corresponding to the target row group is stored in the multi-level cache space. In this embodiment, it is assumed that the multi-level cache space includes the first cache space, the second cache space, and the third cache space mentioned above. For the target row group, the logical address of the target row group can be used to query the first cache space to determine whether the data block list corresponding to the target row group exists in the first cache space. If it does not exist, the logical address of the target row group is mapped to a physical address, and the second cache space is queried based on the physical address of the target row group to determine whether the byte stream (actually the compressed byte stream) corresponding to the target row group exists in the second cache space. If it does not exist, the third cache space is queried based on the physical address of the target row group to determine whether the file fragment (actually the compressed file fragment) corresponding to the target row group exists in the second cache space. If it does not exist, the file fragment (actually the compressed file fragment) corresponding to the target row group is downloaded from the cloud storage service using the physical address of the target row group. After obtaining the file fragment from the cloud storage service, on the one hand, the data corresponding to each row of the target row group is decompressed, and the target data that meets the data filtering conditions is selected for feedback to the user. On the other hand, the data corresponding to the target row group is written to each level of cache space in reverse order of the query order of each level of cache space. However, because the data storage requirements of each level of cache space are different, the above file fragment needs to be processed differently. For the third cache space, after obtaining the file fragment corresponding to the target row group, the executor can directly store the file fragment and the physical address of the target row group into the third cache space. However, in an optional embodiment, the executor can also determine whether there is another row group in the third cache space whose physical address is adjacent to the physical address of the target row group. If so, the file fragment corresponding to the target row group is merged with the file fragment corresponding to the other row group to obtain a merged file fragment, and the physical address of the target row group is merged with the physical address of the other row group to obtain a merged physical address. The merged file fragment and the merged physical address are then stored in the third cache space, and the file fragment and physical address corresponding to the other row group are deleted. Of course, if there is no file segment in the third cache space corresponding to another row group whose physical address is adjacent to the physical address of the target row group, then the physical address of the target row group and the file segment can be stored in the third cache space. It can be understood that the file segments stored in the third cache space are compressed file segments. The reason for merging file segments corresponding to adjacent physical addresses is to reduce file fragmentation in the third cache space and improve the data query efficiency of the third cache space.Because the third cache space is deployed on disk, its storage capacity is very large, resulting in a large number of file fragments. Merging file fragments with adjacent physical addresses can improve the data query efficiency of the third cache space. It should be noted that if the file fragments corresponding to the target row group are merged with those of another row group, then subsequent queries to the third cache space using the physical address of the target row group will find the merged file fragment. Since writing file fragments is time-consuming, in practical applications, the executor can start an asynchronous thread to complete the process of obtaining and writing the file fragments corresponding to the target row group to the cache space. This can be implemented using the embodiment shown in Figure 6. Specifically, assuming a user-triggered data query request, the executor starts a thread to perform the query operation; this thread is referred to as the main thread for distinction. When the main thread's query process reaches the point where it needs to query the third cache space, optionally, the executor starts another thread, referred to as the cache management thread in Figure 6. This cache management thread queries the third cache space based on the physical address of the target row group. If the file fragment corresponding to the target row group is found, it is fed back to the main thread, updating the access count of the file fragment corresponding to the target row group. If the file segment corresponding to the target line group is not found, a data retrieval request is sent to OSS, including the physical address of the target line group. The OSS file management thread first checks its local buffer for the requested data. If not, it initiates a file read task to read the file segment corresponding to the target line group from the ORC file at the physical address, and then sends this file segment back to the cache management thread. The cache management thread writes this file segment to the third cache space and sends it back to the main thread, allowing the main thread to perform data filtering. For the second cache space, the byte stream contained in the file segment corresponding to the target line group is read, and the read byte stream and the physical address of the target line group are stored in the second cache space. This byte stream is compressed data. Afterward, the remaining storage capacity of the second cache space after writing the byte stream corresponding to the target line group needs to be calculated so that a cache eviction mechanism is triggered when the remaining storage capacity falls below a set threshold. For the first cache space, the aforementioned byte stream corresponding to the target row group is decompressed to obtain multiple rows of data corresponding to the target row group. These multiple rows of data are then divided into multiple data blocks according to a set data block size. These multiple data blocks and the logical address of the target row group are stored in the first cache space. For example, the multiple rows of data corresponding to the target row group may be 10,000 rows, and the data block size may be 1,000 rows, resulting in a data block list consisting of 10 data blocks, which is then stored in the first cache space.Next, it is necessary to calculate the remaining storage capacity of the first cache space after writing multiple data blocks corresponding to the target row group, so as to trigger the cache eviction mechanism when the remaining storage capacity is lower than a set threshold. The following will describe in detail one or more embodiments of the data query apparatus of this disclosure. Those skilled in the art will understand that these apparatuses can all be configured using commercially available hardware components through the steps taught in this solution. Figure 7 is a schematic diagram of the structure of a data query apparatus provided in an embodiment of this disclosure. As shown in Figure 7, the apparatus includes: a determining module 11, a filtering module 12, and a query module 13. The determining module 11 is configured to determine the target column and data filtering conditions to be queried in response to a query request for a target database. The filtering module 12 is configured to filter at least one row group from the multiple row groups that matches the target column and the data filtering conditions, based on the index information of multiple row groups contained in the columnar storage file corresponding to the target database, wherein different row groups correspond to different rows of multiple columns in the target database. The query module 13 is configured to query the multi-level cache spaces sequentially according to the address information of the at least one row group and the set query order. If data corresponding to the target row group in the at least one row group is obtained from the target cache space, target data that meets the data filtering conditions is filtered out from the data corresponding to the target row group. The multi-level cache spaces include cache spaces located in memory and cache spaces located on the local disk. The target row group is any one of the at least one row group, and the target cache space is any one of the multi-level cache spaces. Optionally, the device further includes: an eviction processing module, configured to update the access count of the data corresponding to the target row group in the target cache space if data corresponding to the target row group is obtained from the target cache space; and when the eviction conditions corresponding to the target cache space are met, delete the data corresponding to a set number of row groups with low access counts from the target cache space based on the access counts of data corresponding to different row groups in the target cache space. Optionally, the query module 13 is further configured to: if the data corresponding to the target row group in the at least one row group is not obtained from the multi-level cache space, download the file fragment corresponding to the target row group from the cloud storage service, the file fragment being contained in the columnar storage file; and filter out the target data that meets the data filtering conditions from the file fragment corresponding to the target row group. The device further includes: a cache processing module, configured to store the file fragment corresponding to the target row group into the multi-level cache space according to the data storage requirements of the multi-level cache space.The device shown in Figure 7 can perform the steps in the foregoing embodiments. Detailed execution processes and technical effects are described in the foregoing embodiments and will not be repeated here. In one possible design, the structure of the device shown in Figure 7 can be implemented as an electronic device. As shown in Figure 8, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code. When the executable code is executed by the processor 21, the processor 21 can at least implement the data query method provided in the foregoing embodiments. Additionally, this disclosure provides a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by the processor of the electronic device, the processor can at least implement the data query method provided in the foregoing embodiments. The device embodiments described above are merely illustrative. The network elements described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort. Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of this disclosure, and not to limit it; although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

Claims 1. A data query method, comprising: In response to a query request to the target database, determine the target columns to be queried and the data filtering conditions; Based on the index information of multiple row groups contained in the columnar storage file corresponding to the target database, at least one row group matching the target column and the data filtering conditions is selected from the multiple row groups. Different row groups correspond to different rows of multiple columns in the target database. Based on the address information of the at least one row group, the multi-level cache spaces are queried sequentially according to the set query order. If data corresponding to the target row group in the at least one row group is obtained from the target cache space, target data satisfying the data filtering conditions is filtered out from the data corresponding to the target row group. The multi-level cache spaces have different data storage formats and different addressing modes, including different cache spaces located in memory. The target row group is any one of the at least one row group, and the target cache space is any one of the multi-level cache spaces.

2. The method according to claim 1, wherein, The multi-level cache space includes a first cache space and a second cache space located in memory, and a third cache space located on the local disk. The data stored in the first cache space is uncompressed, while the data stored in the second and third cache spaces is compressed.

3. The method according to claim 2, wherein, The first cache space stores data in the form of data blocks, the second cache space stores data in the form of byte streams, and the third cache space stores data in the form of file fragments.

4. The method according to claim 2, wherein, The addressing mode of the first cache space is logical address, and the addressing mode of the second cache space and the third cache space is physical address; the address information of the at least one row group includes the logical address and physical address of the at least one row group, wherein the physical address is obtained based on the mapping of the corresponding logical address.

5. The method according to any one of claims 1-4, wherein, The method further includes: if data corresponding to the target row group is obtained from the target cache space, then updating the access count of the data corresponding to the target row group in the target cache space; when the eviction condition corresponding to the target cache space is met, deleting the data corresponding to a set number of row groups with low access counts from the target cache space according to the access counts of data corresponding to different row groups in the target cache space.

6. The method according to any one of claims 2-4, wherein, The method further includes: if the data corresponding to the target row group in the at least one row group is not obtained from the multi-level cache space, then downloading the file fragment corresponding to the target row group from the cloud storage service, the file fragment being contained in the columnar storage file; filtering out the target data that meets the data filtering conditions from the file fragment corresponding to the target row group; and storing the file fragment corresponding to the target row group into the multi-level cache space according to the data storage requirements of the multi-level cache space.

7. The method according to claim 6, wherein, The step of storing the file fragment corresponding to the target row group into the multi-level cache space according to the data storage requirements of the multi-level cache space includes: if there is a file fragment corresponding to another row group whose physical address is adjacent to the physical address of the target row group in the third cache space, then the file fragment corresponding to the target row group is merged with the file fragment corresponding to the other row group to obtain a merged file fragment, and the physical address of the target row group is merged with the physical address of the other row group to obtain a merged physical address; the merged file fragment and the merged physical address are stored in the third cache space, and the file fragment and physical address corresponding to the other row group are deleted; if there is no file fragment corresponding to another row group whose physical address is adjacent to the physical address of the target row group in the third cache space, then the physical address and file fragment of the target row group are stored in the third cache space; wherein, each file fragment stored in the third cache space is a compressed file fragment.

8. The method according to claim 7, wherein, The step of storing the file segment corresponding to the target line group into the multi-level cache space according to the data storage requirements of the multi-level cache space includes: reading the byte stream contained in the file segment corresponding to the target line group, and storing the read byte stream and the physical address of the target line group into the second cache space, wherein the byte stream is compressed data.

9. The method according to claim 8, wherein, The step of storing the file fragment corresponding to the target line group into the multi-level cache space according to the data storage requirements of the multi-level cache space includes: decompressing the byte stream to obtain multiple lines of data corresponding to the target line group; dividing the multiple lines of data into multiple data blocks according to a set data block size; and storing the multiple data blocks and the logical address of the target line group into the first cache space.

10. An electronic device, comprising: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the data query method as described in any one of claims 1 to 9.

11. A non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, causes the processor to perform the data query method as described in any one of claims 1 to 9.

12. A computer program product, comprising: A computer program, when executed by a processor of an electronic device, causes the processor to perform the data query method as described in any one of claims 1 to 9. 18

Images