Data query method, query engine, device, medium, and program product

Abstract

Embodiments of the present disclosure provide a data query method, a query engine, a device, a medium, and a program product. The method comprises: in response to a query request, determining data to be queried corresponding to the query request, wherein said data can be stored in a data organization mode of a file system and / or a database system; then determining statistical information of said data on the basis of the data organization mode of said data; determining execution costs of different query strategies on the basis of the statistical information; determining a target query strategy of said data on the basis of the execution costs; and finally, responding to the query request according to the target query strategy. Therefore, in the query method, the statistical information used for determining query strategies can be determined on the basis of a mode corresponding to a data organization form. That is, the method provides a universal statistical information acquisition process, and data having different data organization modes can be queried by means of the process.

Description

[0001] Data Query Method, Query Engine, Device, Medium, and Program Product This disclosure claims priority to Chinese patent application No. 202410346722.6, filed with the China Patent Office on March 25, 2024, entitled "Data Query Method, Query Engine, Device, Medium, and Program Product," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of data query technology, and more particularly to a data query method, query engine, device, medium, and program product. Background: Big data is a collection of data characterized by large capacity, diverse types, fast access speed, and high application value. In practice, various storage systems for storing big data have emerged, such as data lakes, data warehouses, and integrated lake-warehouse systems. Furthermore, the data stored in these systems can come from different sources, meaning that the data can be stored in the storage systems in different data organization formats. Common data sources may include file systems, database systems, and so on. In practice, queries on lake data can be implemented using statistical information on the lake data. Based on the above description, how to query data in different data organization forms in a storage system becomes an urgent problem to be solved. SUMMARY OF THE INVENTION In view of this, embodiments of the present disclosure provide a data query method, query engine, device, medium, and program product for querying data in different data organization forms in a storage system. In a first aspect, embodiments of the present disclosure provide a data query method, comprising: determining, in response to a query request, data to be queried corresponding to the query request, the data to be queried comprising data stored in a data organization manner of a file system and / or database system; determining statistical information of the data to be queried based on the data organization manner of the data to be queried; determining, based on the statistical information, the execution costs of different query strategies; and responding to the query request according to a target query strategy determined based on the execution costs.In a second aspect, an embodiment of the present disclosure provides a data query method, comprising: determining, in response to a query request, data to be queried corresponding to the query request, the data to be queried including files in a file directory stored in a first logical storage unit, the first logical storage unit belonging to a logical storage unit at a first level in the file system; extracting the data to be queried to obtain sample data, the sample data being stored in a second logical storage unit at a second level in the file system, the second level being at least one level lower than the first level, the first logical storage unit including the second logical storage unit; determining statistical information of the data to be queried based on statistical information of the sample data; determining execution costs of different query strategies based on the statistical information of the data to be queried; and responding to the query request according to a target query strategy determined based on the execution costs. In a third aspect, embodiments of the present disclosure provide a data query method, comprising: determining, in response to a query request, data to be queried corresponding to the query request, the data to be queried comprising data stored in a data organization manner of a database system; receiving statistical information about the data to be queried fed back by the database system; determining, based on the statistical information, execution costs of different query strategies; and responding to the query request according to a target query strategy determined based on the execution costs. In a fourth aspect, embodiments of the present disclosure provide a query engine, comprising: a processing component and a query optimization component; the processing component, in response to a query request, determining, in response to the query request, data to be queried corresponding to the query request, the data to be queried comprising data stored in a data organization manner of a file system and / or database system; determining statistical information about the data to be queried based on the data organization manner of the data to be queried; and responding to the query request using a target query strategy; the query optimization component, in response to the statistical information, determining execution costs of different query strategies; and determining the target query strategy based on the execution costs. In a fifth aspect, embodiments of the present disclosure provide an electronic device, including: a memory configured to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data query method described in any one of aspects 1 to 4. The electronic device may also include a communication interface for communicating with other devices or communication systems. In a sixth aspect, embodiments of the present disclosure provide a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code. When the executable code is executed by a computing system of the electronic device, the computing system is enabled to implement at least the data query method described in any one of aspects 1 to 4.In a seventh aspect, embodiments of the present disclosure provide a computer program product. This computer program product includes a computer program or instructions. When executed by a processor, the computer program or instructions cause the processor to implement the data query method described in any one of aspects 1 to 4. In the data query method provided by embodiments of the present disclosure, in response to a query request, a query device can determine the data to be queried corresponding to the query request. This data to be queried can include data stored in a file system and / or database system using a data organization method. The query device then determines statistical information about the data to be queried based on the data organization method, further determines execution costs of different query strategies based on this statistical information, and then determines a target query strategy for the data to be queried based on the execution costs. Ultimately, the query request is responded to according to the target query strategy. As can be seen, in the query method described above, the statistical information used to determine the query strategy can be determined using a method corresponding to the data organization method. In other words, the method provides a universal statistical information acquisition process that can be used to collect statistical information for data in a storage system with different data organization methods, thereby ultimately implementing data query. To more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. It should be apparent that the drawings described below represent some embodiments of the present disclosure. Persons skilled in the art can also derive other drawings based on these drawings without inventive effort. Figure 1 is a flowchart of a data query method provided in an embodiment of the present disclosure; Figure 2 is a flowchart of a statistical information determination method provided in an embodiment of the present disclosure; Figure 3 is a schematic diagram of a statistical information determination process provided in an embodiment of the present disclosure; Figure 4 is a flowchart of another statistical information determination method provided in an embodiment of the present disclosure; Figure 5 is a schematic diagram of a stratified sampling process provided in an embodiment of the present disclosure; Figure 6 is a flowchart of another data query method provided in an embodiment of the present disclosure; Figure 7 is a flowchart of yet another data query method provided in an embodiment of the present disclosure; Figure 8 is a structural diagram of a query engine provided in an embodiment of the present disclosure; Figure 9 is a schematic diagram of the structure of a data query device provided in an embodiment of the present disclosure; Figure 10 is a schematic diagram of the structure of yet another data query device provided in an embodiment of the present disclosure; Figure 11 is a schematic diagram of the structure of yet another data query device provided in an embodiment of the present disclosure; and Figure 12 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.To further clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be described below in detail and in full, with reference to the accompanying drawings. It should be understood that the described embodiments represent only a portion of the embodiments of the present disclosure, and are not exhaustive. Based on the embodiments of the present disclosure, all other embodiments devised by persons of ordinary skill in the art without inventive effort are within the scope of protection of the present disclosure. The terms used in the embodiments of the present disclosure are intended solely to describe specific embodiments and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are intended to include the plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one. It should be understood that the term "and / or" as used herein is merely a term used to describe an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " generally indicates an "or" relationship between the associated objects. Depending on the context, the terms "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining" or "in response to identifying." Similarly, depending on the context, the phrases "if it is determined" or "if (the stated condition or event) is identified" can be interpreted as "upon determination" or "in response to determining" or "upon identifying (the stated condition or event)" or "in response to identifying (the stated condition or event)." It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse. It should also be noted that the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such product or system.Without further restrictions, elements defined by the phrase "comprising a..." do not preclude the presence of other identical elements in the product or system that includes the element. Before describing the methods and engines provided in the following embodiments of this disclosure, the following concepts should be explained: Data lake: A storage system for storing large-scale, diverse data. Data on the lake can belong to different data sources. That is, data generated by different data sources can be stored on the lake without changing their data organization. For example, a data source can be a file system or a database system. Data from the file system can be stored on the lake in the file system's data organization format, and data from the database system can be stored on the lake in the database system's data organization format. However, it should be noted that data from different data sources is actually stored on the data lake through mapping. That is, data in a remote file system can be mapped to the data lake to form a local file system within the data lake. Database systems operate similarly: data from a remote database system can be mapped to the data lake to form a local database system within the data lake. Data in a local file system or a remote file system can be stored using an Object Storage Service (OSS). Furthermore, both remote and local file systems include multiple layers of logical storage units, and data can be stored in these different layers. A file system's multiple layers of logical storage units may include: file directory - folder within the file directory - file - file block - file page, with the layers of these logical storage units decreasing in order. A database system, both remote and local, may include: table - data partition - data block - data page - record, with the layers of these logical storage units decreasing in order. Statistical information for data on the lake: The statistical results are obtained by taking the data in any logical storage unit at any layer and performing a field-by-field statistical analysis. These statistical results are the statistical information for that logical storage unit. Common statistical information may include at least one of the following: number of rows, data volume, the maximum value of data in a field, the null value ratio (Null Rate) of a field, the number of distinct values ​​(NDV) in a field (i.e., the field cardinality of the field), etc. Query Engine: Component used to implement data queries in the data lake.More specifically, the query engine can perform statistics on the data in the data lake and complete queries based on the statistical information of the data. The following detailed description of some embodiments of the present disclosure is provided in conjunction with the accompanying drawings. The following embodiments and features can be combined with each other unless they conflict with each other. Furthermore, the sequence of steps in the following method embodiments is merely an example and not a strict limitation. Figure 1 is a flowchart of a data query method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by a query device. Optionally, the query device can include a software device or hardware device with query capabilities. The software device can specifically support different data architectures, such as a database, a data warehouse, or a lake-warehouse integration. As shown in Figure 1, the method can include the following steps:

[0002] 5101. In response to a query request, determine data to be queried corresponding to the query request, where the data to be queried includes data stored in a data organization manner of a file system and / or a database system.

[0003] At step 5102, statistical information of the data to be queried is determined based on the data organization mode of the data to be queried. A user may generate a data query request. In response to the request, the query device may determine the data to be queried corresponding to the query request. The query device may then further determine a statistical mode corresponding to the data organization mode of the data to be queried, thereby generating statistical information about the data to be queried. This statistical information is then used to determine a query strategy corresponding to the query request. Finally, the query device may execute this query strategy to respond to the query request. Optionally, the data to be queried involved in the query request may be stored in a storage system. This storage system may store data from different sources, and the sources of the data to be queried may be the same or different. Specifically, in one case, all the data to be queried may be stored using the data organization mode of a file system. In another case, all the data to be queried may be stored using the data organization mode of a database system. In yet another case, some of the data to be queried may be stored using the data organization mode of a file system, while others may be stored using the data organization mode of a database system. Optionally, when the data to be queried is stored in the data organization format of a file system, the query device may determine the statistical information of the data to be queried by sampling the data to be queried. For details on this process, see the description in the following related embodiments. Optionally, when the data to be queried is stored in the data organization format of a database system, the query device may determine the statistical information of the data to be queried by interacting with a remote database system. For details on this process, see the description in the following related embodiments. Optionally, the statistical information of the data to be queried may include at least one of the aforementioned items: the total number of rows, the amount of data, the maximum value in a column, the null value ratio in a column, the field cardinality in a column, etc.

[0004] 5103. Determine the execution costs of different query strategies based on the statistical information.

[0005] At step 5104, the query request is responded to according to the target query strategy determined based on the execution cost. Based on the statistical information of the query data obtained in the above steps, the query device can determine the execution costs of different query strategies based on the statistical information and determine the target query strategy based on the execution costs. Ultimately, the query device can respond to the query request by executing the query strategy. Optionally, the query strategy can include query strategies for table join order and data propagation mode. The table join order can be determined by the Join operator, and the data propagation mode can be determined by the Exchange operator. The query device can use different operators to determine query strategies that include different table join orders and data propagation modes, i.e., different query strategies. The execution costs of the operators can then be estimated based on the statistical information of the query data, thereby determining the execution costs of the different query strategies. The query device can then determine the target query strategy based on the execution costs and respond to the query request according to this target query strategy. The target query strategy can have the lowest execution cost. The query device may respond to a query request by joining multiple tables involved in the query request according to the table join order specified in the target query strategy to obtain candidate data. The query conditions specified in the query request are then used to obtain target data that meets the query conditions from the candidate data. Finally, the target data can be fed back to the user as a response to the query request using the data propagation method specified in the target query strategy, effectively completing the query request. In this embodiment, in response to the query request, the query device may determine the target data corresponding to the query request. This target data may be stored in a data organization format of a file system and / or database system. Then, based on the data organization format of the target data, statistical information about the target data is determined. The execution costs of different query strategies are further determined based on this statistical information. The target query strategy for the target data is then determined based on the execution costs. Finally, the query request is responded to according to the target query strategy. As can be seen, in the query method described above, the statistical information used to determine the query strategy can be determined in a manner consistent with the data organization format. In other words, the above method provides a general statistical information acquisition process that can be used to determine statistical information for data with different data organization methods, thereby ultimately implementing data query. Optionally, the storage system storing the data to be queried in the above and following embodiments of the present disclosure may specifically be a data lake. That is, the data to be queried may be stored in the data lake in the data organization method of a file system and / or a database system.In this data lake scenario, continuing with step S102 of the embodiment shown in FIG1 , when all the data to be queried is stored in the data lake using the file system's data organization, it is stored in the data lake's local file system. When all the data to be queried is stored in the data lake using the database system's data organization, it is stored in the data lake's local database system. Some of the data to be queried is stored in the data lake using the file system's data organization, while others are stored in the data lake using the database system's data organization. Optionally, when the data to be queried is stored in the local file system on the lake using the file system's data organization, the query device can directly read data from logical storage units at different levels in the local file system. That is, the query device can read data in the local file system at different granularities, such as file directories, folders, and files. The query device can then determine statistical information about the data to be queried by sampling it. For details, see the description in the following related embodiments. Optionally, when the data to be queried is stored in a local database system on the data lake in the form of a database system's data organization, the query device can read data stored in the highest-level logical storage unit in the local database system. That is, the query device can read data in the local database system in tables. The query device can then determine statistical information about the data to be queried by interacting with the remote database system. The specific process can also be found in the description of the relevant embodiments below. As can be seen, when the data to be queried is specifically on-lake data, the method provided in the above and following embodiments of this disclosure is actually a universal statistical information acquisition process for on-lake data. This process can be used to determine statistical information for data with different data organization methods on the data lake, ultimately enabling querying data on the data lake. Furthermore, optionally, the process of determining statistical information in the above and following embodiments of this disclosure can be specifically performed by a processing component included in the query engine of the query device, which receives query requests as input and outputs statistical information. The process of determining the query strategy can be specifically performed by a query optimization component included in the query engine, which receives statistical information as input and outputs the query strategy. In practice, a query request may include a query condition. Optionally, the query condition may describe the storage location of the data involved in the query request or a data filtering condition, i.e., a filtering predicate. The data to be queried determined by the query device using the query condition may include two parts: the data involved in the query request and the data that meets the filtering condition. For clarity in the subsequent description, the data involved in the query request may be referred to as raw data, and the data within the raw data that meets the filtering condition may be referred to as filtered data.The raw data in the data to be queried can be stored in a logical storage unit. This logical storage unit can be a logical storage unit at any level provided by the data organization scheme corresponding to the raw data. Optionally, it is typically the highest-level logical storage unit. In a data lake scenario, the logical storage unit storing the raw data can be the highest-level logical storage unit in the local file system or local database system of the data lake. Specifically, the raw data can be all files in at least one folder under a file directory in the local file system, and the filtered data can be at least one row in all the files. The raw data can also be a table in the local database system, and the filtered data can be at least one record in the table. Optionally, the raw data in the data to be queried can be stored in the same or different logical storage units. Furthermore, the connection relationship between different logical storage units storing the raw data can be included in the query conditions. The query device can then retrieve the raw data based on the connection relationship in the query request. This connection relationship can also be understood as a table connection relationship. Optionally, the different logical storage units involved in the connection relationship can belong to the same or different data organization schemes. Furthermore, for the two parts of data to be queried, the statistical information of the data to be queried may specifically include statistical information for both the original data and the filtered data. For clarity in subsequent descriptions, the statistical information for each of the two parts of data may be referred to as first statistical information and second statistical information, respectively. Optionally, the manner in which the first statistical information and the second statistical information are determined is related to the data organization method. The following example illustrates the relationship between the data to be queried, the query conditions, and the statistical information: Assume that data is stored in a data lake using the data organization format of a database system. The data lake may store data for the fields "username" and "height": [User 1, height 150cm], [User 2, height 170cm], [User 3, height 160cm], [User 4, height 150cm], [User 5, height 180cm], [User 6, height 175cm], [User 7, height 180cm], [User 8, height 185cm], [User 9, height 160cm]. The above data can be stored in two tables respectively, that is, the data contained in Table 1 can be: [User 1, height 150cm], [User 2, height 170cm], [User 3, height 160cm], [User 4, height 150cm], [User 5, height 180cm], [User 6, height 150cm].Table 2 may contain the following data: [User 7, height 180cm], [User 8, height 185cm], [User 9, height 160cm]. Suppose a query request is for users in Table 2 and Table 1 whose height is greater than 160cm. The query involves the entire raw data in Tables 1 and 2, and the filter condition is "users whose height is greater than 160cm." The data to be queried is [User 2, height 170cm], [User 3, height 160cm], [User 5, height 180cm] in Table 1, as well as the entire data in Table 2. The query device can perform statistics on the data in the "Height" field. The resulting first statistical information may include: the maximum value in Table 1 is 180cm, the minimum value is 150cm, the total number of rows is 6, and the field cardinality is 4. The maximum value in Table 2 is 185cm, the minimum value is 160cm, the total number of rows is 3, and the field cardinality is 3. The second statistical information may include: a maximum value of 185 cm, a minimum value of 160 cm, a total number of rows of 6, and a field cardinality of 4. O As can be seen from the above description, the statistical information of the data to be queried is the statistical information of both the original data and the filtered data. The specific method for determining the statistical information is related to the data organization format. Optionally, the query device can obtain metadata of the data to be queried locally and use this metadata to determine the data organization format of the data to be queried. Optionally, the metadata can specifically include the Data Definition Language (DDL) corresponding to the data to be queried. The following further details the process of determining the first statistical information and the second statistical information when the original data and the filtered data are in different organizational formats. When the original data is stored in the local file system of the data lake in the data organization format of a file system, the query device can determine the first statistical information through sampling. The specific sampling process can be seen in the embodiment shown in Figure 2. Figure 2 is a flowchart of a statistical information determination method provided in an embodiment of the present disclosure. This method can also be executed by the query device. As shown in Figure 2, the method may include the following steps:

[0006] S201: Determine the total amount of raw data. The raw data includes files in a file directory stored in a first logical storage unit. The first logical storage unit is a logical storage unit at the first level of the file system. The query device may first determine the total amount of raw data. Since the raw data is stored in the data lake in the file system's data organization format, the raw data may include multiple files in a file directory stored in the first logical storage unit. The first logical storage unit is a logical storage unit at the first level of the local file system. The first logical storage unit may store a file directory. Regarding how to obtain the total amount of data, one optional method is for the query device to directly obtain the total amount of raw data using an interface provided by the remote file system. Another optional method is for the query device to traverse the raw data in the local file system to obtain the total amount of raw data. To improve the efficiency of obtaining the total amount of data, another optional method is to extract sample files from the raw data according to a sampling ratio, and then determine the total amount of raw data based on the data volume of the sample files and the sampling ratio. Here, total amount of data = data volume of sample files / sampling ratio. In practice, this approach is typically suitable for scenarios involving large amounts of raw data. To distinguish this from the following steps, the sampling ratio used when obtaining the total amount of data can be referred to as the second ratio, and the sample file sampled from the raw data using this second ratio can be referred to as the second sample file. Both this second ratio and the first ratio in step S202 can be pre-set, and there is no strict relationship between the two.

[0007] S202: Extract a first sample file from the original data according to a first ratio, and store the sample file in a first logical storage unit.

[0008] S203: Determine first statistical information of the original data based on the statistical information of the first sample file, the data volume of the first sample file, and the total data volume. Next, the query device may sample the original data at a first ratio to extract the first sample file. This is the sampling granularity of files in step S202. The first statistical information of the original data is then determined based on the data volume of the first sample file, the statistical information of the first sample file, and the total data volume of the original data. Here, first statistical information of the original data = (total data volume of the original data / total data volume of the first sample file) * statistical information of the first sample file. The method for obtaining the data volume of the first sample file can optionally be the same as the method for obtaining the total data volume of the original data. For details, refer to the relevant description in step S201 and will not be repeated here. Regarding the method for obtaining the statistical information of the first sample file, one optional method is for the query device to traverse and read the first sample file to obtain the statistical information of the first sample file. If the first sample file is stored in a local file system in a format with metadata, another optional method is for the query device to directly read the statistical information contained in the metadata. Optionally, the storage format of the first sample file can be columnar, such as Optimized Row Combination (ORC) or Parquet. Optionally, in this embodiment, the statistical information of the first sample file and the first statistical information of the original data can be the same item, such as the number of rows or the null value ratio. This embodiment provides a sampling and statistical method suitable for data lake scenarios. Specifically, the query device first samples the original data represented by multiple files according to a preset ratio, using the file as the granularity, and then amplifies the statistical information of the first sample text according to the ratio to estimate the first statistical information of the original data. When using this method, sampling can reduce the amount of data read by the query device when determining the first statistical information, thereby improving the speed of determining the first statistical information, improving the speed of determining the query strategy, and ultimately improving the response speed of the query request. In addition, the embodiment shown in Figure 2 does not limit the number of first logical storage units storing the original data. Optionally, the number of first logical storage units can be at least one. When there is only one first logical storage unit, the original data is multiple files in a file directory, and the file directory can be stored in the first logical storage unit. The query device can determine a set of statistical information corresponding to the files in the file directory in the manner shown in the embodiment of FIG. 2 . This statistical information is the first statistical information of the original data.When there are multiple first logical storage units, the original data is multiple files under multiple file directories. Each of these multiple file directories can be stored in a corresponding first logical storage unit. The query device can determine a set of statistical information corresponding to each file under each file directory, as described in the embodiment of FIG2 . These sets of statistical information can be collectively referred to as the first statistical information of the original data. When the original data is stored in a local database system of the data lake in the form of a database system, the query device cannot determine whether to perform statistical analysis on the data in the local database system. In this case, the query device can interact with a remote database system to obtain the first statistical information of the original data from the remote database system. The first statistical information can be obtained by a query optimization component in the remote database and fed back to the query device. Alternatively, after the data is stored locally, the remote database system can perform statistical analysis on the data and save the statistical information as metadata. The query device can then directly access the metadata access interface provided by the remote database to retrieve the first statistical information of the original data. Optionally, the remote database providing the interface may be a relational database, such as MySQL, provided by a relational database service.

[0009] (Relational Database Service, RDS) database, etc. Alternatively, the query device can send a statistical information query request to a remote database system. After the remote database system responds to the request, it can provide the query device with the first statistical information of the original data. Optionally, the remote database system that can respond to the statistical information query request can be a database system that supports the Structured Query Language (SQL) protocol, such as MaxCompute. The optimized query component in the remote database system can respond to the statistical information query request. Furthermore, the interaction provided in this embodiment can also be understood in conjunction with FIG3 . It should be noted that, as described in the embodiment shown in FIG1 , statistical information can be input into the query optimization component included in the query engine of the query device. Therefore, the interaction between the query device and the remote database system in this embodiment is actually the interaction between the optimized query components included in each of the two. In this embodiment, without intruding on the remote database system architecture, the query device can obtain the first statistical information of the original data through interaction between the query optimization components of the query device and the remote database system. When the filtered data is stored in the local database system of the data lake using the data organization of a database system, the query device can also interact with the remote database system to obtain second statistical information about the filtered data fed back by the remote database system. Furthermore, the interaction method provided in this embodiment can also be understood in conjunction with Figure 3. In this embodiment, without intruding on the remote database system architecture, the query device can obtain the second statistical information about the filtered data through interaction between itself and the query optimization components of the remote database system. When the filtered data is stored in the local file system of the data lake using the data organization of a file system, the query device can also determine the second statistical information through sampling. The second statistical information contains the same items as the first statistical information, such as rows or null value rates. The specific sampling process can be seen in the embodiment shown in Figure 4. Compared to the file-based sampling in the embodiment shown in Figure 2, the sampling granularity in this embodiment is smaller and richer. Figure 4 is a flow chart of another statistical information determination method provided in an embodiment of the present disclosure. This method can also be performed by the query device. As shown in Figure 4, the method may include the following steps:

[0010] S301: Extract sample data from the original data. The sample data is stored in a second logical storage unit at the second level of the file system. The second logical storage unit is contained in the first logical storage unit. The second level includes at least one level lower than the first level. The query device can extract sample data from the original data. Unlike the sample files sampled in the embodiment shown in FIG2 , the sample data extracted in this embodiment has a finer granularity. Specifically, the sample data can be stored in a second logical storage unit at the second level of the file system. The second logical storage unit is contained in the first logical storage unit. The second level can be at least one level lower than the first level. For example, the second level of the file system can correspond to a granularity smaller than that of a file, such as a file block and / or file page. Therefore, step S202 actually samples the original data in file units to sample a first proportion of files from the original data containing multiple files. Step S301 actually samples in file blocks and / or file pages to obtain the sample data. Regarding the specific sampling method for sample data, optionally, assuming that the second level corresponds to file pages, and a file page includes at least one row, the query device can first randomly select at least one file from the original data containing multiple files, then select at least one file block from the at least one file, and then further randomly select a certain number of file pages from the at least one file block. The selected file pages serve as the sample data extracted from the original data. The number of sample data extracted can optionally be pre-set or determined based on the first statistical information of the original data. The process for determining the first statistical information can be found in the description of the related embodiments above and will not be repeated here. Regarding the process of determining the number of sample data based on the first statistical information, specifically, the query device can first determine reference statistical information based on the third ratio and the first statistical information of the original data. Then, based on the reference statistical information and the statistical information of the data stored in each of the multiple second logical storage units, the query device determines a third logical storage unit from the multiple second logical storage units, and ultimately determines the data in the third logical storage unit as the sample data. The statistical information of the data in the third logical storage unit satisfies the reference statistical information. For example, assuming that the statistical information is the number of rows, and the second logical storage unit stores file pages, the query device may determine the third logical storage unit based on a preset number of rows (i.e., the reference statistical information) and the number of rows of file pages stored in different second logical storage units (i.e., the statistical information of the data stored in the second logical storage units). The number of rows of file pages stored in the third logical storage unit is greater than or equal to the preset number of rows.Because the sample data is determined in file page units, the number of rows in a file page serving as the sample data may not necessarily be exactly equal to the preset number of rows; it may also be greater than the preset number of rows. The statistical information of the data stored in the second logical storage unit is used in the process of determining the sample data quantity. The statistical information of the data stored in any second logical storage unit can optionally be estimated as follows: The statistical information of the data stored in any logical storage unit is determined based on the amount of data stored in the logical storage unit, the total amount of raw data, and the first statistical information of the raw data. The statistical information of the data stored in any logical storage unit = (first statistical information * amount of data stored in any logical storage unit) / total amount of raw data. Based on the description of step S301, it can be seen that compared to the embodiment shown in FIG2 , this embodiment performs a more multi-level sampling of the raw data. The multi-level sampling process can also be understood in conjunction with FIG5 . FIG5 does not illustrate the process of dividing the file pages into file rows at a finer granularity.

[0011] S302: Determine second statistical information of the filtered data based on the statistical information of the original data and sample data, as well as the statistical information of the data in the sample data that meets the filtering conditions. The query device can then further determine the data in the sample data that meets the query conditions and the statistical information of the data. Ultimately, the second statistical information of the filtered data = the statistical information of the data in the sample data that meets the filtering conditions * (the first statistical information of the original data / the statistical information of the sample data). The statistical information of the different data mentioned in this embodiment can be the same item as the first statistical information, such as the number of rows or the null value ratio. Similar to the embodiment shown in FIG2 , this embodiment also provides a sampling and statistical method suitable for data lake scenarios. Specifically, the query device first performs multi-level, multi-granular sampling on the original data represented by multiple files to obtain sample data. The query device then performs statistical analysis on the data in the sample data that meets the query conditions to estimate the second statistical information of the filtered data. Similarly, when using the above approach, sampling can reduce the amount of data read by the query device during the process of determining the second statistical information, thereby increasing the speed of determining the second statistical information, increasing the speed of determining the query strategy, and ultimately increasing the response speed of the query request. As can be seen from the above embodiments, the data to be queried can include data from one or more sources. For data to be queried originating from the local file system, the query device can determine the data's statistical information using sampling at different levels. For data to be queried originating from the local database system, the query device can determine the data's statistical information by interacting with a query optimization component in a remote database system. The following uses a specific query request as an example to illustrate the specific process of implementing a data query using the methods provided in the above embodiments of the present disclosure. Assume that the query request is: SELECT TABLE_A.col_a FROM TABLE_A, TABLE_B WHERE TABLE_A.col_a=TABLE_B.col_a and TABLE_A.col_b=1. In the query request, "TABLE_A.col_a=TABLE_B.col_a" represents the table join relationship described by the query condition, indicating that the query involves Table A and Table B. "TABLE_A.col_b=1" represents the filter condition described by the query condition. This query request intends to retrieve the data in Table A that satisfies the filter condition "TABLE_A.col_b=1" as well as all the data in Table B.Assume that Table A is stored in the data lake using the file system's data organization format, and Table B is stored in the data lake using the database system's data organization format. For the above query request, the data in Table A and Table B can be collectively referred to as the raw data mentioned in the above embodiments. The data in Table A that meets the filter condition "TABLE_A.col_b=1" is the filtered data mentioned in the above embodiments. The statistical information required to respond to this query request may include the number of rows in Table A (i.e., the first statistical information in the above embodiments), the number of rows in Table B (i.e., the first statistical information in the above embodiments), and the number of rows in Table A that meet the filter condition (i.e., the second statistical information in the above embodiments). Based on the above premise, the query device can determine the number of rows in Table A using the sampling method shown in Figure 2, the number of rows in Table A that meet the filter condition using the stratified sampling method shown in Figure 4, or the number of rows in Table B through interaction with query optimization components. Ultimately, the query request can determine a query strategy based on the statistical information obtained above and respond to the query request according to this query strategy. For details not described in detail in this embodiment and the technical effects that can be achieved, please refer to the relevant descriptions of the above embodiments and will not be repeated here. Figure 6 is a flowchart of another data query method provided by an embodiment of the present disclosure. When the data to be queried is stored in a data lake using the data organization of a file system, the following method can be used to respond to the query request. As shown in Figure 6, the method may include the following steps:

[0012] S401. In response to a query request, determine to-be-queried data corresponding to the query request, where the to-be-queried data includes files in a file directory stored in a first logical storage unit, and the first logical storage unit belongs to a logical storage unit at a first level in a file system.

[0013] S402: Extract the data to be queried to obtain sample data. The sample data is stored in a second logical storage unit at a second level in the file system. The second level is at least one level lower than the first level. The first logical storage unit includes the second logical storage unit.

[0014] 5403. Determine statistical information of the data to be queried based on the statistical information of the sample data.

[0015] 5404. Determine the execution costs of different query strategies based on statistical information of the data to be queried.

[0016] S405: Respond to the query request according to the target query strategy determined based on the execution cost. In response to the query request, the query device determines the target data to be queried corresponding to the query request. The target data to be queried may include files in a file directory stored in a first logical storage unit, which is a logical storage unit at the first level of the file system. The query device then extracts the target data to be queried to obtain sample data. The sample data is stored in a second logical storage unit at the second level of the file system, where the second level is at least one level of the first level, and the first logical storage unit includes the second logical storage unit. The sample data extraction process and the method for determining statistical information about the sample data can be found in the description of the embodiment shown in FIG4 . Finally, the query device can determine the execution costs of different query strategies based on the statistical information about the target data, determine a target query strategy based on the execution costs, and execute the target query strategy to respond to the query request. The query strategies and the specific query request response process can be found in the description of the embodiment shown in FIG1 and will not be further described here. Optionally, the data to be queried mentioned in this embodiment may also be stored in different storage systems, such as on a data lake. In this embodiment, the query device may determine statistical information of the data to be queried stored in the data organization form of a file system through stratified sampling of different granularities. Sampling can reduce the amount of data read by the query device in the process of determining statistical information, thereby increasing the speed of determining the second statistical information, increasing the speed of determining the query strategy, and ultimately increasing the response speed of the query request. In addition, for the contents not described in detail in this embodiment and the technical effects that can be achieved, please refer to the relevant descriptions of the above embodiments, and will not be repeated here. Figure 7 is a flowchart of another data query method provided by an embodiment of the present disclosure. When the data to be queried is stored in the local database system of the data lake in the data organization form of a database system, the query device may use the following method to implement the response to the query request. As shown in Figure 7, the method may include the following steps:

[0017] 5501. In response to a query request, determine to-be-queried data corresponding to the query request, where the to-be-queried data includes data stored in a data organization manner of a database system.

[0018] 5502. Receive statistical information of the data to be queried fed back by the database system.

[0019] 5503. Determine the execution costs of different query strategies based on the statistical information.

[0020] At step 5504, the query request is responded to according to the target query strategy determined based on the execution cost. In response to the query request, the query device determines the data to be queried corresponding to the query request. The data to be queried includes data stored in the database system's data organization format. Optionally, the data to be queried may include data in the local database system of the data lake. The query device may then receive statistical information about the data to be queried from the remote database system and use this statistical information to respond to the query request. In this embodiment, without intruding on the remote database system architecture, the query device can obtain statistical information about the data to be queried through interaction between itself and the query optimization components of the remote database system. Details not described in detail in this embodiment and the technical effects achieved can be found in the relevant descriptions of the above embodiments and will not be elaborated upon here. The above description of the query device's response to a query request has been described from a method perspective. As can be seen from the description of the embodiment shown in FIG1 , the above methods can more accurately be executed by a query engine deployed in the query device. The query device can be a device independent of the data lake. The data query process described above will also be described from the perspective of the query engine. Figure 8 is a schematic diagram of the structure of a query engine provided by an embodiment of the present disclosure. As shown in Figure 8, the query engine may include a processing component and a query optimization component. The processing component may determine the query data associated with the query request in response to a query request. The query data may be stored in a file system and / or database system data organization. The processing component may then determine statistical information about the query data based on the data organization of the query data. This statistical information may be sent to the query optimization component. The query optimization component may then determine the execution costs of different query strategies based on the statistical information and determine a target query strategy based on the execution costs. Ultimately, the processing component may respond to the query request by executing the target query strategy. For details not described in detail in this embodiment, please refer to the relevant descriptions of the above embodiments and will not be repeated here. In this embodiment, the processing component in the query engine responds to a query request to determine the query data corresponding to the query request. This query data may be stored in a file system and / or database system data organization. The processing component may then determine statistical information about the query data based on the data organization of the query data. The query optimization component in the query engine can further determine the execution costs of different query strategies based on this statistical information, and then determine the target query strategy for the data to be queried based on the execution costs. Ultimately, the processing component can respond to the query request according to this target query strategy.As can be seen, in the above process, the statistical information used to determine the query strategy can be determined by the processing component in a manner corresponding to the data organization format. In other words, the above method provides a universal statistical information acquisition process. By executing this process, the query engine can determine statistical information for data with different data organization formats, thereby ultimately implementing data query. Optionally, the data to be queried mentioned in this embodiment can be stored in any storage system, such as a data lake. The above query engine operation process is thus a universal statistical information acquisition process for lake data. Optionally, as described in the above embodiments, the data to be queried corresponding to the query request can include raw data and filtered data within the raw data that meets the query criteria. The processing component in the query engine can then determine statistical information using sampling or interactive methods based on the respective data organization formats of the raw data and filtered data. The specific process of obtaining statistical information can be found in the relevant descriptions of the embodiments shown in Figures 2 to 5 above and will not be elaborated upon here. The data query device according to one or more embodiments of the present disclosure will be described in detail below. Those skilled in the art will appreciate that these data query devices can be constructed using commercially available hardware components and configured using the steps taught in this solution. FIG9 is a schematic diagram of the structure of a data query device provided in an embodiment of the present disclosure. As shown in FIG9 , the device may include: a first data determination module 11, configured to, in response to a query request, determine the data to be queried corresponding to the query request, wherein the data to be queried includes data stored in a data organization manner of a file system and / or a database system; a first information determination module 12, configured to determine statistical information of the data to be queried based on the data organization manner of the data to be queried; a first response module 13, configured to determine the execution costs of different query strategies based on the statistical information; and respond to the query request according to a target query strategy determined based on the execution costs. Optionally, the data to be queried is stored in a data lake. The first data determination module 11 is configured to obtain raw data based on query conditions carried in the query request, wherein the raw data is stored in a logical storage unit provided by the data organization manner corresponding to the raw data; determine filtered data within the raw data that meets the query conditions; and determine the data to be queried that includes the raw data and the filtered data. Optionally, the first information determining module 12 is configured to determine first statistical information of the original data according to a data organization mode of the original data; and determine second statistical information of the filtered data according to a data organization mode of the filtered data.Optionally, the first information determination module 12 is configured to determine the total amount of the original data, where the original data includes files in a file directory stored in a first logical storage unit, the first logical storage unit being a logical storage unit at a first level in the file system; extract first sample files from the original data according to a first ratio, the sample files being stored in the first logical storage unit; and determine first statistical information of the original data based on statistical information of the first sample files, the data amount of the first sample files, and the total amount of data. Optionally, the first information determination module 12 is configured to extract second sample files from the original data according to a second ratio; and determine the total amount of the original data based on the data amount of the second sample files and the second ratio. Optionally, the first information determination module 12 is configured to extract sample data from the raw data, the sample data being stored in a second logical storage unit at a second level in the file system, the second logical storage unit being included in the first logical storage unit, the second level including at least one level lower than the first level; determine second statistical information of the filtered data based on statistical information of each of the raw data and the sample data, and statistical information of data in the sample data that meets the query condition. Optionally, the first information determination module 12 is configured to determine reference statistical information according to a third ratio and the first statistical information of the raw data; determine a third logical storage unit in the second logical storage unit based on the reference statistical information and statistical information of data stored in each of the second logical storage units; and determine the data in the third logical storage unit as the sample data. Optionally, the first information determination module 12 is configured to determine statistical information of the data stored in any one of the second logical storage units based on the data volume stored in the any one logical storage unit, the total volume of the raw data, and the first statistical information of the raw data. Optionally, the raw data includes a data table stored in the database system. The first information determination module 12 is configured to receive first statistical information of the raw data and second statistical information of the filtered data fed back by the database system. Optionally, the query condition includes a connection relationship between different logical storage units, where the different logical storage units belong to at least one data organization form. The first data determination module 11 is configured to determine the data stored in each of the different logical storage units corresponding to the connection relationship as the raw data.Optionally, the device further includes: an organization method determination module 14, configured to obtain metadata for the data to be queried; and to determine a data organization method for the data to be queried based on the metadata. The device shown in FIG9 can execute the method of the embodiments shown in FIG1 to FIG5 . For portions not described in detail in this embodiment, reference can be made to the relevant descriptions of the embodiments shown in FIG1 to FIG5 . The execution process and technical effects of this technical solution are described in the embodiments shown in FIG1 to FIG5 , and will not be repeated here. FIG10 is a schematic structural diagram of another data query device provided in an embodiment of the present disclosure. As shown in FIG10 , the device may include: a second data determination module 21, configured to, in response to a query request, determine the data to be queried corresponding to the query request. The data to be queried includes files in a file directory stored in a first logical storage unit, the first logical storage unit being a logical storage unit at a first level in the file system. A sampling module 22, configured to extract sample data from the data to be queried, the sample data being stored in a second logical storage unit at a second level in the file system, the second level being at least one level lower than the first level, the first logical storage unit including the second logical storage unit. The second information determination module 23 is configured to determine the statistical information of the data to be queried based on the statistical information of the sample data. The second response module 24 is configured to determine the execution costs of different query requests based on the statistical information of the data to be queried, and respond to the query requests according to the target query strategy determined based on the execution costs. The device shown in FIG10 can execute the method of the embodiment shown in FIG6 . For portions not described in detail in this embodiment, reference can be made to the relevant description of the embodiment shown in FIG6 . The execution process and technical effects of this technical solution are described in the embodiment shown in FIG6 and will not be repeated here. FIG11 is a schematic diagram of the structure of another data query device provided in an embodiment of the present disclosure. As shown in FIG11 , the device may include: a third data determination module 31, configured to determine, in response to a query request, the data to be queried corresponding to the query request, the data to be queried including data stored in the data organization mode of a database system; a receiving module 32, configured to receive statistical information of the data to be queried fed back by the database system; a third response module 33, configured to determine the execution costs of different query strategies based on the statistical information, and respond to the query request according to the target query strategy determined based on the execution costs. The device shown in FIG11 can execute the method of the embodiment shown in FIG7. For parts not described in detail in this embodiment, reference can be made to the relevant description of the embodiment shown in FIG7.The implementation process and technical effects of this technical solution are described in the embodiment shown in FIG. 7 and will not be repeated here. In one possible design, the data query method provided in the above embodiments can be applied to an electronic device. As shown in FIG. 12 , the electronic device may include a processor 41 and a memory 42. The memory 42 is used to store a program that supports the electronic device in executing the data query method provided in the embodiments shown in FIG. 1 to FIG. 7 . The processor 41 is configured to execute the program stored in the memory 42. The program includes one or more computer instructions. When executed by the first processor 41, the one or more computer instructions can implement all or part of the steps in the embodiments shown in FIG. 1 to FIG. 7 . The electronic device may also include a communication interface 43 for communicating with other devices or communication systems. In addition, embodiments of the present disclosure provide a computer storage medium for storing computer software instructions used by the above electronic device, including the program for executing the data query method shown in FIG. 1 to FIG. 7 . In addition, embodiments of the present disclosure provide a computer program product. The computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is enabled to implement the steps or functions of the data query method shown in Figures 1 to 7 . Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present disclosure and are not intended to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art will understand that the technical solutions described in the aforementioned embodiments may be modified or some of the technical features may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

Claims 1. A data query method, comprising: In response to a query request, determining data to be queried corresponding to the query request, where the data to be queried includes data stored in a data organization manner of a file system and / or a database system; Determining statistical information of the data to be queried based on a data organization mode of the data to be queried; determining execution costs of different query strategies based on the statistical information; The query request is responded to according to a target query strategy determined according to the execution cost.

2. The method according to claim 1, wherein the data to be queried is stored in a data lake; The determining of the data to be queried corresponding to the query request includes: obtaining original data according to the query conditions carried in the query request, wherein the original data is stored in a logical storage unit provided by a data organization method corresponding to the original data; determining filtered data that meets the query conditions in the original data; and determining the data to be queried including the original data and the filtered data.

3. The method according to claim 2, wherein determining the statistical information of the data to be queried according to the data organization mode of the data to be queried comprises: determining first statistical information of the original data according to a data organization mode of the original data; Determine second statistical information of the filtered data according to a data organization mode of the filtered data.

4. The method according to claim 3, wherein determining the first statistical information of the original data comprises: Determining a total amount of the original data, where the original data includes files in a file directory stored in a first logical storage unit, where the first logical storage unit belongs to a logical storage unit at a first level in the file system; extracting a first sample file from the original data according to a first ratio, and storing the sample file in the first logical storage unit; Determine first statistical information of the original data according to the statistical information of the first sample file, the data volume of the first sample file, and the total data volume.

5. The method according to claim 4, wherein determining the total amount of the original data comprises: extracting a second sample file from the original data according to a second ratio; and determining a total amount of the original data according to a data amount of the second sample file and the second ratio.

6. The method according to claim 4 or 5, wherein determining the second statistical information of the filtered data comprises: extracting sample data from the original data, wherein the sample data is stored in a second logical storage unit at a second level in the file system, wherein the second logical storage unit is contained in the first logical storage unit, and wherein the second level includes at least one level lower than the first level; and determining the second statistical information of the filtered data based on the respective statistical information of the original data and the sample data, and the statistical information of the data in the sample data that meets the query condition.

7. The method according to claim 6, wherein extracting sample data from the original data comprises: determining reference statistical information according to the third ratio and the first statistical information of the original data; determining a third logical storage unit in the second logical storage unit according to the reference statistical information and statistical information of the data stored in each of the second logical storage units; The data in the third logical storage unit is determined as the sample data.

8. The solution according to claim 7, wherein the method further comprises: For the data stored in any logical storage unit in the second logical storage unit, according to any logical storage unit The statistical information of the data stored in any one of the logical storage units is determined based on the data volume of the data stored in the storage unit, the total data volume of the original data, and the first statistical information of the original data.

9. The method according to claim 3, wherein the original data comprises a data table stored in the database system; and determining the first statistical information of the original data comprises: Receive first statistical information of the original data and second statistical information of the filtered data fed back by the database system.

10. The method according to any one of claims 2 to 8, wherein the query condition includes a connection relationship between different logical storage units, and the different logical storage units belong to at least one data organization form; and obtaining the original data stored in the logical storage units according to the query condition carried in the query request comprises: The data stored in each of the different logical storage units corresponding to the connection relationship is determined as the original data.

11. The method according to any one of claims 1 to 0, further comprising: Obtaining metadata of the data to be queried; A data organization mode of the data to be queried is determined according to the metadata.

12. A data query method, comprising: In response to a query request, determining to-be-queried data corresponding to the query request, the to-be-queried data including files in a file directory stored in a first logical storage unit, the first logical storage unit being a logical storage unit at a first level in the file system; extracting the data to be queried to obtain sample data, where the sample data is stored in a second logical storage unit at a second level in the file system, where the second level is at least one level lower than the first level, and the first logical storage unit includes the second logical storage unit; and determining statistical information of the data to be queried based on statistical information of the sample data; Determine the execution costs of different query strategies based on the statistical information of the data to be queried; The query request is responded to according to a target query strategy determined according to the execution cost.

13. A data query method, comprising: In response to a query request, determining data to be queried corresponding to the query request, where the data to be queried includes data stored in a data organization manner of a database system; receiving statistical information of the data to be queried fed back by the database system; determining execution costs of different query strategies according to the statistical information; The query request is responded to according to a target query strategy determined according to the execution cost.

14. A query engine comprising: processing components and query optimization components; The processing component is configured to determine, in response to a query request, data to be queried corresponding to the query request, where the data to be queried includes data stored in a data organization manner of a file system and / or a database system; Determining statistical information of the data to be queried according to a data organization mode of the data to be queried; Responding to the query request using a target query strategy; The query optimization component is used to determine the execution costs of different query strategies according to the statistical information; The target query strategy is determined according to the execution cost.

15. An electronic device, comprising: A memory and a computing system; wherein the memory stores executable code, and when the executable code is executed by the computing system, the computing system executes the data query method according to any one of claims 1 to 13.

16. A non-transitory machine-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a computing system of an electronic device, the computing system is caused to perform the operation as claimed in claim 16. Find any one of the data query methods 1 to 13.

17. A computer program product, comprising a computer program or instructions, which, when executed by a processor, enables the processor to implement the steps of the data query method according to any one of claims 1 to 13. 17

Images