Data query method, system and device, storage medium and program product

Abstract

The embodiment of the invention provides a data query method, system and device, a storage medium and a program product. The technical scheme provided by the embodiment of the invention is suitable for the technical field of big data. The distributed system comprises a main node provided with a cache used for storing column statistical information, and a query component and a management component run on the main node; the method comprises the steps that when a query assembly receives a query request for a to-be-queried table in a data lake, a management assembly is called according to the query request; the management component obtains target column statistical information associated with the target column from the cache, and screens out a first data file, matched with the first query condition, of the to-be-queried table according to the target column statistical information; the management component returns file information of the first data file to the query component; and the query component determines a target query result of the query request according to the file information of the first data file. According to the technical scheme provided by the embodiment of the invention, the query efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data query method, system, device, storage medium and program product. Background Technology

[0002] Data skipping is a technique used in big data processing to improve query performance. It primarily helps the query optimizer more accurately assess query costs and decide whether to skip certain data. The basic idea is to skip data irrelevant to the current query during data reading, thereby reducing the amount of data that needs to be scanned and processed, and improving query efficiency. During query execution, column statistics are used to evaluate the match between query conditions and each data file, determining whether to skip that data file to reduce unnecessary I / O (Input / Output) operations and improve query performance / efficiency.

[0003] In practical applications, when the dataset is small, using column statistics to skip data can improve query efficiency. However, when the dataset is large, the efficiency gains from using column statistics to skip data decrease, and it may even lead to a deterioration in the overall performance of the query system. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a data query method, system, device, storage medium and program product that solves or at least partially solves the above problems.

[0005] A first aspect of this application provides a data query method for a distributed system, the distributed system comprising: a master node, the master node including a cache for storing column statistics, and the master node running a query component for querying data and a management component for managing a data lake; the method comprising:

[0006] When the query component receives a query request for a table to be queried in the data lake, it calls the management component according to the query request, wherein the query request includes a first query condition related to the target column;

[0007] After being invoked by the query component, the management component retrieves the target column statistics associated with the target column from the cache, and filters out the first data file of the table to be queried that matches the first query condition based on the target column statistics.

[0008] The management component returns the file information of the first data file to the query component;

[0009] The query component determines the target query result of the query request based on the file information of the first data file.

[0010] A second aspect of this application provides a distributed system, comprising: a master node, the master node including a cache for storing column statistics, and running a query component for querying data and a management component for managing a data lake; wherein...

[0011] The query component is used to invoke the management component according to the query request when it receives a query request for a table to be queried in the data lake, wherein the query request includes a first query condition related to the target column;

[0012] The management component is used to retrieve target column statistics associated with the target column from the cache after being invoked by the query component, and to filter out the first data file of the table to be queried that matches the first query condition based on the target column statistics.

[0013] The management component is used to return file information of the first data file to the query component;

[0014] The query component is used to determine the target query result of the query request based on the file information of the first data file.

[0015] A third aspect of this application provides an electronic device. The electronic device includes: a memory and a processor, wherein,

[0016] The memory is used to store programs;

[0017] The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a computer, can implement the method described in any of the preceding claims.

[0019] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0020] The technical solution provided in this application uses column statistics to filter data files, selecting those that match the current query conditions. This reduces the amount of data actually read and processed during the query process, thereby saving computing resources and improving query efficiency. Furthermore, the column statistics are cached in the cache of the master node in the distributed system, and subsequent queries retrieve the column statistics from the master node's cache. Obtaining column statistics from the master node's cache is faster, more efficient, and consumes fewer computing resources compared to obtaining them from the file system. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of a distributed system provided in an embodiment of this application;

[0023] Figure 2 A schematic flowchart illustrating a data query method provided in an embodiment of this application;

[0024] Figure 3 A flowchart illustrating a data query method provided in yet another embodiment of this application;

[0025] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0027] Furthermore, some processes described in the specification, claims, and accompanying drawings of this application include multiple operations that appear in a specific order. These operations may be performed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "At least one" herein refers to one or more.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0029] First, the terminology used in the embodiments of this application will be explained. It should be understood that this explanation is for the purpose of making the embodiments of this application clearer and does not necessarily constitute a limitation on the embodiments of this application.

[0030] Data Query Language (DQL) statements are a type of Structured Query Language (SQL) statement. Their basic structure consists of the SELECT, FROM, and WHERE keywords. The SELECT keyword specifies the field names to be selected / queried (i.e., the fields / columns to be queried), with each field name separated by a semicolon, such as SELECT ID, City, ... . When selecting all fields in a table, you can directly use SELECT *. The FROM keyword specifies the table name to be retrieved (i.e., the table to be queried). The WHERE keyword specifies the query conditions (or filtering conditions, query predicates). Query conditions refer to the conditions that the query results must meet; they define which records should be included in the query results and which should be excluded.

[0031] A table's data file refers to the file used to store the records of that table. Each data file can contain multiple records. Each record refers to a row of data in the table. The data file format used in this article can be a columnar storage format (e.g., Parquet). Columnar storage is a data storage technique where data is stored column-wise rather than row-wise.

[0032] Column statistics are statistical descriptions of the data distribution and characteristics of columns in a data table. Column statistics are obtained by statistically analyzing a single data file. Column statistics may include, but are not limited to, minimum values, maximum values, and the number of null values. Minimum and maximum values ​​represent the minimum and maximum values ​​of the column data in the data file (e.g., a columnar storage format file); the number of null values ​​indicates the number of NULL values ​​in the column, which helps optimize queries involving NULL values. When storing data, some statistics can be calculated for each data file. Column statistics are typically stored in the database's metadata table so that query optimizers and other database components can access and use them when needed. The accuracy and validity of column statistics are crucial for database performance. Therefore, during database design and maintenance, column statistics need to be updated and maintained regularly to ensure their accuracy and validity. In this article, column statistics for a specific column in a data file refer to the statistical information of the column data stored in that data file, such as minimum values, maximum values, and the number of null values. Optionally, column statistics can be stored as a metadata table.

[0033] Partition pruning, also known as partition elimination, is a technique associated with partitioned tables. Its main purpose is to improve query performance by avoiding access to unnecessary data partitions. In partition pruning, query operations are restricted to specific partitions or sets of partitions, thereby reducing the amount of data that needs to be processed. Specifically, the principle of partition pruning is that when query conditions are related to the partition key (also called the partition column), it can be calculated which partitions contain the required data, and then only those partitions are accessed, instead of scanning the entire table. For example, if a table is partitioned by time, and the query only cares about data within a specific time range, then the query can only query partitions that involve that specific time range, skipping partitions that do not involve that specific time range.

[0034] Research revealed that as datasets continue to grow, the volume of column statistics also increases. This leads to a greater demand for computational resources and a longer processing time for reading and analyzing column statistics from the file system. The increased computational resource requirements result in performance bottlenecks, while the longer processing time leads to longer query latency.

[0035] To achieve efficient data querying, this application proposes a caching scheme for column statistics. This caching scheme can alleviate the system burden caused by the expansion of column statistics and ensure the fast accessibility of column statistics during querying, thereby improving query performance or query efficiency.

[0036] Before introducing the data query method provided in the embodiments of this application, the distributed system involved in the data query method provided in the embodiments of this application will be introduced. For example... Figure 1 As shown, the distributed system includes a master node 100 and multiple worker nodes 200. The master node 100 is communicatively connected to each of the worker nodes 200. The master node 100 is responsible for central scheduling, dispatching multiple worker nodes to execute tasks. The worker nodes execute the specific tasks distributed by the master node. The master node 100 includes a cache for storing column statistics, and runs a query component for retrieving data and a management component for managing the data lake.

[0037] For example, in a distributed system that supports querying tables in the Hadoop Upserts Deletes and Incrementals (Hudi) component, the query component can be an SQL component, and the management component can be a Hudi component.

[0038] The query component is used to invoke the management component according to the query request when it receives a query request for a table to be queried in the data lake. The query request includes a first query condition related to the target column.

[0039] The management component is used to retrieve target column statistics associated with the target column from the cache after being invoked by the query component, and to filter out the first data file of the table to be queried that matches the first query condition based on the target column statistics.

[0040] The management component is used to return file information of the first data file to the query component;

[0041] The query component is used to determine the target query result of the query request based on the file information of the first data file.

[0042] In one alternative implementation, the system further includes: multiple working nodes;

[0043] The query component is specifically configured to: determine at least one first working node from the plurality of working nodes; generate a query task for the query request based on the file information of the first data file; distribute the query task to the at least one first working node; and determine the target query result of the query request based on the execution result returned by the at least one first working node.

[0044] The specific processing logic within each of the above components and the calling process between components will be described in detail in the following embodiments. Similarly, the specific processing logic within each of the above nodes and the specific interaction process between the master node and worker nodes will be described in detail in the following embodiments.

[0045] Figure 2 This is a flowchart illustrating a data query method for a distributed system according to an embodiment of this application. The method is applied to the aforementioned distributed system. Figure 2 As shown, the method includes:

[0046] 201. When the query component receives a query request for a table to be queried in the data lake, it calls the management component according to the query request.

[0047] The query request includes a first query condition related to the target column.

[0048] Users can send query requests to the master node through their terminals. The information that may be carried in the query request includes, but is not limited to: user information, table information of the table to be queried, column information of the column to be queried, and query conditions.

[0049] The user information is used to uniquely identify users and may include, but is not limited to, user identifiers, such as user IDs. Table information is used to uniquely identify tables. For example, table information can be the table name. In practical applications, sometimes tables have the same name; therefore, the aforementioned table information can specifically be the table's storage path. The table's storage path refers to the table's storage path on the disk or file system (also called the storage directory), and the table's storage path can uniquely identify the table. Query conditions refer to the conditions that the query results must meet. Query conditions define which records should be included in the query results and which records should be excluded from the query results. In this embodiment, the query conditions include: a first query condition related to the target column.

[0050] The query request can be a query request written in Structured Query Language (SQL), also known as a DQL statement.

[0051] After receiving a query request, the query component of the master node can determine whether the table to be queried is a data table in the data lake, i.e., whether it is a data lake table, based on the table information (e.g., table name) carried in the query request. For example, a data lake table could be a Hudi table. When it is determined that the table to be queried is a data lake table, the relational API interface for that data lake table is retrieved from the cache. This relational API interface is provided by the management component to the query component to provide an interface for accessing and manipulating relevant data in the data lake table. The query component calls this relational API interface to invoke the management component to filter out the first data file of the table to be queried that matches the first query condition.

[0052] 202. After being invoked by the query component, the management component retrieves the target column statistics associated with the target column from the cache.

[0053] The cache contains statistical information of columns used in historical queries.

[0054] Optionally, the cache may cache column statistics for columns involved in the query conditions of historical query requests. For example, if the first query condition in a historical query request involves column a of table A, then the cache may cache column statistics for column a of table A. Here, the column statistics for column a of table A refer to the column statistics for column a of each of the multiple data files of table A.

[0055] In one feasible implementation, the target column statistics of the target column of the table to be queried can be retrieved from the cache based on the table information of the table to be queried and the column information of the target column.

[0056] 203. The management component filters out the first data file of the table to be queried that matches the first query condition based on the statistical information of the target column.

[0057] In one feasible implementation, the first data file matching the first query condition of the table to be queried can be selected from multiple data files of the table to be queried based on the target column statistics information of the target column.

[0058] It should be noted that the first data file of the table to be queried that matches the first query condition refers to the data file containing table records that satisfy the first query condition. The first data file of the table to be queried that does not match the first query condition refers to the data file that does not contain table records that satisfy the first query condition.

[0059] In other words, based on the target column statistics, a first data file containing table records that meet the first query conditions is selected from at least one second data file of the table to be queried.

[0060] For example, the first query condition is: column a is greater than or equal to 50. The column statistics of column a in the second data file are: minimum value is 0 and maximum value is 40. Therefore, it can be determined that the second data file does not contain any table records that meet the first query condition. Thus, the second data file is filtered out.

[0061] For example, the first query condition is: column a is greater than or equal to 50, and a is a non-partitioned column; the column statistics of column a in the second data file are: minimum value is 40 and maximum value is 80. Therefore, it can be determined that the second data file contains table records that meet the first query condition, so the second data file is retained.

[0062] For example, the query condition is: a = null (empty value), where a is a non-partitioned column; the column statistics for a in the second data file are: the number of empty values ​​is 0; therefore, it can be determined that the second data file does not contain any table records that meet the first query condition, and thus, the second data file can be filtered out.

[0063] The process of selecting the first data file that matches the first query condition from the target column statistics can be called the data skipping process.

[0064] 204. The management component returns the file information of the first data file to the query component.

[0065] The file information of the first data file can be a filename, which uniquely identifies the first data file. There can be one or more first data files. For example, the management component can send a list of filenames consisting of multiple first data file filenames to the query component.

[0066] 205. The query component determines the target query result of the query request based on the file information of the first data file.

[0067] The query component can generate a query task based on the file information of the first data file and the query request. The query component can distribute the query task to one or more worker nodes, which will then execute the query task. Based on the execution results returned by the one or more worker nodes, the target query result of the query request is determined.

[0068] The technical solution provided in this application utilizes column statistics to filter data files, selecting those that match the current query conditions. This reduces the amount of data actually read and processed during the query process, thereby saving computing resources and improving query efficiency. Furthermore, column statistics used in historical queries are cached in the master node's cache within the distributed system. Subsequent queries retrieve column statistics from the master node's cache. Obtaining column statistics from the master node's cache is faster, more efficient, and consumes fewer computing resources compared to obtaining them from the file system.

[0069] Furthermore, in a distributed system, the computing and storage resources allocated to the master node are relatively fixed and will not be arbitrarily reclaimed, while the computing and storage resources allocated to the worker nodes are dynamic. Once a worker node is idle, its resources may be reclaimed. If column statistics are cached on the worker nodes, once the worker node's resources are reclaimed, the cached column statistics will become unusable, which is equivalent to the cache being invalid. Therefore, the embodiment of this application caches column statistics on the master node, which helps to ensure the effectiveness of the cache.

[0070] In an optional implementation, the distributed system further includes multiple worker nodes; the step 205 above, "the query component determines the target query result of the query request based on the file information of the first data file," can be implemented using the following steps:

[0071] 2051. The query component determines at least one first working node from the plurality of working nodes.

[0072] The determination method can be set according to actual needs, and this application embodiment does not impose specific limitations on it. For example, a working node with a load rate lower than a preset threshold can be selected as the first working node, or at least one first working node can be randomly selected from multiple working nodes. This application embodiment does not impose specific limitations on it.

[0073] 2052. The query component generates a query task for the query request based on the file information of the first data file.

[0074] For example, a query task can be generated based on the filename of a first data file. This query task includes at least one subtask. For example, there can be multiple first data files, which can be grouped into multiple sets of files; each set of files corresponds to a subtask, and each subtask carries the filename of that set of files. Optionally, each subtask may also carry table information of the table to be queried, such as the table's storage path.

[0075] 2053. The query component distributes the query task to the at least one first working node.

[0076] For example, at least one subtask is distributed to at least one first worker node; after receiving the subtask, the at least one first worker node executes the received subtask and obtains the execution result.

[0077] The first working node can find the corresponding data file based on the table information of the table to be queried in the subtask and the file name of each data file; then, it executes the query operation of the query request (including data reading and data processing operations) in the found data file, and then obtains the execution result.

[0078] For example, the number of subtasks can be the same as the number of first working nodes. The number of subtasks can be determined according to actual needs, and this application embodiment does not specifically limit this.

[0079] 2054. The query component determines the target query result of the query request based on the execution result returned by the at least one first working node.

[0080] In a distributed system, after receiving the execution results returned by at least one first worker node, the query component needs to process them to obtain the final query result. This processing typically includes, but is not limited to, result integration, result filtering, and error handling. Result integration refers to the master node receiving and integrating the results from each worker node. This usually involves sorting, merging, or deduplicating the results from each node according to certain rules to ensure the accuracy and completeness of the final result. Result filtering refers to the master node further filtering and sorting the integrated results if the query has specific filtering conditions or sorting requirements. For example, in a search engine, users may only care about the top N results or sort them according to some relevance; in this case, the master node needs to filter and sort the results. Error handling refers to the fact that in a distributed system, due to network latency, node failures, etc., some worker nodes may return incorrect results or no results. The master node needs to be able to handle these error situations, such as by retrying, using backup nodes, or ignoring erroneous results to ensure the correctness of the final result.

[0081] In distributed systems, data tables are often stored in the form of partitions, especially when dealing with large-scale datasets. Partitioning is a technique that physically divides the data of a table into smaller, more manageable segments. A data file within a partition only records the table records of that partition. Therefore, in practical applications, the query request may include a second query condition regarding the partition column. When the query request includes a first query condition regarding the target column (which is a non-partitioned column) and a second query condition regarding the partition column, the above method may further include the following steps:

[0082] 206. After being invoked by the query component, the management component determines the partition to be queried in the table to be queried based on the second query condition.

[0083] The second query condition includes the specified column value of the partition column.

[0084] For example, the value range of the relevant partition column corresponding to each of the multiple partitions of the table to be queried is obtained; the specified column value of the partition column in the second query condition is compared with the value range of the relevant partition column corresponding to each of the multiple partitions; the partition corresponding to the value range that the column value of the partition column in the query condition falls into is determined as the partition to be queried.

[0085] In this context, the value of the partition column for any record within a partition is within the range of values ​​for the relevant partition column corresponding to that partition.

[0086] It should be noted that the range of values ​​for the partition column corresponding to a partition can include one or more values. When the range of values ​​for the partition column corresponding to a partition includes only one value, the column value of the partition column is the same for any record within that partition.

[0087] In this embodiment, determining the partition to be queried in the table to be queried is equivalent to cutting off the data files in other partitions of the table to be queried, and using at least one second data file in the partition to be queried as the initial set of data files to be read, thus realizing partition cutting.

[0088] Accordingly, in step 203 above, "the management component filters out the first data file of the table to be queried that matches the first query condition based on the statistical information of the target column," can be implemented using the following steps:

[0089] 2031. The management component, based on the target column statistics, filters out the first data file of the table to be queried that matches the first query condition from at least one second data file within the partition to be queried.

[0090] Based on the target column statistics, at least one second data file in the partition to be queried is filtered to obtain the first data file of the table to be queried that matches the first query condition.

[0091] In this embodiment of the application, by partitioning and cropping, the number of data files that need to be filtered in step 203 can be reduced, thereby reducing the computing resources and time consumed in the filtering.

[0092] In one optional implementation, the above method may further include the following steps:

[0093] 207. If the target column statistics information is not obtained from the cache, the management component determines at least one third data file in the multiple data files of the table to be queried that matches the first query condition, based on the storage path of the target column statistics information table of the table to be queried in the file system and the first query condition.

[0094] 208. The management component determines the data file in the intersection of the at least one third data file and the at least one second data file as the first data file.

[0095] In the above 207, the target column statistics information table of the table to be queried is used to store the column statistics information of the table to be queried. It should be noted that each table record in the target column statistics information table of the table to be queried corresponds to a data file of the table to be queried, that is, the table record is used to record the column statistics information of the data file.

[0096] If the target column statistics are not cached in the master node's cache, then the target column statistics cannot be found in the master node's cache.

[0097] If the target column statistics are not found in the cache of the master node, the management component executes steps 207 and 208.

[0098] For example, the management component may perform the following steps: generating a data filtering task for the table to be queried based on the storage path of the target column statistics information table in the file system and the first query condition; determining at least one second working node from the plurality of working nodes; determining at least one second working node from the plurality of working nodes; distributing the data filtering task to the at least one second working node; the data filtering task is used to instruct the at least one second working node to filter at least one third data file that matches the first query condition from the plurality of data files of the table to be queried.

[0099] The data filtering task may include at least one subtask, which may be distributed to at least one second working node; after receiving the subtask, the at least one second working node executes the received subtask and obtains the execution result.

[0100] For example, the number of subtasks can be the same as the number of second working nodes. The number of subtasks can be determined according to actual needs, and this application embodiment does not specifically limit this.

[0101] For example, at least one second working node can be randomly selected from the plurality of working nodes, or the working node with a load rate lower than a preset threshold among the plurality of working nodes can be determined as the second working node.

[0102] For example, the table to be queried includes multiple partitions; the data filtering task may include multiple subtasks, the same number as the number of partitions; each subtask corresponds one-to-one with a partition, and each subtask corresponds one-to-one with a second working node. Each subtask contains the storage path of the target column statistics table of the table to be queried in the file system, the first query condition, and the partition information of the partition corresponding to the subtask. After receiving the subtask, each second working node reads the column statistics of the target column from the target column statistics table in the file system according to the storage path of the target column statistics table of the table to be queried carried in the subtask; based on the column statistics of the target column, it filters at least one data file that matches the first query condition from the data files in the partition corresponding to the subtask; and returns the filename of the at least one filtered data file as the execution result to the master node. That is, the execution result returned by each second working node includes the filename of at least one data file that matches the first query condition in the partition corresponding to the subtask that the second working node is responsible for.

[0103] For example, the management component determines at least one third data file of the table to be queried that matches the first query condition based on the union of the execution results returned by the at least one second working node.

[0104] In step 208 above, the management component determines the data file contained in the intersection of the at least one third data file and the at least one second data file as the first data file.

[0105] By finding the intersection, we can filter out the data files that match the first query condition from at least one second data file within the partition to be queried. In other words, the data files in the intersection are the data files within the partition to be queried that match the first query condition.

[0106] In this embodiment, when the column statistics required for the query are not cached in the cache, they are obtained from the file system, thereby implementing a fallback solution to avoid the inability to accelerate the query based on the column statistics.

[0107] Analysis revealed that data previously queried by a user is highly likely to be accessed again by the same user. Therefore, caching statistics on used columns helps improve the cache hit rate, thereby improving subsequent query efficiency. Specifically, the above method may also include the following steps:

[0108] 209. The management component determines the target column statistics based on the column statistics of the target column.

[0109] The column statistics information of the target column is obtained from the target column statistics information table during the process of determining the at least one third data file.

[0110] In one embodiment, the column statistics of the target column read from the file system are the same for each second worker node. Therefore, only one worker node among at least one second worker node needs to return the column statistics of the target column read from the file system to the master node. To save bandwidth resources, in one feasible solution, the management component can specify a target node from at least one second worker node to return the column statistics of the target column. For example, the management component can include an instruction to return column statistics in one of the above-mentioned subtasks. The second worker node receiving the instruction (i.e., the specified target node) can return the column statistics of the target column while returning the execution result.

[0111] In distributed systems, the column statistics returned by worker nodes are all in Data Frame (DFrame) structure, but the master node cache does not support the storage of DFrames. A DFrame represents two-dimensional tabular data; it is a tabular data structure containing a set of ordered columns, each of which can be of a different value type (numeric, string, or boolean value). Therefore, in this embodiment, the target column statistics in the master node cache are stored in the form of a dataset and DFrame pattern within an in-memory data structure.

[0112] In one optional implementation, the column statistics of the target column can be directly used as the target column statistics. In another optional implementation, the column statistics of the target column may include: column statistics of the target column for each of the multiple partitions of the table to be queried. The column statistics of the target column for each partition refers to the column statistics of the target column for each of at least one data file within that partition.

[0113] In another alternative implementation, column statistics of the target column for each of the at least one second data file are filtered from the column statistics of the target column.

[0114] At least one of the second data files is a data file within the partition to be queried. That is, the column statistics of the target column of the partition to be queried are filtered from the column statistics of the target column to serve as the column statistics of the target column.

[0115] 210. The management component caches the target column statistics in the cache.

[0116] For example, the cache of the master node only caches column statistics for the target column of the partition to be queried.

[0117] In practical applications, analysis revealed that the partitions accessed by users within a given timeframe remain relatively constant. Therefore, during subsequent queries, partition pruning is performed to obtain the partitions the user wants to access. When skipping data within those partitions, only the column statistics of the relevant columns within that partition need to be considered. Thus, to save cache space, only the column statistics of the target columns for the partition being queried can be cached. Furthermore, this approach reduces the risk of Out of Memory (OOM) errors.

[0118] In an optional implementation, the "the management component caches the target column statistics in the cache" in step 210 above can be achieved by the following steps:

[0119] 2101. The management component determines the target column statistics as the value in the key-value pair.

[0120] 2102. The management component generates the key in the key-value pair based on the table information of the table to be queried, the partition information of the target partition, and the column information of the target column.

[0121] 2103. The management component caches the key-value pairs in the cache.

[0122] For example, in step 2101 above, a dataset and a data frame pattern of an in-memory data structure can be generated based on the statistical information of the target column, and the dataset and data frame pattern of the in-memory data structure can be used as values ​​in key-value pairs. The data frame pattern describes the data structure of the data frame, defining the column names and data types of each column. The in-memory data structure can be in array form; this embodiment does not specifically limit its implementation.

[0123] Accordingly, the target column statistics in the master node's cache are stored in the form of a dataset and data frame mode in an in-memory data structure. Step 203 above, where "the management component filters out the first data file of the table to be queried that matches the first query condition based on the target column statistics," can be implemented using the following steps:

[0124] 2031. The management component transforms the dataset of the memory data structure according to the data frame mode to obtain the target data frame.

[0125] 2032. The management component filters out the first data file of the table to be queried that matches the first query condition based on the target data frame.

[0126] For example, the dataset in array form is transformed according to the data frame pattern to obtain the target data frame. The target data frame contains not only the data frame pattern but also each piece of data in the dataset.

[0127] In this embodiment, the master node can quickly filter out data files based on the target data frame.

[0128] In the above 2102, the key includes the table information of the table to be queried, the partition information of the target partition, and the column information of the target column.

[0129] For example, the table information of the table to be queried, the partition information of the target partition, and the column information of the target column can be concatenated to form the key in the key-value pair.

[0130] For example, the information set consisting of the table information of the table to be queried, the partition information of the target partition, and the column information of the target column can be used as the key in the key-value pair.

[0131] In step 2103 above, the generated key-value pairs are stored in the cache of the master node.

[0132] In this embodiment, the column statistics are stored in the form of key-value pairs, which is equivalent to adding an index to the column statistics. Adding an index to the column statistics in the cache helps to improve the efficiency of finding column statistics in the cache.

[0133] In practical applications, step 202 above, "the management component searches for the target column statistics from the cache," can be implemented using the following steps:

[0134] 2021. The management component generates a query key based on the table information of the table to be queried, the partition information of the target partition, and the column information of the target column.

[0135] The query key contains table information of the table to be queried, partition information of the target partition, and column information of the target column.

[0136] In the aforementioned 2021, the method for generating the query key is the same as the method for generating the key of the key-value pair in the cache, and will not be repeated here.

[0137] 2022. The management component obtains the target key-value pair from the cache based on the query key.

[0138] The information set contained in the query key is a subset of the information set contained in the key of the target key-value pair. This subset includes proper subsets and inproper subsets.

[0139] For example, the key of key-value pair A includes: a / d / c1 (table storage path), 1 (partition information of the partition to be queried), z1 (column name), z2 (column name); the key of the query key includes: a / d / c1 (table storage path), 1 (analysis information of the partition to be queried), z1 (column name); the set of column names J1 contained in the key of key-value pair A includes: z1 (column name), z2 (column name); the set of column names J2 contained in the key of the query key includes: z1 (column name); J2 is a proper subset of J1; therefore, key-value pair A can be the target key-value pair.

[0140] In other words, in this embodiment, the query key does not need to be strictly identical to the key in the cache; a cache hit is also considered successful when the information set contained in the query key is a proper subset of the information set contained in the key of the target key-value pair, thus allowing the corresponding column statistics information to be retrieved from the cache. This not only expands the applicability of the cache but also improves the hit rate and reduces the need to reread and recalculate column statistics information.

[0141] 2023. The management component obtains the target column statistics from the values ​​of the target key-value pairs.

[0142] When the information set contained in the query key is a non-proper subset of the information set contained in the key of the target key-value pair, the value of the target key-value pair can be directly used as the target column statistics.

[0143] When the information set contained in the query key is a proper subset of the information set contained in the key of the target key-value pair, it can be determined whether there is column information in the key of the target key-value pair that is not contained in the query key. If so, it means that the value of the target key-value pair not only contains the target column statistics information of the target column, but also the column statistics information of other columns. Therefore, it is necessary to filter out the target column statistics information of the target column from the value of the target key-value pair.

[0144] In practical applications, the data in a table is not static. When a user performs an insert operation on the table...

[0145] When operations such as update or delete occur, the data in the table changes, and therefore, the column statistics of the table may also change. In other words, the column statistics cached in the master node have an expiration date. Filtering data based on outdated column statistics may lead to incorrect query results. When users have high real-time requirements, the following methods can be used to ensure the correctness of the query. Specifically, the step 202 above, "searching for the target column statistics from the cache of the master node," can be achieved using the following steps:

[0146] 2024. The management component obtains the timeline of the current table to be queried.

[0147] The timeline of a table to be queried refers to an ordered sequence of events used to track the history of changes to the table. Specifically, the timeline of a table to be queried consists of a series of timestamps and related operation events (such as writes, updates, and deletes), which are arranged in chronological order.

[0148] 2025. The management component uses a hash algorithm to determine the first hash value of the timeline of the current query table.

[0149] A hash algorithm is used to calculate the first hash value of the current timeline of the table to be queried. The specific implementation of the hash algorithm can be found in existing technologies, and this application does not specifically limit it.

[0150] 2026. The management component obtains the second hash value of the historical timeline of the table to be queried.

[0151] The historical timeline refers to the timeline of the table to be queried when the column statistics information of the table to be queried was last added to the cache of the master node.

[0152] The second hash value is calculated using the hash algorithm described above, based on the timeline of the table to be queried when the column statistics information of the table to be queried was most recently added to the cache of the master node.

[0153] For example, the second hash value can also be stored in the cache of the master node. The master node can then retrieve the second hash value from the cache based on the table information of the table to be queried.

[0154] 2027. When the first hash value is equal to the second hash value, the management component retrieves the target column statistics from the cache.

[0155] If the first hash value is equal to the second hash value, it means that since the column statistics information of the table to be queried was last added to the cache of the master node, the table to be queried has not been involved in write, update, or delete events. Therefore, the cache of the column statistics information of the table to be queried is effective and accurate.

[0156] If the first hash value and the second hash value are not equal, it indicates that since the last time the column statistics information of the table to be queried was added to the cache of the master node, the table to be queried has been involved in write, update, or delete events. Therefore, the cache of the column statistics information of the table to be queried may be invalid. In practical applications, when users have high real-time requirements, in order to avoid the situation where the queried data is not the latest data, the cache of the column statistics information of the table to be queried can be directly considered invalid.

[0157] Optionally, the step 202 above, "the management component searches for the target column statistics from the cache," may further include the following steps:

[0158] 2028. When the first hash value is not equal to the second hash value, the management component deletes the column statistics information of the table to be queried from the cache.

[0159] For example, key-value pairs whose keys contain table information of the table to be queried are deleted from the master node cache.

[0160] In this embodiment of the application, deleting invalid column statistics from the cache can avoid query errors caused by skipping data based on invalid column statistics in subsequent queries.

[0161] Optionally, when the first hash value is not equal to the second hash value, the corresponding column statistics information can be retrieved from the file system based on the table information, partition information, and column information included in the key of the deleted key-value pair, and used as the new value for the key, and the new key-value pair can be cached. Of course, in practical applications, this step can be omitted, and the column statistics information can be retrieved from the file system and cached when a new query request for the table to be queried is triggered.

[0162] Of course, in practical applications, if users do not have high requirements for real-time performance, they do not need to check the timeliness of the cache. That is, for scenarios where the real-time requirements of data are not high, users can choose to continue to use the existing cache for data queries, which can speed up the query process. Furthermore, due to the use of partition pruning technology, even after changes in the timeline, even without invalidated cache, the query results will be accurate if the query does not involve updating the partition.

[0163] To mitigate the risk of OutOfMemoryError (OOM), a maximum memory usage limit can be set for the cache to prevent excessive memory consumption. Considering the challenge of directly obtaining the cache size, a memory usage estimation strategy can be employed. This involves estimating memory usage based on a pre-defined correlation between the number of rows in column statistics and memory usage. When the estimated memory usage exceeds a preset memory threshold (i.e., the maximum memory usage limit), the master node will initiate the clearing of Least Recently Used (LRU) cache entries or cache entries used less frequently than a preset frequency threshold, maintaining memory usage at a reasonable level.

[0164] Figure 3 This is a flowchart illustrating a data query method provided in one embodiment of this application. This data query method is applicable to distributed systems; that is, the execution entity of the method provided in this embodiment is a distributed system. Figure 3 As shown, the method may include the following steps:

[0165] 11. Receive user query requests.

[0166] 12. Parse the query request to obtain the table name, first query condition, and second query condition of the table to be queried.

[0167] Parse the query request to obtain the user information, the table information of the table to be queried, and the query conditions carried in the query request.

[0168] When the query conditions only include the first query condition related to the non-partitioned column (i.e., the target column mentioned above) and do not include the second query condition related to the partitioned column, step 13 can be skipped and step 14 can be executed directly.

[0169] When the query conditions only include the second query condition related to the partition column, and do not include the first query condition related to the non-partition column, and do not involve the non-partition column, only the partition pruning operation needs to be performed on the data file of the table to be queried, without performing the data skipping operation. That is, when the query conditions only involve the partition column and do not involve the non-partition column, after performing the partition pruning (i.e., step 13), step 15 is executed directly.

[0170] When the query conditions include both the first query condition related to the non-partitioned column (i.e., the target column mentioned above) and the second query condition related to the partitioned column, step 13 can be executed first, followed by step 14.

[0171] It should be noted that the tables involved in the embodiments of this application can be database tables or data lake tables, and the embodiments of this application do not specifically limit them.

[0172] 13. Determine the partition to be queried in the table based on the specified column values ​​of the partition column included in the second query condition.

[0173] The second query condition includes a specified column value for the partition column. The process for determining the partition to be queried can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0174] 14. Determine whether the user has enabled the data filtering function.

[0175] User configuration information can be obtained based on the user information carried in the query request. In one embodiment, this configuration information may be manually generated by the user; based on the user's configuration information, it can be determined whether the user has enabled the data filtering function.

[0176] Optionally, the configuration information may include, but is not limited to: indications of whether the user has enabled the data filtering function, indications of whether the user has enabled the column statistics caching function, and indications of whether the user has requirements for data real-time performance.

[0177] If the user has not enabled the data filtering function, proceed to step 15; if the user has enabled the data filtering function, proceed to step 16.

[0178] 15. Execute the query operation of the query request in at least one second data file of the partition to be queried, and obtain the query result.

[0179] For example, data is read from at least one second data file; the data to be queried by the query request is filtered from the read data using query conditions, and then the query results are obtained.

[0180] It should be noted that the second data file of the partition to be queried only stores records within that partition and does not store records from other partitions.

[0181] 16. Determine whether the user has enabled the column statistics information caching function.

[0182] Optionally, the system can determine whether a user should enable the column statistics caching function based on their configuration information.

[0183] If the user has enabled the statistics caching function, proceed to step 17; if the user has not enabled the column statistics caching function, proceed to step 26.

[0184] 17. Determine if the user has requirements for data real-time performance.

[0185] Optionally, the system can determine whether the user has requirements for data real-time performance based on the user's configuration information.

[0186] If the user requires real-time data processing, proceed to step 18; if the user does not require real-time data processing, skip step 18 and proceed directly to step 19.

[0187] It should be noted that when users require real-time data, it is necessary to ensure the timeliness and accuracy of the cache; otherwise, the final queried data may not be the latest data.

[0188] 18. Evaluate the effectiveness of caching for the table to be queried.

[0189] Optionally, obtain the current timeline of the table to be queried. The timeline of the table to be queried refers to an ordered sequence of events used to track the history of changes in the table. Specifically, the timeline of the table to be queried consists of a series of timestamps and related operation events (such as write, update, and delete), which are arranged in chronological order. Calculate the first hash value of the current timeline of the table to be queried using a hash algorithm. Obtain the second hash value recorded when the column statistics of the table to be queried were cached most recently. The second hash value is calculated using the hash algorithm described above on the timeline of the table to be queried when the column statistics were cached most recently. If the first hash value and the second hash value are equal, it means that since the last time the column statistics of the table to be queried were cached, the table to be queried has not been involved in write, update, or delete events. Therefore, the cache of the column statistics of the table to be queried is valid and accurate. If the first hash value and the second hash value are not equal, it means that since the last time the column statistics of the table to be queried were cached, the table to be queried has been involved in write, update, or delete events. Therefore, the cache of the column statistics of the table to be queried may be invalid. In practical applications, when users have high requirements for real-time performance, in order to avoid the situation where the queried data is not the latest data, the cache of the column statistics information of the table to be queried can be directly considered invalid.

[0190] If the cache validity evaluation result shows that it is valid, proceed to step 19; if the cache validity evaluation result shows that it is invalid, proceed to step 25, and then proceed to step 26.

[0191] Alternatively, if the cache validity evaluation result shows that it is invalid, step 26 can be executed directly, while step 25 can be executed asynchronously.

[0192] 19. Construct the query key based on the storage path of the table to be queried, the partition information of the partition to be queried, and the column names of the non-partition columns in the first query condition.

[0193] The first query condition includes the column name of the non-partitioned column.

[0194] Optionally, the partition information can be a partition identifier, such as a partition number.

[0195] Optionally, the query key includes: the storage path of the table to be queried, the partition information of the partition to be queried, and the column name of the non-partition column in the query conditions.

[0196] The storage path of the table to be queried, the partition information of the partition to be queried, and the column names of the non-partition columns in the query conditions can be combined in a preset way to form the query key. The preset combination method can be set according to actual needs, and this application embodiment does not specifically limit it.

[0197] 20. Match among multiple key-value pairs in the cache based on the query key.

[0198] Optionally, the key of each key-value pair includes: the table's storage path, partition information, and column name. The key value of the key-value pair includes: column statistics for that column in at least one data file; at least one data file refers to a data file within that partition of the table. The column statistics for that column in each data file refer to information obtained by statistically analyzing the column data for that column in that data file.

[0199] In distributed systems, column statistics retrieved from the file system are typically in DataFrame format. When caching column statistics retrieved from the file system, the DataFrame format can be converted to in-memory data structure column statistics and the DataFrame schema. These in-memory data structure column statistics and the DataFrame schema are then stored as key-value pairs. Subsequently, the DataFrame format column statistics can be reconstructed based on the DataFrame schema and the in-memory data structure column statistics.

[0200] The data frame mode describes the data structure of the data frame, defining the column names and data types of each column.

[0201] The matching process for target key-value pairs can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0202] If a target key-value pair is matched, proceed to step 21; otherwise, proceed to step 26.

[0203] 21. Extract column statistics of the relevant non-partitioned columns of at least one second data file of the partition to be queried from the key values ​​of the target key-value pair.

[0204] It should be noted that the non-partitioned columns in step 21 refer to the non-partitioned columns involved in the query conditions.

[0205] Optionally, the key value of the target key value pair contains only column statistics of the relevant non-partitioned columns of at least one second data file of the partition to be queried. Therefore, the key value of the target key value pair can be directly used as the column statistics of the relevant non-partitioned columns of at least one second data file of the partition to be queried.

[0206] 22. Construct a data frame based on the column statistics of the relevant non-partitioned columns of at least one second data file of the partition to be queried.

[0207] Optionally, the column statistics of the memory data structure in the key value of the target key value pair can be converted to obtain the column statistics of the data frame structure.

[0208] It should be noted that when the data query system is a non-distributed system, the entire solution does not involve the process of converting column statistics information from a data frame structure to an in-memory data structure, nor does it involve the data frame construction process.

[0209] 23. Based on the data frame and the first query condition, perform data skipping processing on at least one second data file of the partition to be queried to obtain at least one first data file.

[0210] Optionally, based on the column statistics of the data frame structure, data filtering is performed on at least one data file of the partition to be queried to obtain at least one first data file that has not been filtered out.

[0211] Obtain column statistics information of the non-partitioned columns of the second data file from the data frame; based on the column statistics information of the non-partitioned columns of the second data file and the first query condition, determine whether to filter out the second data file. Specifically, based on the column statistics information of the non-partitioned columns of the second data file, determine whether there are records in the second data file that meet the first query condition; when it is determined that there are records in the second data file that meet the first query condition, determine that the second data file needs to be retained (i.e., not filtered); when it is determined that there are no records in the second data file that meet the first query condition, determine that the second data file needs to be filtered out.

[0212] For example, the first query condition is: 'a' is greater than or equal to 50, and 'a' is a non-partitioned column; the column statistics for 'a' in the second data file are: minimum value 0, maximum value 40. Therefore, it can be determined that the second data file needs to be filtered out.

[0213] For example, the first query condition is: a is greater than or equal to 50, and a is a non-partitioned column; the column statistics of a in the second data file are: the minimum value is 40 and the maximum value is 80. Therefore, it can be determined that the second data file needs to be retained.

[0214] For example, the first query condition is: a = null (empty value), where a is a non-partitioned column; the column statistics for a in the second data file are: the number of empty values ​​is 0; therefore, it can be determined that the second data file needs to be filtered out.

[0215] 24. Execute the query operation of the query request in the at least one first data file to obtain the query result.

[0216] For example, data is read from at least one first data file; the data to be queried by the query request is filtered from the read data using query conditions, and then the query results are obtained.

[0217] 25. Delete column statistics related to the table being queried from the cache.

[0218] For example, the delete key contains a key-value pair with the storage path of the table to be queried.

[0219] 26. Read column statistics information of the table to be queried from the file system.

[0220] For example, column statistics for each of the multiple data files of the table to be queried can be read from the file system.

[0221] 27. Obtain column statistics of at least one non-partitioned column from the column statistics of the table to be queried.

[0222] Specifically, the filenames of at least one second data file of the partition to be queried are obtained; based on the filenames of at least one second data file, the column statistics of each of the at least one second data file are determined from the column statistics of the table to be queried; the column statistics of the non-partitioned columns are determined from the column statistics of each of the at least one second data file, and used as the column statistics of the non-partitioned columns in the at least one second data file of the partition to be queried.

[0223] Step 27 is equivalent to partitioning and pruning the column statistics of the table to be queried, and obtaining the column statistics of non-partitioned columns in at least one second data file of the partition to be queried.

[0224] Optionally, column statistics can be stored separately for different partitions of the table in the file system. That is, different partitions of the table correspond to different column statistics tables, and each column statistics table only stores the column statistics of the partition corresponding to that column statistics file. Thus, in step 26 above, the column statistics of the partition to be queried can be read from the column statistics table corresponding to the partition to be queried in the file system based on the partition information of the partition to be queried. Therefore, in step 27, there is no need to perform partition pruning on the read column statistics. In this embodiment, when a query request arrives, since the physical storage of the data has already been naturally partitioned, even in the event of a cache miss, the system can more quickly locate and access the required data (i.e., only the data file of the column statistics table corresponding to the partition to be queried needs to be read), thereby further improving the efficiency of the entire query process. This approach can bring significant performance improvements when processing large datasets with many partitions or scattered data.

[0225] 28. Use the column statistics of the relevant non-partitioned columns of at least one second data file as the key value of the query key, and store the query key and the key value of the query key in the cache.

[0226] Optionally, the column statistics of the non-partitioned columns of at least one data file of the partition to be queried are transformed to obtain the column statistics and data frame pattern of the in-memory data structure.

[0227] 29. Record the hash value of the timeline of the table to be queried at this time.

[0228] Step 29 can be executed when step 28 has finished.

[0229] 30. Based on the column statistics of the table to be queried and the first query condition, perform data skipping processing on multiple data files of the table to be queried to obtain at least one third data file.

[0230] At least one third data file refers to the data file of the table to be queried that matches the first query condition. The specific data skipping process can be found in the corresponding content of the above embodiments, and will not be repeated here.

[0231] 31. The intersection of at least one second data file and at least one third data file shall be used as at least one first data file.

[0232] After step 31 is completed, step 24 is executed to obtain the query results.

[0233] The technical solution provided in this application optimizes the user experience. Especially when processing large datasets, this solution can provide a significant speed improvement and resource saving. When the cache is hit, the query performance is improved by several to tens of times compared to queries without caching capabilities.

[0234] Optionally, after step 26, steps 30, 31, and 24 can be executed first, followed by steps 27, 28, and 29.

[0235] in, Figure 3 Steps 11 and 12 are executed by the query component, steps 15 and 24 are completed by the query component and the worker node in collaboration, and steps 13, 14, 16-23 and steps 25 to 31 are all executed by the management component.

[0236] It should be noted that any steps in the method provided in this application that are not described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the method provided in this application may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0237] The terminals mentioned in the above embodiments can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories, peripherals, or any combination thereof for these devices. It is also foreseeable that the terminal devices can support any type of user-facing interface device (e.g., wearable devices).

[0238] The various nodes mentioned in the embodiments of this application specifically refer to one or more servers. A server can also be a physical server or a virtual server, etc. A server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0239] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Figure 4As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Electrically Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0240] The memory 1101 is used to store programs;

[0241] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the methods provided in the above-described method embodiments.

[0242] Furthermore, such as Figure 4 As shown, the electronic device also includes: communication component 1103, power supply component 1104, and other components. Figure 4 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 4 The components shown.

[0243] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0244] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0245] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0246] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data query method for a distributed system, characterized in that, The distributed system includes: a master node, the master node including a cache for storing column statistics, and a query component for querying data and a management component for managing the data lake running on the master node; the method includes: When the query component receives a query request for a table to be queried in the data lake, it calls the management component according to the query request, wherein the query request includes a first query condition related to the target column; After being invoked by the query component, the management component retrieves the target column statistics associated with the target column from the cache, and filters out the first data file of the table to be queried that matches the first query condition based on the target column statistics. The management component returns the file information of the first data file to the query component; The query component determines the target query result of the query request based on the file information of the first data file.

2. The method according to claim 1, characterized in that, The query request also includes a second query condition regarding the partition column; the method further includes: After being invoked by the query component, the management component determines the partition of the table to be queried based on the second query condition. The management component filters out the first data file of the table to be queried that matches the first query condition based on the statistical information of the target column, including: The management component, based on the target column statistics, filters out the first data file of the table to be queried that matches the first query condition from at least one second data file within the partition to be queried.

3. The method according to claim 2, characterized in that, Also includes: If the target column statistics information is not obtained from the cache, the management component determines at least one third data file that matches the first query condition among multiple data files of the table to be queried, based on the storage path of the target column statistics information table of the table to be queried in the file system and the first query condition. The target column statistics information table of the table to be queried is used to store the column statistics information of the table to be queried. The management component identifies the data file in the intersection of the at least one third data file and the at least one second data file as the first data file.

4. The method according to claim 3, characterized in that, Also includes: The management component determines the target column statistics based on the column statistics of the target column; The column statistics of the target column are obtained from the target column statistics table during the process of determining the at least one third data file; The management component caches the target column statistics in the cache.

5. The method according to claim 4, characterized in that, The management component determines the target column statistics based on the target column's column statistics, including: The management component filters out the column statistics of the target column from the column statistics of the target column for each of the at least one second data file; The management component determines the column statistics of the target column for each of the at least one second data file as the target column statistics.

6. The method according to claim 5, characterized in that, The management component caches the target column statistics in the cache, including: The management component determines the target column statistics as the values ​​in the key-value pairs; The management component generates a key in the key-value pair based on the table information of the table to be queried, the partition information of the target partition, and the column information of the target column. The key contains the table information of the table to be queried, the partition information of the target partition, and the column information of the target column. The management component caches the key-value pairs in the cache.

7. The method according to claim 6, characterized in that, The management component obtains the target column statistics from the cache, including: The management component generates a query key based on the table information of the table to be queried, the partition information of the target partition, and the column information of the target column. The query key contains the table information of the table to be queried, the partition information of the target partition, and the column information of the target column. The management component matches the target key-value pair in the cache based on the query key. The information set contained in the query key is a subset of the information set contained in the key of the target key-value pair. The subset includes proper subsets and non-proper subsets. The management component obtains the target column statistics from the values ​​of the target key-value pairs.

8. The method according to any one of claims 1 to 7, characterized in that, The management component obtains the target column statistics from the cache, including: The management component obtains the timeline of the current table to be queried; The management component uses a hash algorithm to determine the first hash value of the timeline of the current table to be queried. The management component obtains the second hash value of the historical timeline of the table to be queried, where the historical timeline is the timeline of the table to be queried when the column statistics information of the table to be queried was last added to the cache of the master node. When the first hash value is equal to the second hash value, the management component retrieves the target column statistics from the cache.

9. The method according to claim 8, characterized in that, Also includes: When the first hash value is not equal to the second hash value, the management component deletes the column statistics information of the table to be queried from the cache.

10. The method according to any one of claims 1 to 7, characterized in that, In the cache, the target column statistics are stored in the form of a dataset and data frame mode in an in-memory data structure; The management component filters out the first data file of the table to be queried that matches the first query condition based on the statistical information of the target column, including: The management component transforms the dataset of the memory data structure according to the data frame pattern to obtain the target data frame; Based on the target data frame, the first data file matching the first query condition of the table to be queried is selected.

11. The method according to any one of claims 1 to 7, characterized in that, The distributed system also includes multiple worker nodes; The query component determines the target query result of the query request based on the file information of the first data file, including: The query component determines at least one first working node from the plurality of working nodes; The query component generates the query task for the query request based on the file information of the first data file; The query component distributes the query task to the at least one first working node; The query component determines the target query result of the query request based on the execution result returned by the at least one first working node.

12. A distributed system, characterized in that, include: The master node includes a cache for storing column statistics, and runs a query component for retrieving data and a management component for managing the data lake; wherein, The query component is used to invoke the management component according to the query request when it receives a query request for a table to be queried in the data lake, wherein the query request includes a first query condition related to the target column; The management component is used to retrieve target column statistics associated with the target column from the cache after being invoked by the query component, and to filter out the first data file of the table to be queried that matches the first query condition based on the target column statistics. The management component is used to return file information of the first data file to the query component; The query component is used to determine the target query result of the query request based on the file information of the first data file.

13. The system according to claim 12, characterized in that, Also includes: Multiple working nodes; The query component is specifically used to: determine at least one first working node from the plurality of working nodes; The query component generates the query task for the query request based on the file information of the first data file; The query component distributes the query task to the at least one first working node; The query component determines the target query result of the query request based on the execution result returned by the at least one first working node.

14. An electronic device, characterized in that, include: Memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 11.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 11.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 11.

Images