Data processing

Abstract

The present disclosure provides a data processing method and apparatus, a program product, a device, and a storage medium. The method comprises: controlling a query engine to push down a filter operator having monotonicity to a storage engine; controlling the storage engine to read data distribution meta-information of at least one data block within a query range, and on the basis of the data distribution meta-information of the at least one data block and the filter operator, filtering the at least one data block to obtain a target data block, wherein the target data block is a data block that is not filtered out by the filter operator; and controlling the storage engine to project the target data block to a memory, and controlling the query engine to execute the filter operator on the target data block to obtain target data, wherein the target data is data that is not filtered out by the filter operator.

Description

Data processing Technical Field

[0001] This disclosure relates to the field of database technology, and more particularly to data processing. Background Technology

[0002] In today's era of rapid development in the internet and information technology, the generation of data is exploding, thus placing increasingly higher demands on databases and their management. When a database executes an SQL (Structured Query Language) statement, it first formulates a data query plan, i.e., the execution plan of the SQL statement, and then executes the various operators within the data query plan to obtain the data query results corresponding to the SQL statement.

[0003] In related technologies, when a database executes a data query plan, the storage engine projects the data within the query range to the query engine, which then executes the various operators within the data query plan to obtain the data query results corresponding to the SQL statement; this method has low data query efficiency. Summary of the Invention

[0004] In view of the above, one or more embodiments of this disclosure provide a data processing method and apparatus, a program product, an electronic device, and a storage medium.

[0005] According to a first aspect of one or more embodiments of this disclosure, a data processing method is proposed, the method comprising: controlling a query engine to push down a filtering operator having monotonicity to a storage engine; controlling the storage engine to read data distribution metadata of at least one data block within a query range, and filtering the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, wherein the target data block is a data block not filtered out by the filtering operator; controlling the storage engine to project the target data block into memory, and controlling the query engine to execute the filtering operator on the target data block to obtain target data, wherein the target data is data not filtered out by the filtering operator.

[0006] In one or more embodiments of this disclosure, the control storage engine reads data distribution metadata of at least one data block within a query range, including: reading the data distribution metadata of the at least one data block from the aggregation information of an intermediate index layer within the query range, wherein the intermediate index layer is a tree index of data blocks within the query range, the intermediate index layer includes at least one layer of index blocks, and the aggregation information of the index blocks is used to describe the data distribution metadata of the next layer of index blocks or data blocks.

[0007] In one or more embodiments of this disclosure, the control storage engine reads data distribution metadata of at least one data block within a query range, including: reading the data distribution metadata of the data block from the encoding information of the data block.

[0008] In one or more embodiments of this disclosure, the data distribution metadata includes at least one of the following: maximum value, minimum value, and whether null values ​​exist.

[0009] In one or more embodiments of this disclosure, the step of filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block includes: determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block satisfying the filtering condition; determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block satisfying the filtering condition; determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block satisfying the filtering condition; and determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block satisfying the filtering condition.

[0010] In one or more embodiments of this disclosure, the step of filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block includes: in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, determining the data block as a target data block and marking it as a pure data block; in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, determining the data block as a target data block and marking it as a pure data block. If the minimum value of a block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically increasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the minimum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically decreasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the maximum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block.

[0011] In one or more embodiments of this disclosure, the step of filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block includes: in response to the filtering operator being monotonically increasing or monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition, determining the data block as a target data block and marking it as a pure data block.

[0012] In one or more embodiments of this disclosure, the step of filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block includes: determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being less than the preset threshold; determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being less than the preset threshold; determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being greater than the preset threshold; and determining that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the maximum value of the data block being greater than the preset threshold.

[0013] In one or more embodiments of this disclosure, controlling the storage engine to project a target data block to the query engine and controlling the query engine to execute the filtering operator on the target data block to obtain target data includes: in response to the target data block being marked as a pure data block, adding data within the target data block to the target data; in response to the target data block not being marked as a pure data block, controlling the query engine to execute the filtering operator on the target data block and adding the obtained data result to the target data.

[0014] In one or more embodiments of this disclosure, the method further includes: generating a data query plan based on an SQL statement to be executed, wherein the data query plan includes at least one filtering operator; and controlling the query engine to push down the filtering operator with monotonicity to the storage engine, including: controlling the query engine to push down the filtering operator with monotonicity in the data query plan to the storage engine.

[0015] According to a first aspect of one or more embodiments of this disclosure, a data processing apparatus is provided, the apparatus comprising: a push-down module for controlling a query engine to push down a filtering operator having monotonicity to a storage engine; a first filtering module for controlling the storage engine to read data distribution metadata of at least one data block within a query range, and filtering the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, wherein the target data block is a data block not filtered out by the filtering operator; and a second filtering module for controlling the storage engine to project the target data block to the query engine, and controlling the query engine to execute the filtering operator on the target data block to obtain target data, wherein the target data is data not filtered out by the filtering operator.

[0016] In one or more embodiments of this disclosure, when the first filtering module controls the storage engine to read the data distribution metadata of at least one data block within a query range, it is configured to: read the data distribution metadata of the at least one data block from the aggregation information of the intermediate index layer within the query range, wherein the intermediate index layer is a tree index of the data blocks within the query range, the intermediate index layer includes at least one layer of index blocks, and the aggregation information of the index blocks is used to describe the data distribution metadata of the next layer of index blocks or data blocks.

[0017] In one or more embodiments of this disclosure, when the first filtering module controls the storage engine to read the data distribution metadata of at least one data block within the query range, it is configured to: read the data distribution metadata of the data block from the encoding information of the data block.

[0018] In one or more embodiments of this disclosure, the data distribution metadata includes at least one of the following: maximum value, minimum value, and whether null values ​​exist.

[0019] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block satisfying the filtering condition; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block satisfying the filtering condition; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block satisfying the filtering condition; and determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block satisfying the filtering condition.

[0020] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, be configured to: determine the data block as a target data block and mark it as a pure data block in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition; and in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block does not satisfy the filtering condition. If the minimum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically increasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the minimum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically decreasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the maximum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block.

[0021] In one or more embodiments of this disclosure, when the first filtering module is used to filter the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, it is used to: determine the data block as a target data block and mark it as a pure data block in response to the filtering operator being monotonically increasing or monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition.

[0022] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being less than the preset threshold; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being less than the preset threshold; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being greater than the preset threshold; and determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the maximum value of the data block being greater than the preset threshold.

[0023] In one or more embodiments of this disclosure, the second filtering module is used to control the storage engine to project a target data block to the query engine, and to control the query engine to execute the filtering operator on the target data block to obtain target data. When the target data is obtained, the second module is used to: add the data in the target data block to the target data in response to the target data block being marked as a pure data block; and control the query engine to execute the filtering operator on the target data block in response to the target data block not being marked as a pure data block, and add the obtained data result to the target data.

[0024] In one or more embodiments of this disclosure, the apparatus further includes a plan generation module for: generating a data query plan based on an SQL statement to be executed, wherein the data query plan includes at least one filtering operator; the push-down module is configured to: control the query engine to push down the filtering operator with monotonicity in the data query plan to the storage engine.

[0025] According to a third aspect of one or more embodiments of the present disclosure, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0026] According to a fourth aspect of one or more embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the method as described in the first aspect.

[0027] According to a fifth aspect of one or more embodiments of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0028] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: In the data processing method provided by the embodiments of this disclosure, the query engine is controlled to push down a filtering operator with monotonicity to the storage engine, thereby controlling the storage engine to read the data distribution metadata of at least one data block within the query range, and filtering the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block. Finally, the storage engine is controlled to project the target data block to the query engine, and the query engine is controlled to execute the filtering operator on the target data block to obtain the target data. Since the storage engine can read the data distribution metadata, it can pre-filter the data block based on the monotonicity of the filtering operator and the distribution metadata of the data block, such as maximum and minimum values, and only project a portion of the data within the query range to the query engine for calculation, thereby increasing the efficiency of data filtering, reducing the amount of projected data and the computational load of the query engine, and improving the efficiency of data query; in particular, the storage engine can measure whether a data block needs to be filtered as a whole through the distribution metadata, avoiding the operation of traversing every data when the query engine calculates, and its efficiency is improved by orders of magnitude. Attached Figure Description

[0029] Figure 1 is a flowchart of a data processing method provided in an exemplary embodiment.

[0030] Figure 2 is a schematic diagram of an intermediate layer index provided in an exemplary embodiment.

[0031] Figure 3 is a flowchart of a data processing method provided by another exemplary embodiment.

[0032] Figure 4 is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0033] Figure 5 is a block diagram of a data processing apparatus provided in an exemplary embodiment. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this disclosure as detailed in the appended claims.

[0035] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this disclosure in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this disclosure. Furthermore, a single step described in this disclosure may be broken down into multiple steps in other embodiments; and multiple steps described in this disclosure may be combined into a single step in other embodiments.

[0036] First, some of the concepts involved in this disclosure will be explained.

[0037] Monotonicity describes a consistent trend of change in a function or sequence as the input increases or decreases. For example, monotonically increasing means that the calculated result of the function increases as the input increases and decreases as the input decreases; monotonically decreasing means that the calculated result of the function decreases as the input increases and increases as the input decreases.

[0038] Computation pushdown: In a query, some computations that are the responsibility of the query engine are pushed down to the storage engine for execution.

[0039] Filtering operators: In databases, filtering operators are operations that perform data filtering tasks. For example, when an SQL statement with a WHERE clause is executed, a filtering operator is generated in the query plan to filter out data rows that do not meet the conditions.

[0040] In today's era of rapid development in the internet and information technology, the generation of data is exploding, thus placing increasingly higher demands on databases and their management. When a database executes an SQL (Structured Query Language) statement, it first formulates a data query plan, i.e., the execution plan of the SQL statement, and then executes the various operators within the data query plan to obtain the data query results corresponding to the SQL statement.

[0041] In related technologies, when a database executes a data query plan, the storage engine projects the data within the query range to the query engine, which then executes the various operators within the data query plan to obtain the data query results corresponding to the SQL statement. This method has relatively low data query efficiency. For example, if the storage engine projects all the data within the query range to the query engine, the amount of data transferred is large, and the query engine needs to perform data query calculations on all the data within the query range, which leads to low data query efficiency.

[0042] Based on this, at least one embodiment of this disclosure provides a data processing method that can push down a filtering operator with monotonicity to the storage engine, so that the storage engine controls the monotonicity of the filtering operator and the distribution metadata of each data block within the query range, such as maximum and minimum values, to pre-filter the data blocks, and project the filtered data into memory, so that the query engine executes the filtering operator on the projected data to obtain the filtering result of the filtering operator.

[0043] Please refer to Figure 1, which exemplarily illustrates the flow of a data processing method, including steps S101 to S103.

[0044] In step S101, the control query engine pushes down the filtering operator with monotonicity to the storage engine.

[0045] For example, a data query plan can be generated in advance based on the SQL statement to be executed, wherein the data query plan contains at least one filtering operator. In this step, the query engine can determine whether each filtering operator in the data query plan has monotonicity in order to filter out the filtering operators with monotonicity; that is, this step can control the query engine to push down the filtering operators with monotonicity in the data query plan to the storage engine.

[0046] In step S102, the control storage engine reads the data distribution metadata of at least one data block within the query range, and filters the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain the target data block, wherein the target data block is the data block that was not filtered out by the filtering operator.

[0047] The distribution metadata of the data block may include at least one of the following: the maximum value of the data within the data block, the minimum value of the data within the data block, and whether there are null values ​​in the data within the data block.

[0048] This step can be performed to read the data distribution metadata of the data block using any of the optional examples below. Optional Example 1

[0049] Data on the disk is organized into multiple data blocks, each of roughly the same size. A data block is the basic unit of data projected into memory. An intermediate index layer is also built on top of these data blocks. This intermediate index layer has a tree structure. A row of data in an upper-level index block is an index to the next level of index block or data block. Each row of data in an index block also records aggregation information for the next level of index block or data block. This aggregation information at least describes the data distribution metadata of the next level of index block or data block. For example, as shown in Figure 2, the intermediate index layer contains three levels of index blocks: level 0, level 1, and level 2. The aggregation information in the upper-level index blocks records the maximum and minimum values ​​of the lower-level index blocks.

[0050] Therefore, in this step, the data distribution metadata of the at least one data block can be read from the aggregation information of the intermediate index layer within the query range. The intermediate index layer is a tree index of the data blocks within the query range, and the intermediate index layer includes at least one layer of index blocks. The aggregation information of the index blocks is used to describe the data distribution metadata of the next layer of index blocks or data blocks. Optional Example 2

[0051] The data block is compressed and encoded according to the characteristics of the stored data, such as dictionary encoding; if the encoding format is ordered, the maximum and minimum values ​​inside the data block can be quickly read from the encoding information.

[0052] Therefore, in this step, the data distribution metadata of the data block can be read from the encoding information of the data block.

[0053] The two optional examples above can be combined. For example, optional example 1 can be used to read the data distribution metadata of the data block in advance, and optional example 2 can be used to read the data distribution metadata of the data block if the data on the disk does not have an intermediate layer index.

[0054] The filtering operator can include a calculation function and a comparison operator. The calculation function is used to perform calculations on the input to obtain the result, and the comparison operator is used to compare the calculation result with the filtering conditions to determine whether the comparison result meets the filtering conditions. For example, when using a filtering operator to filter certain data, the data is used as the input of the filtering operator. If the calculation result of the calculation function meets the filtering conditions, the data is filtered out; otherwise, if the calculation result of the calculation function does not meet the filtering conditions, the data is retained.

[0055] Assuming the filtering operator is filter[x], where x is the input, filter[x] returns true, meaning the input is filtered, and filter[x] returns false, meaning the input is not filtered, i.e., it is retained.

[0056] The filter[x] can be expanded to func(x)op Y, where x is the input, func is the function to evaluate x, op is the comparison operator, and Y is the filtering condition.

[0057] The target data block can be a pure data block from which all data needs to be retained, or a non-pure data block from which some data needs to be retained. In this step, when filtering the at least one data block based on its data distribution metadata and the filtering operator, the main purpose is to identify the non-target data blocks from which all data needs to be filtered, thus allowing the remaining data blocks to be used as target data blocks. Optionally, this step can determine pure and non-pure data blocks within the target data block.

[0058] In some embodiments of this disclosure, if the filtering condition is greater than or less than a preset threshold, then this step can be performed according to any of the following examples.

[0059] For example, this step can determine a data block as a non-target data block in any of the following situations, i.e., the data block meets the filtering conditions and needs to be filtered out entirely. Situation 1

[0060] In response to the monotonically increasing filtering operator, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold Y, and the minimum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition.

[0061] In this case, filter[minimum] = true, meaning func(minimum) > Y; since the filter operator is monotonically increasing, the calculation results of other values ​​within this data block are all greater than func(minimum), i.e., all greater than Y, thus satisfying the filtering condition. Case 2

[0062] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition.

[0063] In this case, filter[maximum] = true, meaning func(maximum) > Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​within this data block are all greater than func(minimum), i.e., all greater than Y, thus satisfying the filtering condition. Case 3

[0064] In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition.

[0065] In this case, filter[maximum] = true, meaning func(maximum) < Y; since the filtering operator is monotonically increasing, the calculation results of other values ​​within this data block are all less than func(maximum), i.e., all less than Y, thus satisfying the filtering condition. Case 4

[0066] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition.

[0067] In this case, filter[minimum] = true, that is, func(minimum) < Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​in this data block are all less than func(minimum), that is, all less than Y, thus satisfying the filtering condition.

[0068] It should be understood that, in the above cases, adjusting the filtering condition for values ​​greater than the preset threshold to a filtering condition greater than or equal to the preset threshold can also be used to perform data block filtering in the manner described in the examples above. Similarly, adjusting the filtering condition for values ​​less than the preset threshold to a filtering condition less than or equal to the preset threshold can also be used to perform data block filtering in the manner described in the examples above.

[0069] For example, this step can determine a data block as a pure data block within the target data block in any of the following situations, i.e., all data within the data block must be retained. Situation 5

[0070] In response to the monotonically increasing filtering operator, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0071] In this case, filter[maximum] = false, meaning func(maximum) ≤ Y; since the filtering operator is monotonically increasing, the calculation results of other values ​​within this data block are all less than func(maximum), i.e., all less than Y, and the filtering condition is not met; therefore, the data block is a pure data block in the target data block. Case 6

[0072] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0073] In this case, filter[minimum] = false, meaning func(minimum) ≤ Y. Since the filtering operator is monotonically decreasing, the calculation results of other values ​​within this data block are all less than func(minimum), i.e., all less than Y, thus failing to meet the filtering condition; therefore, the data block is a pure data block within the target data block. Case 7

[0074] In response to the monotonically increasing filtering operator, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0075] In this case, filter[minimum] = false, meaning func(minimum) ≥ Y. Since the filtering operator is monotonically increasing, the calculation results of other values ​​within this data block are all greater than func(maximum), i.e., all greater than Y, thus not satisfying the filtering condition; therefore, the data block is a pure data block within the target data block. Case 8

[0076] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0077] In this case, filter[maximum] = false, that is, func(maximum) ≥ Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​in this data block are all greater than func(maximum), that is, all greater than Y, and the filtering condition is not met; then the data block is a pure data block in the target data block.

[0078] It should be understood that, in the above cases, adjusting the filtering condition for values ​​greater than the preset threshold to a filtering condition greater than or equal to the preset threshold can also be used to perform data block filtering in the manner described in the examples above. Similarly, adjusting the filtering condition for values ​​less than the preset threshold to a filtering condition less than or equal to the preset threshold can also be used to perform data block filtering in the manner described in the examples above.

[0079] It should be understood that the above-mentioned cases must be executed if there are no null values ​​in the data block. That is, only when there are no null values ​​in a data block can the above-mentioned cases be used to determine that the data block is a pure data block in the target data block.

[0080] In some embodiments of this disclosure, if the filtering condition is not equal to a preset threshold (i.e., the retention condition is equal to a preset threshold), then this step can be performed according to any of the following examples.

[0081] For example, this step can determine a data block as a non-target data block in any of the following situations, i.e., the data block meets the filtering conditions and needs to be filtered out entirely. Situation 9

[0082] In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being less than the preset threshold, it is determined that the data block satisfies the filtering condition.

[0083] In this case, filter[maximum] = true, meaning func(maximum) < Y; since the filter operator is monotonically increasing, the calculated results of other values ​​within this data block are all less than func(maximum), that is, all less than Y, and not equal to Y, thus satisfying the filtering condition. Case 10

[0084] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the minimum value of the data block being less than the preset threshold, it is determined that the data block satisfies the filtering condition.

[0085] In this case, filter[minimum] = true, meaning func(minimum) < Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​within this data block are all less than func(minimum), that is, all less than Y, and not equal to Y, thus satisfying the filtering condition. Case 11

[0086] In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the minimum value of the data block being greater than the preset threshold, it is determined that the data block satisfies the filtering condition.

[0087] In this case, filter[minimum] = true, meaning func(minimum) > Y. Since the filter operator is monotonically increasing, the calculation results of other values ​​within this data block are all greater than func(minimum), i.e., all greater than Y, and not equal to Y, thus satisfying the filtering condition. Case 12

[0088] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being greater than the preset threshold, it is determined that the data block satisfies the filtering condition.

[0089] In this case, filter[maximum] = true, that is, func(maximum) > Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​in this data block are all greater than func(maximum), that is, all greater than Y, not equal to Y, thus satisfying the filtering condition.

[0090] For example, this step can determine a data block as a pure data block within the target data block in any of the following situations, i.e., all data within the data block must be retained. Situation 13

[0091] In response to the monotonically increasing filtering operator, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0092] In this case, filter[maximum] = false and filter[minimum] = false, that is, func(maximum) = Y and func(minimum) = Y; since the filtering operator is monotonically increasing, the calculation results of other values ​​in this data block are all equal to Y, which does not satisfy the filtering condition; therefore, the data block is a pure data block in the target data block.

[0093] In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

[0094] In this case, filter[maximum] = false and filter[minimum] = false, that is, func(maximum) = Y and func(minimum) = Y; since the filtering operator is monotonically decreasing, the calculation results of other values ​​in this data block are all equal to Y, which does not satisfy the filtering condition; therefore, the data block is a pure data block in the target data block.

[0095] It should be understood that the above-mentioned cases must be executed if there are no null values ​​in the data block. That is, only when there are no null values ​​in a data block can the above-mentioned cases be used to determine that the data block is a pure data block in the target data block.

[0096] Combining the various scenarios in the two embodiments above, the data block filtering rules shown in the table below can be obtained. The storage engine can perform fast filtering according to the rules shown in this table. Table 1: Data Block Filtering Rules Table

[0097] Among them, the filtering operators of type one are either monotonically increasing, with the filtering condition being that the calculated result is greater than the preset threshold Y; or monotonically decreasing, with the filtering condition being that the calculated result is less than the preset threshold. These correspond to cases 1, 4, 5, and 8.

[0098] Type 2 filtering operators: monotonically decreasing, with the filtering condition that the calculated result is greater than a preset threshold Y; or monotonically increasing, with the filtering condition that the calculated result is less than a preset threshold. These correspond to cases 2, 3, 6, and 7.

[0099] Type 3 filtering operator: monotonically increasing, the filtering condition is that the calculation result is not equal to the preset threshold Y, corresponding to cases 9, 11, and 13.

[0100] Type 4 filtering operator: monotonically decreasing, the filtering condition is that the calculation result is not equal to the preset threshold Y, corresponding to cases 10, 12, and 14.

[0101] In step S103, the storage engine is controlled to project the target data block into memory, and the query engine is controlled to execute the filtering operator on the target data block to obtain the target data, wherein the target data is the data that was not filtered out by the filtering operator.

[0102] For example, in response to the target data block being marked as a pure data block, the data within the target data block is added to the target data; in response to the target data block not being marked as a pure data block, the query engine is controlled to execute the filtering operator for the target data block, and the resulting data is added to the target data.

[0103] The target data is the result of a data query using an SQL statement.

[0104] In the data processing method provided in this embodiment, the query engine pushes down a monotonic filtering operator to the storage engine, thereby controlling the storage engine to read data distribution metadata of at least one data block within the query range. Based on the data distribution metadata of the at least one data block and the filtering operator, the storage engine filters the at least one data block to obtain a target data block. Finally, the storage engine projects the target data block to the query engine, and the query engine executes the filtering operator on the target data block to obtain the target data. Since the storage engine can read the data distribution metadata, it can pre-filter data blocks based on the monotonicity of the filtering operator and the data block distribution metadata, such as maxima and minima, while only projecting a portion of the data within the query range to the query engine for calculation. This increases the efficiency of data filtering, reduces the amount of projected data, disk I / O, and the computational load of the query engine, and improves the efficiency of data querying. In particular, the storage engine can use the distribution metadata to comprehensively assess whether a data block needs filtering, avoiding the query engine's operation of traversing every data point during calculation, resulting in an order-of-magnitude improvement in efficiency.

[0105] Please refer to Figure 3, which exemplarily illustrates a flowchart of the data processing method obtained by combining the above embodiments.

[0106] First, the query engine analyzes the monotonicity of the filtering operator, that is, it determines whether the filtering operator has monotonicity. If it does not have monotonicity, the data is projected and the query engine calculates the result; if it has monotonicity, the calculation is pushed down to the storage engine, that is, the filtering operator is pushed down to the storage engine.

[0107] Next, considering the pushdown storage engine, determine whether the index row records aggregation information. If so, perform filtering based on the index row aggregation information, i.e., determine whether all data within the data block has been filtered.

[0108] Next, if the result of filtering based on the index row aggregation information is full filtering, then no more data blocks will be read; if the result of filtering based on the index row aggregation information is not full filtering, or if the index row does not record aggregation information, then the data block will be opened, and filtering will be performed based on the encoding information, that is, to determine whether all the data in the data block has been filtered.

[0109] Finally, if the result of filtering based on the encoding information is that all data is filtered, then no more data will be projected; if the result of filtering based on the encoding information is that not all data is filtered, then the data will be projected and the query engine will calculate the result.

[0110] Figure 4 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 4, at the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other hardware required for tasks. One or more embodiments of this disclosure can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into the memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this disclosure do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0111] Please refer to Figure 5. The data processing device can be applied to the device shown in Figure 4 to implement the technical solution of this disclosure. The device includes: a push-down module 501, used to control the query engine to push down a filtering operator with monotonicity to the storage engine; a first filtering module 502, used to control the storage engine to read data distribution metadata of at least one data block within the query range, and to filter the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, wherein the target data block is a data block that has not been filtered out by the filtering operator; and a second filtering module 503, used to control the storage engine to project the target data block to the query engine, and to control the query engine to execute the filtering operator on the target data block to obtain target data, wherein the target data is data that has not been filtered out by the filtering operator.

[0112] In one or more embodiments of this disclosure, when the first filtering module controls the storage engine to read the data distribution metadata of at least one data block within a query range, it is configured to: read the data distribution metadata of the at least one data block from the aggregation information of the intermediate index layer within the query range, wherein the intermediate index layer is a tree index of the data blocks within the query range, the intermediate index layer includes at least one layer of index blocks, and the aggregation information of the index blocks is used to describe the data distribution metadata of the next layer of index blocks or data blocks.

[0113] In one or more embodiments of this disclosure, when the first filtering module controls the storage engine to read the data distribution metadata of at least one data block within the query range, it is configured to: read the data distribution metadata of the data block from the encoding information of the data block.

[0114] In one or more embodiments of this disclosure, the data distribution metadata includes at least one of the following: maximum value, minimum value, and whether null values ​​exist.

[0115] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block satisfying the filtering condition; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block satisfying the filtering condition; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block satisfying the filtering condition; and determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block satisfying the filtering condition.

[0116] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, be configured to: determine the data block as a target data block and mark it as a pure data block in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition; and in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block does not satisfy the filtering condition. If the minimum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically increasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the minimum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block; if the filtering operator is monotonically decreasing, the filtering condition of the filtering operator is that the calculation result is less than a preset threshold, and the maximum value of the data block does not meet the filtering condition, the data block is determined as a target data block and marked as a pure data block.

[0117] In one or more embodiments of this disclosure, when the first filtering module is used to filter the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, it is used to: determine the data block as a target data block and mark it as a pure data block in response to the filtering operator being monotonically increasing or monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition.

[0118] In one or more embodiments of this disclosure, the first filtering module is configured to, when filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being less than the preset threshold; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being less than the preset threshold; determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the minimum value of the data block being greater than the preset threshold; and determine that the data block satisfies the filtering condition in response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to the preset threshold, and the calculation result corresponding to the maximum value of the data block being greater than the preset threshold.

[0119] In one or more embodiments of this disclosure, the second filtering module is used to control the storage engine to project a target data block to the query engine, and to control the query engine to execute the filtering operator on the target data block to obtain target data. When the target data is obtained, the second module is used to: add the data in the target data block to the target data in response to the target data block being marked as a pure data block; and control the query engine to execute the filtering operator on the target data block in response to the target data block not being marked as a pure data block, and add the obtained data result to the target data.

[0120] In one or more embodiments of this disclosure, the apparatus further includes a plan generation module for: generating a data query plan based on an SQL statement to be executed, wherein the data query plan includes at least one filtering operator; the push-down module is configured to: control the query engine to push down the filtering operator with monotonicity in the data query plan to the storage engine.

[0121] One or more embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0122] One or more embodiments of this disclosure also provide a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0123] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0124] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0128] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0129] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” used in one or more embodiments of this disclosure and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0130] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0131] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0132] The above description is merely a preferred embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of one or more embodiments of this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this disclosure should be included within the scope of protection of one or more embodiments of this disclosure.

Claims

1. A data processing method, the method comprising: The query engine controls the pushdown of monotonic filtering operators to the storage engine; The control storage engine reads data distribution metadata of at least one data block within the query range, and filters the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block, wherein the target data block is a data block that has not been filtered out by the filtering operator. The storage engine is controlled to project the target data block into memory, and the query engine is controlled to execute the filtering operator on the target data block to obtain the target data, wherein the target data is the data that was not filtered out by the filtering operator.

2. The data processing method according to claim 1, wherein the control storage engine reads data distribution metadata of at least one data block within the query range, comprising: The data distribution metadata of the at least one data block is read from the aggregation information of the intermediate index layer within the query range, wherein the intermediate index layer is a tree index of the data blocks within the query range, the intermediate index layer includes at least one layer of index blocks, and the aggregation information of the index blocks is used to describe the data distribution metadata of the index blocks or data blocks of the next layer.

3. The data processing method according to claim 1, wherein the control storage engine reads data distribution metadata of at least one data block within the query range, comprising: Read the data distribution metadata of the data block from the encoding information of the data block.

4. The data processing method according to claim 1, wherein the data distribution metadata includes at least one of the following: maximum value, minimum value, and whether null values ​​exist.

5. The data processing method according to claim 4, wherein filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block comprises: In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block satisfying the filtering condition, it is determined that the data block satisfies the filtering condition.

6. The data processing method according to claim 4, wherein filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block comprises: In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is greater than a preset threshold, and the minimum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block. In response to the monotonically increasing filtering operator, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the minimum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is less than a preset threshold, and the maximum value of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

7. The data processing method according to claim 4, wherein filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block comprises: In response to the filtering operator being monotonically increasing or monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the maximum and minimum values ​​of the data block not satisfying the filtering condition, the data block is determined to be a target data block and marked as a pure data block.

8. The data processing method according to claim 4, wherein filtering the at least one data block based on the data distribution metadata of the at least one data block and the filtering operator to obtain a target data block comprises: In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being less than the preset threshold, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the minimum value of the data block being less than the preset threshold, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically increasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the minimum value of the data block being greater than the preset threshold, it is determined that the data block satisfies the filtering condition. In response to the filtering operator being monotonically decreasing, the filtering condition of the filtering operator being that the calculation result is not equal to a preset threshold, and the calculation result corresponding to the maximum value of the data block being greater than the preset threshold, it is determined that the data block satisfies the filtering condition.

9. The data processing method according to claim 1, wherein controlling the storage engine to project the target data block to the query engine, and controlling the query engine to execute the filtering operator on the target data block to obtain the target data, includes: In response to the target data block being marked as a plain data block, the data within the target data block is added to the target data; In response to the target data block not being marked as a pure data block, the control query engine executes the filtering operator on the target data block and adds the resulting data to the target data.

10. The data processing method according to claim 1, further comprising: Generate a data query plan based on the SQL statement to be executed, wherein the data query plan contains at least one filtering operator; The control query engine pushes down monotonic filtering operators to the storage engine, including: The control query engine pushes down the monotonic filtering operators in the data query plan to the storage engine.

11. A data processing apparatus, the apparatus comprising: The pushdown module controls the query engine to push down filter operators with monotonicity to the storage engine; The first filtering module is used to control the storage engine to read the data distribution metadata of at least one data block within the query range, and to filter the at least one data block according to the data distribution metadata of the at least one data block and the filtering operator to obtain the target data block, wherein the target data block is the data block that has not been filtered out by the filtering operator. The second filtering module is used to control the storage engine to project the target data block to the query engine, and to control the query engine to execute the filtering operator on the target data block to obtain the target data, wherein the target data is the data that has not been filtered out by the filtering operator.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 10 by executing the executable instructions.

14. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 10.

Images