Filter construction method and apparatus, and device

Abstract

Provided in the embodiments of the present disclosure are a filter construction method and apparatus, and a device. The method comprises: for a data query plan, acquiring filter construction information corresponding to an operator chain corresponding to the data query plan, in order to determine join keys on which filters can be constructed; and then separately calculating the performance gain and / or the performance overhead brought by each filter; and determining, on the basis of the performance gain and / or the performance overhead, whether to construct a filter on a corresponding join key. In the embodiments of the present disclosure, the performance gain and the performance overhead brought by constructing filters can be fully taken into consideration, so as to selectively construct filters, thereby avoiding the overhead of constructed filters affecting the system performance.

Description

[0001] This disclosure claims priority to Chinese Patent Application No. 202410657157.5, filed on May 24, 2024, entitled "Filter Construction Method, Apparatus, and Device," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of data filtering technology, and more particularly to a filter construction method, apparatus, and device. Background Art With the advent of the big data era, data systems such as distributed databases and massively parallel processing (MPP) systems are increasingly widely used due to their efficient data processing and management methods. In related technologies, to improve the query performance of data systems, filters are typically constructed in these data systems. These filters are used to create filtering conditions, which are then applied during the data query execution process to filter the data, thereby reducing unnecessary data scanning and processing and improving data query efficiency. However, the filter construction process, transmission process, and filter calculation process all incur certain performance overhead, which can reduce the data query performance of the data system in some scenarios. The present disclosure provides a filter construction method, apparatus, and device to avoid the impact of filter construction overhead on the data query performance of the data system. In a first aspect, embodiments of the present disclosure provide a filter construction method, comprising: generating an operator chain corresponding to a data query plan based on the data query plan, the operator chain including multiple linearly connected operators, the operators including connection operators and scan operators; obtaining filter construction information corresponding to the operator chain based on the connection operators and scan operators, the filter construction information including the correspondence between the construction channels of the connection operators and the filtering channels of the scan operators in the operator chain; determining a target connection key for constructing the filter from the connection keys included in the operator chain based on the performance gains and / or performance overhead of the filter corresponding to the filter construction information; and constructing a filter at the construction end of the target connection key, the target connection key being used to filter data based on the filter when executing the data query plan. Secondly, embodiments of this disclosure also provide a data query method, comprising: responding to a data query plan, constructing a filter in an operator chain corresponding to the data query plan based on the filter construction method of the first aspect; executing the operator chain corresponding to the data query plan, and filtering data based on the filter to obtain a data query result corresponding to the data query plan.Thirdly, embodiments of this disclosure also provide a filter construction apparatus, comprising: a generation module, used to generate an operator chain corresponding to a data query plan based on the data query plan, the operator chain including multiple linearly connected operators, the operators including connection operators and scan operators; an acquisition module, used to acquire filter construction information corresponding to the operator chain based on the connection operators and scan operators, the filter construction information including the correspondence between the construction channels of the connection operators and the filtering channels of the scan operators in the operator chain; a determination module, used to determine a target connection key for constructing a filter in the connection keys included in the operator chain based on the performance gains and / or performance overhead of the filter corresponding to the filter construction information; and a construction module, used to construct a filter at the construction end of the target connection key, the target connection key being used to filter data based on the filter when executing the data query plan. Fourthly, embodiments of this disclosure provide a data query apparatus, comprising: the filter construction apparatus of the third aspect, used to construct a filter in an operator chain corresponding to the data query plan in response to the data query plan; and an execution device, used to filter data based on the filter when executing the operator chain corresponding to the data query plan. Fifthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer-executable instructions thereon. When executed by a processor, the computer-executable instructions implement the filter construction method provided in the first aspect, and / or implement the data query method provided in the second aspect. Sixthly, embodiments of this disclosure provide a computer program product including computer instructions. When executed by a processor, the computer instructions implement the filter construction method provided in the first aspect, and / or implement the data query method provided in the second aspect. In the filter construction method provided in embodiments of this disclosure, for a data query plan, filter construction information corresponding to the operator chain corresponding to the data query plan is obtained to determine the connection key where filters can be constructed. Then, the performance gains and / or performance overhead brought by each filter are calculated respectively. Based on the performance gains and / or performance overhead, it is determined whether to construct a filter in the corresponding connection key. In embodiments of this disclosure, the performance gains and performance overhead brought by constructing filters can be fully considered, thereby selectively constructing filters and avoiding the impact of the overhead of constructed filters on system performance. The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure.In the accompanying drawings: Figure 1 is a schematic diagram of a filter construction method provided in an embodiment of the present disclosure; Figure 2 is a flowchart of a filter construction method provided in an embodiment of the present disclosure; Figure 3 is a schematic diagram of an operator chain corresponding to a data query plan provided in an embodiment of the present disclosure; Figure 4 is a schematic diagram of the principle of obtaining filter construction information provided in an embodiment of the present disclosure; Figure 5 is a schematic diagram of the parallelism of the operator chain provided in an embodiment of the present disclosure; Figure 6 is a schematic diagram of the principle of constructing a filter provided in an embodiment of the present disclosure; Figure 7 is a schematic diagram of the principle of constructing a filter provided in an embodiment of the present disclosure; Figure 8 is a flowchart of a data query method provided in an embodiment of the present disclosure; Figure 9 is a schematic diagram of the principle of a transit filter provided in an embodiment of the present disclosure; Figure 10 is a schematic diagram of the principle of the execution process of a scan operator provided in an embodiment of the present disclosure; Figure 11 is a schematic diagram of the principle of the execution process of a data query plan Q5 provided in an embodiment of the present disclosure; Figure 12 is a structural schematic diagram of a filter construction device provided in an embodiment of the present disclosure; Figure 13 is a structural schematic diagram of a data query device provided in an embodiment of the present disclosure; Figure 14 is a structural schematic diagram of an electronic device provided in an embodiment of the present disclosure. To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. First, the meanings of some terms appearing in the embodiments of this disclosure are explained: Runtime Filter: A query optimization technique used to improve the query performance of distributed database systems or massively parallel processing (MPP) databases. Runtime filters are mainly used to accelerate join operations in large data warehouses and big data systems. When executing distributed join queries, runtime filters can dynamically create filtering conditions and apply these conditions to the data partitions to be processed during query execution, thereby reducing unnecessary data scanning and processing. Bloom Filter: A highly space-efficient probabilistic data structure used to test whether an element is a member of a set.Bloom filters can quickly determine whether an element is definitely not in a set or might be in a set. Therefore, they have a characteristic that allows for a certain error rate: they may produce false positives (i.e., they determine an element is in the set when it is not), but they will never produce false negatives (i.e., if they determine an element is not in the set, then that element is definitely not in the set). The inventors discovered that in related technologies, for data systems using runtime filtering, for the hashjoin operator, a certain type of data summary is constructed from the build end of the hashjoin operator and passed to the probe end for pre-filtering. Specifically, from the scan operator to the probe side of the join operator, each node in this data processing chain has the opportunity to gain performance benefits. Specific performance benefits include, but are not limited to, at least one of the following:

[0002] ① The amount of data in the Scan operator is reduced, thereby reducing the input / output (Input / Output, 10) overhead or the computational overhead of the pushdown filter;

[0003] ② The amount of data exchanged is reduced, thereby improving Exchange speed;

[0004] ③ The amount of data involved in Probe is reduced, which improves the processing speed of the Hashjoin operator;

[0005] ④ Reduce the computational overhead of expressions for projects and filters on the Scan-Probe link of the scan operator. However, certain performance overhead may also be incurred in these nodes, including but not limited to at least one of the following:

[0006] ①Additionally introduces the overhead of building, passing, and filtering runtime filters.

[0007] ② It introduces complexity to the optimizer and scheduler. Specifically, in some scenarios, runtime filter builder and filter operators need to be inserted at the build and probe ends of the Join operator, respectively, to perform filter construction and filtering operations. Simultaneously, rules for runtime filter merging, pushdown, and transposition need to be introduced into the optimizer. This makes the optimizer design more complex and expands the search space for cost-based optimization (CBO), increasing optimizer overhead. Furthermore, the construction of runtime filters often depends on operator cost estimation. For example, the size of the constructed filter needs to be initialized based on the output rows of the Join operator and the estimated cardinality from the Join builder. However, the estimated results differ significantly from the actual output rows, severely impacting the filter's accuracy. In summary, if this performance overhead cannot be offset by performance gains, constructing runtime filters will cause a severe performance regression for the data system. In view of this, embodiments of this disclosure provide a filter construction method, apparatus, and device. For a data query plan, filter construction information corresponding to the operator chain corresponding to the data query plan is obtained to determine the connection key where filters can be constructed. Then, the performance benefits and / or performance overhead brought by each filter are calculated respectively. Based on the performance benefits and / or performance overhead, it is determined whether to construct a filter in the corresponding connection key. In embodiments of this disclosure, the performance benefits and performance overhead brought by constructing filters can be fully considered, thereby selectively constructing filters and avoiding the impact of the overhead of constructed filters on system performance. The technical solutions provided by various embodiments of this disclosure are described in detail below with reference to the accompanying drawings. Figure 1 is a schematic diagram of a scenario of a filter construction method provided by an embodiment of this disclosure. As shown in Figure 1, the scenario includes: a data query plan and an electronic device. The data query plan represents the sequence of operations used to execute a Structured Query Language (SQL) query. That is, the data query plan can be an SQL execution plan. In a database, the process of generating an execution plan from SQL is called query optimization. The SQL statement submitted by the user is first parsed by a parser to obtain the data query plan in this embodiment of the disclosure.In some embodiments, the electronic device can be a terminal or a server. For example, the terminal can be a personal digital assistant (PDA) device, a handheld device with wireless communication capabilities (e.g., a smartphone, tablet), a computing device (e.g., a personal computer, PC), an in-vehicle device, a wearable device (e.g., a smartwatch, smart bracelet), a smart home device (e.g., a smart display device), etc. In this embodiment, the electronic device can be equipped with a database system, such as the MPP database mentioned above. When executing the scheme of this embodiment, when the electronic device obtains a data query plan, it constructs a corresponding filter for the data query plan, applies the constructed filter to filter the data during the data query process, and outputs the data query results. Below, the technical solution of this disclosure will be described in detail through specific embodiments, in conjunction with the application scenario shown in Figure 1 above. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that the execution subject of the embodiments of this disclosure is an electronic device. Referring to FIG2, FIG2 is a schematic flowchart of a filter construction method provided by an embodiment of this disclosure. As shown in FIG2, the filter construction method includes the following steps:

[0008] S201. Generate an operator chain corresponding to the data query plan based on the data query plan. The operator chain (Pipeline) includes multiple linearly connected operators, including join operators and scan operators. In some optional embodiments, the SQL statement submitted by the user is first parsed by a parser, which can convert it into an intermediate representation, typically an Abstract Smantics Tree (AST), i.e., the data query plan in this embodiment. The AST is then passed to the query optimizer, which determines the most efficient way to execute the data query plan. Specifically, the optimizer evaluates different query execution strategies, including choosing which join method to use (such as nested loops, hash joins, or sort-merge joins), determining the order of operations, and selecting indexes. It should be understood that this evaluation phase typically relies on statistical information, such as data distribution, index characteristics, and table size, which helps the optimizer estimate the cost of various execution strategies. Furthermore, after evaluating all possible execution strategies and their costs, the optimizer selects the lowest-cost execution plan, generating a data query plan or execution plan tree. The data query plan details how the database engine executes the query, including the order of operations, the algorithms used, and the data structures. Finally, the database execution engine executes the query operation according to the execution plan and returns the results to the user, aiming to provide database users with fast and efficient query performance. Specifically, generating the operator chain corresponding to the data query plan involves the following steps:

[0009] ① Analyze the execution plan tree corresponding to the data query plan; The execution plan tree corresponding to the data query plan is usually composed of a series of interconnected nodes, each node representing a query operation, such as a select, join, or aggregate operation.

[0010] ② Identify operator pipeline segments; Specifically, when generating operator pipelines based on the data query plan, the process typically starts from the leaf node (usually a scan node) of the execution plan tree corresponding to the data query plan and traverses upwards to identify the sequence of operations that can be executed in parallel. Operators that do not require waiting for preceding operations to complete all outputs (i.e., streaming operators) can be organized into a pipeline. In some optional implementations, during the identification process, operators can be labeled as pipeline producers (data sources) and consumers (data receivers).

[0011] ③ Constructing the Pipeline; It should be understood that when constructing a Pipeline, data dependencies and flow control between query operations must be considered to ensure that data flows in the correct order. For example, sorting or certain aggregation operations may be the end point of the Pipeline (i.e., blocking operators) because they require all input data to be entered before processing can begin. In some embodiments, the Pipeline includes the following two operator execution modes: Mode 1: Blocking execution, where an operation must wait for all its inputs to be processed before it can execute; Mode 2: Non-blocking execution, where an operator can immediately process the output of an upstream operator without waiting for all inputs to be processed. In other words, the Pipeline allows another operation to begin processing the output of the previous operation while one operation is processing the current batch of data, thereby improving the overall performance of database queries. Furthermore, based on data flow and dependencies, independent query operations can be combined into one or more...

[0012] Each Pipeline is an "execution unit" that can be executed independently or in parallel. Different Pipelines may be connected through intermediate data structures (such as buffers).

[0013] ④ Pipeline Optimization: In some optional implementations, after the pipeline is built, further optimization can be performed, such as reducing the size of intermediate results, optimizing data transmission, and merging operations where possible to reduce context switching, etc. It should be noted that during the pipeline construction process, resource usage, the potential for parallel computing, and the characteristics of various operators can be balanced to fully utilize system resources and optimize query performance. Referring to Figure 3, Figure 3 is a schematic diagram of an operator chain corresponding to a data query plan provided in an embodiment of this disclosure. As shown in Figure 3, it is an execution plan tree corresponding to a data execution plan containing two Join operators. This data execution plan includes operator chain 1, operator chain 2, and operator chain 3.

[0014] 5202. Based on the join operator and the scan operator, obtain the filter construction information corresponding to the operator chain. The filter construction information includes the correspondence between the construction channels of the join operator and the filtering channels of the scan operator in the operator chain. In other words, the filter construction information indicates that the filter built on a certain build key column of the join operator (Hashjoin) should be matched with the scan operator in the pipeline to which the HashJoin operator belongs.

[0015] On a specific column of (Scan), the column is used for filtering calculations by the Scan operator. Specifically, a filter build information S contains a build channel for the Hash Join operator.

[0016] The filter construction information S = {Channel; , Channels} refers to a filter channel for the Scan operator. A channel refers to the position of a data column within an executor data unit (Chunk). That is, the filter construction information S = {Channel; , Channels}. It should be understood that a Hash Join operator may have multiple join keys, and a Pipeline contains multiple Hash Join operators. In this step, the number of filter construction information S is equal to the sum of the number of join keys for all Hash Join operators; that is, there is a one-to-one correspondence between the filter construction information S and all join keys for the Hash Join operator.

[0017] 5203. Based on the performance gains and / or performance costs of the filters corresponding to the filter construction information, determine the target join key to be used for filter construction from the join keys contained in the operator chain. For example, after a record is read from the Scan operator, a hash table probe is performed on the Hash Join operator, and its memory access cost is C. After introducing the Runtime Filter, the Scan operator needs to perform K mappings on this record, and the memory access cost and mapping computation cost of each mapping position are 2. Among them, the mapping computation cost is generally much smaller than the memory access cost, so C*2 can be approximated as the pure memory access cost. Furthermore, the performance cost increased by introducing the Runtime Filter is: R ■ K ■ C2; The performance gain brought by introducing the Runtime Filter is: (1 — fpp) (1 — selectvity) ■ R・C LWhere fpp is the false positive rate of the filter, (1 - fpp) is the correction value for false positives, and (1 - selectvity) is the selection rate of the Hashjoin operator. In this step, the performance gain and / or performance cost of each filter are calculated based on the above scheme. For example, for any Join key N in the Hashjoin operator, the performance cost of its corresponding Nth filter is: R - K - C2, and the performance gain is: (1 - fpp) (1 - selectvity) ■ R・C L If the performance overhead and / or performance gain of the Nth filter meet the preset requirements, then it is determined that constructing the Nth filter will not cause a performance regression in the system, and the Join key N corresponding to the Nth filter is the target join key. The specific method for determining the filter that meets the preset requirements based on performance gains and / or performance overhead will be shown in subsequent embodiments.

[0018] S204. Construct a filter at the Build end of the target join key. The target join key is used to filter data based on the filter when executing the data query plan. In this embodiment, the performance benefits and overhead of constructing filters are fully considered, allowing for selective filter construction and avoiding the impact of filter overhead on system performance. In one embodiment, step S202 is described in detail with specific examples: When obtaining filter construction information corresponding to the operator chain based on the join operator and scan operator, the specific steps include:

[0019] ① Obtain the first identifier information of the build key and the second identifier information of the probe key for the connection operators in the operator chain. The first identifier information of the build key includes the table name and column name information of the connection key; the second identifier information of the probe key includes the table name and column name information of the probe key. It should be noted that a Pipeline and its scheduling information are collectively referred to as a Pipeline Fragment. A Pipeline Fragment is the basic unit of a Pipeline under the scheduling mechanism, and each Pipeline Fragment is configured with an in-fragment runtime filter manager (Fragment-RF-Manager).

[0020] ② For each join key on the join operator, based on the first identifier information of the build key on the join key, the build channel in the filter build information is obtained from the metadata of the join operator; and based on the second identifier information of the probe key on the join key, the filter channel corresponding to the build channel is obtained from the metadata of the scan operator. Specifically, please refer to Figure 4, which is a schematic diagram of the principle of obtaining filter build information provided in an embodiment of this disclosure. As shown in Figure 4, for a Build Channel, during the top-down traversal of the execution plan tree, when encountering a Hash Join operator, we obtain the Fragment-RF-Manager of the Pipeline Fragment to which the operator belongs. We register and save the {Build key, Probe key} information of each Join key pair in the Hash Join operator to the Fragment-RF-Manager. Simultaneously, based on the first identifier information (i.e., the table name and column name information of the Build key), we search the metadata of the Hash Join operator to find the Build Channel (i.e., Channel). For Filter Channels, during the top-down traversal of the execution plan tree, when encountering the Scan operator, based on all the {Build key, Probe key} information collected by the Fragment-RF-Manager, we sequentially read the second identifier information (i.e., the table name and column name information of the Probe key). Simultaneously, based on the second identifier information... The corresponding channel is found in the metadata of the Scan operator to obtain the Filter Channel (i.e., Channels).

[0021] ③ Based on the construction channel and filtering channel corresponding to the connection key contained in the operator chain, obtain the filter construction information corresponding to the operator chain. In summary, for each operator chain, by performing the above steps ① and ②, the filter construction information corresponding to each operator chain can be obtained. Example 2: In this example, a specific implementation of step S203 is described in detail: Specifically, when determining the target connection key for constructing the filter from the connection keys contained in the operator chain based on the performance gain and / or performance overhead of the filter corresponding to the filter construction information, at least one of the following methods is included:

[0022] ① For each filter, construct the corresponding filter and obtain its performance gain. If the performance gain is not less than a preset performance gain, then the connection key corresponding to the filter is determined as the target connection key. It should be noted that the performance gain of a filter must exceed its corresponding performance overhead for the system to experience a performance benefit. That is, if the performance gain of the filter is greater than its performance overhead, then the system will receive a performance benefit. ± If the performance overhead R·K·spoon is greater than the target join key, then the join key corresponding to this filter is the target join key. That is, if (1 — / pp)(l — selectvity) ■ R ■ C x If R ■ K ■ C2, then the Join key corresponding to this filter is the target join key.

[0023] ② For each filter built, the performance overhead of the filter is obtained. If the performance overhead is not greater than a preset performance overhead, the connection key corresponding to the filter is determined as the target connection key. In some embodiments, a preset performance overhead can also be used. If the performance overhead of a filter is less than the preset performance overhead, it is considered that building the filter will not cause a performance regression in the system, and the connection key corresponding to the filter is the target connection key. It should be noted that the specific value of the preset performance overhead is not limited in this embodiment. In this embodiment, only the performance overhead of the filter needs to be calculated, without calculating the performance gain of the filter, which greatly reduces the amount of computation and improves the efficiency of filter construction.

[0024] ③ For each filter construction information corresponding to a filter, obtain the corresponding performance overhead and performance gain of the filter. If the performance overhead is less than the performance gain, then the connection key corresponding to the filter is determined as the target connection key. Similarly, in some embodiments, a performance gain can be preset. If the performance gain value corresponding to a certain filter is less than the preset performance gain, then it is considered that building the filter will bring performance gains to the system, and the connection key corresponding to the filter is the target connection key. It should be noted that the specific value of the preset performance gain is not limited in this embodiment. In this embodiment, only the performance gain of the filter needs to be calculated, and the performance overhead of the filter does not need to be calculated, which greatly reduces the amount of calculation and can improve the efficiency of filter construction. Embodiment 3 In this embodiment, another implementation of the above step S203 is described in detail with reference to specific examples: Specifically, when determining the target connection key of the filter to be built from the connection keys contained in the operator chain according to the performance gain and / or performance overhead of the filter corresponding to the filter construction information, the following steps are included:

[0025] ① For each filter, construct the corresponding filter information and obtain the corresponding performance overhead and performance gain of the filter; For the method of obtaining the corresponding performance overhead and performance gain of the filter, please refer to the above embodiment, which will not be repeated here.

[0026] ② Based on the performance overhead and performance benefits, obtain the memory access efficiency ratio corresponding to the filter; where the memory access efficiency ratio is K • x, representing the ratio of the memory access efficiency of the filter to the memory access efficiency of the hash table of Hashjoin.

[0027] ③ Obtain the screening rate of the join operator after false positive correction; where (1 — fpp) is the correction value of the filter for false positives, (1 — selectvity) is the screening rate of the Hash Join operator, and the screening rate of the join operator after false positive correction is (1 - fpp)(1 — selectvity).

[0028] ④ Determine whether the join key corresponding to the filter is the target join key based on the memory access efficiency ratio and / or the selection rate. Specifically, determining whether the join key corresponding to the filter is the target join key based on the memory access efficiency ratio and / or the selection rate includes at least one of the following methods: Method 1: If the memory access efficiency ratio is less than the selection rate, determine that the join operator corresponding to the filter is the target join key that meets the preset requirements; if the memory access efficiency ratio is greater than or equal to the selection rate, determine that the join operator corresponding to the filter is not the target join key. Specifically, if (1 - fpp) (1 - selectvity), then the join operator corresponding to the filter is the target join key. The join key is used to indicate the target. Conversely, if (1 - fpp) (1 - selectvity), then the join operator corresponding to this filter is not the target. Method 2: If the selection rate (1 - Selectivity) is greater than or equal to the preset selection rate, the join key corresponding to the filter is determined to be the target join key; if the selection rate (1 - Selectivity) is less than the preset selection rate, the join operator corresponding to the filter is determined not to be the target join key. In other words, in this scheme, the memory access efficiency ratio corresponding to the filter may not be calculated; only the selection rate corresponding to the filter is used to determine whether the filter can cause performance regression in the system. It should be understood that the preset selection rate is not specifically limited in this embodiment. This scheme can significantly reduce the amount of computation, thereby improving system performance to a certain extent. Method 3: If the difference between the memory access efficiency ratio and the preset memory access efficiency ratio is less than the first preset value, the join key corresponding to the filter is determined to be the target join key based on the line number in the build. Similarly, in this scheme, the filtering rate can be omitted, and the performance regression of the filter can be determined solely based on the memory access efficiency ratio corresponding to the filter. It should be understood that this disclosure does not specifically limit the size of the preset memory access efficiency ratio or the first preset value; for example, the preset memory access efficiency ratio can be 1. This scheme significantly reduces the computational load, thereby improving system performance to a certain extent. Furthermore, in this embodiment, after constructing the hash table, the values ​​of Ci, C2, and Selectivity can be calculated by combining existing information. These values ​​can be substituted into the formula in Method 1 above, or directly compared using Methods 2 and 3. If the conditions are met, the Runtime Filter can be enabled. This forms an adaptive Runtime Filter switching mechanism, greatly improving the automation of filter construction and thus improving data query efficiency. In Embodiment 4, the inventors discovered that the Join operator often exhibits significant deviations in cardinality and selectivity estimation, which is difficult to avoid in databases. Therefore, the selectivity obtained by the above scheme may differ significantly from the true value. In view of this, when the Selectivity estimate is inaccurate, the present embodiment can also determine whether the connection key corresponding to the filter is the target connection key through at least one of the following situations: Situation 1: If the number of rows on the Build side is greater than a second preset value or less than a third preset value, then the connection key corresponding to the filter is determined to be the target connection key. Situation 2: If the number of rows on the Build side is less than or equal to the second preset value, or if the number of rows on the Build side is greater than or equal to the third preset value, then the connection key corresponding to the filter is determined not to be the target connection key.It should be noted that when the number of rows in the build phase is too small, the value of K·hú is also close to 1, which is unlikely to bring any benefit to the system. In practical applications, the number of rows in the build phase can be deduced from the memory space usage of the hash table, thereby determining the second preset value. For example, the memory space occupied by the physical machine (e.g., L1 cache space) can be known.

[0029] The cache size is 32KB, and the hash table occupies a memory space (e.g., load factor).

[0030] If the value is 0.5, then the following condition is met: next_power_of_2(r / Load Factor) * 4Bytes < 32KB. The number of lines in the Build is r. Therefore, we can conclude that r < 4096. We roughly agree that the lower bound of the number of lines in the Build is 4096, as one of the conditions for enabling the Runtime Filter. That is, the second preset value can be set to 4096. Similarly, when the number of lines in the Build is too large, the value of K • color is also close to 1, and it is impossible to bring any benefit to the system.

[0031] In practical applications, the third preset value can be determined based on the memory space occupied by the Runtime Filler. For example, taking the Runtime Filter memory space occupied (e.g., L3 Cache Size) as an example, if the physical machine's L3 Cache Size is known to be 16MB, then next_power_of_2(r / 8) * 8Bytes > 16 MB, where the number of lines r on the Build side is between 10 million and 20 million. Therefore, we can roughly agree that having less than 20 million lines on the Build side is one of the conditions for enabling the Runtime Filter; that is, the third preset value can be set to 20 million. In summary, if the number of lines on the Build side is greater than 4096 or less than 20 million, then the join key corresponding to the filter is determined to be the target join key; that is, the target join key can be used to construct the filter. If the number of rows in the Build phase is less than or equal to 4096, or greater than or equal to 20 million, then the join key corresponding to the filter is determined to be not the target join key; that is, the target join key does not need to have a filter built. In some optional cases, the filter's filtering performance can be used to determine whether to disable the filter after it has been built. When building the Runtime Filter in the Build phase, the filtering performance of the Probe phase is not yet known. Therefore, before building the filter, it is impossible to determine whether to build it based on the Probe-side filtering performance. Only after the filter is built can its filtering performance be used to determine whether to disable it, thus minimizing the performance loss caused by the filter. In view of this, we can introduce a dynamic sampling and checking mechanism in the Scan phase to check the filtering performance of the first 5-10 chunks in each batch. If the filtering performance of the first 5-10 chunks is very poor, then the runtime filter calculation for this batch is terminated, i.e., the filter is disabled. Of course, by this time the Runtime Filter has been built, and performance loss in the first 5-10 chunks cannot be avoided. However, it can prevent further performance loss to the system. As can be understood in Embodiment 5, to fully utilize the system's multi-core CPU parallel capabilities, each pipeline can be executed by multiple degrees of parallelism. In this embodiment, after generating at least one operator chain corresponding to the data query plan based on the data query plan, the following steps are also included:

[0032] ① For each operator chain, determine the number of parallelisms in the operator chain;

[0033] ② Based on the parallelism level, set the same number of threads as the parallelism level for the operator chain. These threads are used to execute the connection operators in the operator chain. Specifically, the parallelism level of the operator chain can be determined based on the assessed available system resources, including the number of CPU cores, memory capacity, and I / O capability. It should be understood that the parallelism level should not exceed the resource limits that the system can effectively process; otherwise, it may lead to excessive context switching, resource contention, and performance degradation. Furthermore, the distribution and partitioning of data are crucial for achieving effective parallel processing. Data needs to be divided into a sufficient number of partitions so that multiple processors or nodes can work in parallel. The uniformity of data distribution also affects load balancing, thereby affecting the efficiency of parallel execution. Please refer to Figure 5, which is a schematic diagram of the parallelism of the operator chain provided in this embodiment. As shown in Figure 5, for the three operator chains shown in Figure 3, parallelism levels of 3, 3, and 4 can be set for these three operator chains, respectively. In this embodiment of the disclosure, by setting the parallelism of the Pipeline, the utilization rate of system resources can be significantly improved. Since setting multiple parallelisms allows different query stages to be processed simultaneously on multiple cores / processors, the data query performance of the system can be greatly improved. In some embodiments, when building a filter at the Build end of the target join key, the following steps are specifically included:

[0034] ① Determine the data partitioning mode corresponding to the target join key;

[0035] ② Based on the data partitioning pattern of the target join key, construct a filter on the Build side of the target join key. In some embodiments, within a Pipeline Fragment, the Hashjoin operator can be divided into different data partitioning patterns (the data partitioning pattern corresponding to the Hashjoin operator is the data partitioning pattern corresponding to the target join key in the Hashjoin operator) according to different data partitioning and parallel strategies. The data partitioning patterns include two modes: a first partitioning pattern (Broadcast pattern) and a second partitioning pattern (Local pattern). The first partitioning pattern indicates that each partition on the Build side of the target join key corresponds to the same hash table; that is, each partition on the Build side shares a hash table. The second partitioning pattern indicates that each partition on the Build side of the target join key corresponds to a different hash table; that is, each partition on the Build side has its own independently maintained hash table. Correspondingly, for target join keys with the second partitioning pattern (Local pattern), a shared filter can be constructed for all partitions; for target join keys with the first partitioning pattern (Broadcast pattern), filters corresponding to each target join key are constructed separately. When constructing the filter, the following scenarios are considered: Scenario 1: If the target join key is in the first partition mode, for each thread corresponding to the target join key, a filter component corresponding to that thread is constructed, and the filter components of the threads corresponding to the target join key are merged to obtain the filter corresponding to the target join key. Specifically, in Broadcast mode, on the Build side of the Hash Join operator, each thread shares the same hash table and ultimately shares the same filter. After the hash table is constructed and the target join key for which the filter needs to be constructed is determined, each thread collaborates to begin constructing the filter. Please refer to Figure 6, which is a schematic diagram of the principle of filter construction provided in an embodiment of this disclosure. As shown in Figure 5, since different threads may map to the same memory region at the same time, if a memory block is allocated and written directly by each thread, a large amount of concurrency safety overhead is required. Therefore, in this embodiment of the disclosure, an additional locking mechanism is needed to ensure that the writing to the memory region is concurrently safe. To avoid concurrency safety overhead, a filter component (RF component) can be constructed locally for each thread, and its size is consistent with the final filter size.During the construction process, each thread writes data to its local component. After construction, each thread acquires a write lock and merges the filter components into a unified filter. It should be noted that merging filter components involves ORing all bits in the memory region with all bits in the target filter's memory region and then assigning the result. This process is very fast due to prefetching capabilities, so write locks can be used to queue the components within each thread. Scenario 2: If the target join key is in the second partition mode, a local filter is constructed for each partition corresponding to the target join key to obtain the filter corresponding to the target join key. Specifically, in Local mode, each partition uses its own independent hash table, and all data within each Chunk comes from the same table partition. In this case, one thread may handle the construction of hash tables and filters for multiple partitions; or multiple threads may jointly complete the construction of hash tables and filters within a single partition. Please refer to Figure 7, which is a schematic diagram of the principle of filter construction provided in one embodiment of this disclosure. As shown in Figure 5, if the target connection key is the first partition mode, in the process of building the filter, firstly, a local filter (Partial Runtime Filter) is formed for each partition. o Furthermore, one thread performs partitioning on each chunk, finding the Partial corresponding to the chunk.

[0036] Runtime Filter. Furthermore, the write lock on the partition holding the Partial Runtime Filter is used before writing data, avoiding concurrency issues that arise when other threads modify the Partial Runtime Filter of the same partition. Finally, the results are stored in an array of Partial Runtime Filters, with each position in the array storing the Partial Runtime Filter for the corresponding partition; merging of individual Partial Runtime Filters is no longer performed. For Embodiment Seven, please refer to Figure 8, which is a flowchart illustrating a data query method provided in an embodiment of this disclosure. As shown in Figure 8, the data query specifically includes the following steps:

[0037] 5801. In response to the data query plan, construct a filter in the operator chain corresponding to the data query plan. It should be noted that the implementation of step S801 is described in the embodiments corresponding to the filter construction method in the above examples, and will not be repeated here.

[0038] 5802. Execute the operator chain corresponding to the data query plan, and filter the data based on the filter to obtain the data query result corresponding to the data query plan. Specifically, the Fragment-RF-Manager component mentioned in Embodiment 1 can be used to pass the constructed filter to the scan operator. In this embodiment of the disclosure, a lightweight design is adopted, and the filter is passed only through memory, without the need for mechanisms such as the client-server model. Please refer to Figure 9, which is a schematic diagram of the principle of passing the filter provided in an embodiment of the disclosure. As shown in Figure 9, the filter passing process specifically includes the following operations: First, the Runtime Filters constructed in parallel under different modes are stored in the fields of Fragment-RF-Manager; Further, when the Scan operator processes each Chunk, it checks whether the fields of Fragment-RF-Manager contain Runtime Filter field data; If they exist, the Runtime Filter is used for additional data filtering; if they do not exist, the data query plan is executed according to the normal mode without Runtime Filter. Specifically, please refer to Figure 10, which is a schematic diagram of the execution process of a scanning operator provided in an embodiment of this disclosure. As shown in Figure 10, the execution process of the Scan operator specifically includes the following stages: In the scheduling stage, the scheduler registers the Runtime Filter Manager (hereinafter referred to as RF Manager) to the Scan operator;

[0039] During execution, Scan enumerates each Runtime Filter item in RF Manager (hereinafter referred to as RF item) and checks whether a Runtime Filter already exists for each item (i.e., based on the filter item key-value pair).

[0040] (RF item key) Determines whether a pre-created Runtime Filter already exists in the item. During the execution phase of the Scan operator, before and after dynamically detecting the Runtime Filter, there are two processes: Process 1: When no Runtime Filter exists, it includes the following operations: Operation 1: Using the Logical Row Group component, enumerate the readable logical data range; Operation 2: Using the Stripe Loader component, perform IO prefetch tasks for each column.

[0041] IO Task); Operation 3: Combining Operation 1 and Operation 2, obtain the filter column, perform filter calculation, and obtain the filter result (e.g., a bitmap). Operation 4: Apply the bitmap to the Project column obtained from Operation 1 and Operation 2. Process 2: When an item in RF Manager is detected to have a pre-created Runtime Filter, the following operations are included: First, for each filter built in the operator chain, obtain the filtering effect of the filter in the process of executing the data query plan. Specifically, when executing the data query plan based on the operator chain, first, add the Runtime Filter column to the Stripe Loader component, create 10 prefetch tasks belonging to this column, then register the RF item to the filter, perform data filtering through the filter, and at the same time, perform validity checks on the first N chunks of each batch through the effect checker (the first N chunks are the effect check samples). It should be understood that the value of N is not particularly limited in this embodiment. Furthermore, if the filtering effect is not less than the threshold, the filter will continue to be used for data filtering during the execution of the data query plan; if the filtering effect is less than the threshold, the filter will be stopped during the execution of the data query plan. Specifically, the filter can perform filtering calculations on Blocks and Runtime Filters, detect the existence of each position in the Block, and obtain a Bitmap. For the Bitmap, the filtering effect is statistically collected. If the filtering effect exceeds the threshold (the threshold is, for example, when...), the filter will continue to be used for data filtering.

[0042] If the filter passes the validity check (25%), then the filter's validity check is passed. That is, when the current N chunks pass the validity check, the Runtime Filter continues to execute subsequent chunks. If the filtering effect does not exceed the threshold, the execution of the Runtime Filter will be rejected. That is, when the current N chunks fail the validity check, the Runtime Filter is disabled to execute subsequent chunks. For example, please refer to Figure 11. Figure 11 is a schematic diagram of the execution process of the data query plan Q5 provided in an embodiment of this disclosure. As shown in Figure 11, during the execution of Q5, there is a large Pipeline, which includes the following three pairs of Join relationships: l_orderkey = o_orderkey

[0043] Lsuppkey = s_suppkey n_nationkey = s_nationkey It should be understood that, in this embodiment of the disclosure, the validity check can be performed by the interface RfefficiencyChecker. Based on this embodiment of the disclosure, three runtime filters corresponding to Q5 can be formed for pipeline projects.

[0044] The filtering effect of the (Llineitem) table Scan is 6 billion rows > 310 million rows. Therefore, according to the benefit calculation method provided in Embodiment 4 above, it can be concluded that the number of rows on the Build side corresponding to n_nationkey = s_nationkey is too small (25 rows), so the Runtime Filter formed by this Join key is disabled. According to the validity check method provided in this embodiment, it can be concluded that the Runtime Filter formed by l.suppkey = s_suppkey has a filtering efficiency of less than 25% when sampled and detected on the Scan side, so it is disabled during the Scan filtering process. In summary, the Runtime Filter formed by l_orderkey = o_orderkey ultimately produces the actual effect. Therefore, the Runtime Filter formed by l_orderkey = o_orderkey continues to be used to filter data, and the Runtime Filter formed by n_nationkey = s_nationkey and l_suppkey = s_suppkey is disabled to filter data. In summary, since the runtime filter formed by l_suppkey = s_suppkey has poor filtering performance and introduces an additional 10% overhead of Lsuppkey, the validity check provided in this embodiment can disable the filter, thereby avoiding the performance impact of a poorly performing filter on the system and improving system performance. Embodiment Eight: In some embodiments, the filter provided in this embodiment can be at least one of a Bloom filter and a blocked Bloom filter. Among them, a Bloom filter is the most common implementation of a runtime filter; it is a data digest that can represent the existence of a batch of data with less storage space.

[0045] During the data filtering process of the Bloom Filter, for a piece of data, it is mapped to different positions (a single bit) in a memory space using the K mapping functions. Each mapped position sets the bit to 1. When it is necessary to determine the existence of the data, the just-mentioned K mapping process is repeated. If all the mapped bits are checked to be 1, it is determined that the data exists. On the contrary, if there is a situation where a bit is 0, it is determined that the data does not exist. It should be noted that the Bloom Filter has a certain error rate, where non-existent data is determined to exist. This characteristic is also known as False-Positive, and the error rate fpp generally ranges from 0.03 to 0.1. Taking the Bloom Filter as an example, there is the following inference: In a Pipeline operator chain, the more complex the execution link from the Scan operator to the Hashjoin operator, the lower the performance requirements for the Runtime Filter itself; on the contrary, the simpler the execution link, the higher the performance requirements for the Runtime Filter itself. Proof: The number of filtered rows of a Runtime Filter on a dataset with a total number of rows of total_rows is Filtered_rows ? The average overhead per row is L; the average overhead per row of the calculation link is M. If the Runtime Filter is effective, it must satisfy M * Filtered_rows > L * total_rows , That is, L < When the Runtime Filter algorithm is the same, the score representing the filtering ability is a fixed value. Therefore, when M is smaller and the complexity is lower, the overhead L also needs to be smaller, and the performance requirements for the Runtime Filter are more stringent. Based on this inference, we construct a sufficiently simple scenario. Assume that a Pipeline consists of a Scan operator and a Hashjoin operator:

[0046] The selectivity of a pair of Join keys of the Hashjoin for the data on the Probe side is Selectivity

[0047] (0 < Selectivity < 1). Let the number of input rows at the Probe end be Ro

[0048] All the data of the Scan operator exists in memory and does not require 10 operations;

[0049] There are no other computationally expensive operators in the execution chain from the Scan operator to the Join operator. Under these conditions, the performance requirements for the Runtime Filter are the most stringent. This is because the execution overhead is minimal, consisting only of hash table probing at the Hashjoin Probe end. The entire process occurs in memory, eliminating overhead from intermediate operators.

[0050] The overhead of building, computing, and passing a Runtime Filter must be lower than this execution overhead to have a chance of achieving positive performance gains. In other words, if this stringent performance requirement can be met, the Runtime Filter can also gain benefits in other execution scenarios. In some embodiments, the memory access efficiency of the Runtime Filter can be improved to reduce the memory access efficiency ratio mentioned above, thereby improving the system's performance gains. Considering complex queries on large-scale datasets, the hash table of Hashjoin cannot be fully accommodated in the CPU L1 / L2 / L3 cache. The probing process is equivalent to random access to main memory, and the memory access efficiency C = 1-240 cycles. Traditional Bloom Filter implementations are built based on the number of elements and fault tolerance, according to the classic Bloom Filter calculation formula, which derives the number of mappings K and the memory size. During reading, the K mappings of a record are randomly scattered to non-adjacent locations in the memory region. Memory access cannot utilize features such as prefetching and cache lines, and the memory access efficiency is K times that of the data region (L1 / L2 / L3 / main memory). Therefore, if accessing the Bloom filter also involves K random accesses to main memory, the memory access efficiency C2 = K * 240 cycles, where K >= 1 and C2 > Ci, is unacceptable. Furthermore, the Bloom filter must at least fit in the L3 cache, meaning it must be accommodated by L3. In recent years, the typical L3 cache size in mainstream CPU architectures is 10MB-64MB, with an access speed of 30-80 cycles, shared by multiple physical CPU cores within a single socket. Additionally, the number of mappings, K, cannot be too large. In summary, to improve upon these issues, in this embodiment, the filter can also use a blocking Bloom filter, which offers better memory access performance.Next, the construction process and memory access efficiency of the Blocked Bloom Filter will be explained: Specifically, when constructing the Blocked Bloom Filter, firstly, a long array with a size approximately equal to (number of elements / 8) is used to construct the memory region. Each record undergoes K=2 mappings, and the mapping location of each record spans no more than 512 bits within the memory region. Furthermore, when reading the Blocked Bloom Filter, the mapping count for each record remains K=2. The two mapped bits of each record are read and checked. Under the new Blocked Bloom Filter algorithm, the number of memory accesses can be approximated as 1. This is because, although the number of mappings is fixed at 2, the target location span is less than 512 bits. Thanks to the CPU's cache and prefetch capabilities, it is highly likely that the data can be contained within a single cache line in a single memory access. Correspondingly, for Blocked Bloom Filters, after completing scheduling and hash table construction for the Hash Join operator, an adaptive Runtime Filter switch can be designed by combining the performance benefit formula mentioned above and context information. Selectivity can be estimated using the database's own statistics and cost estimation model; the fpp value is between 0.03 and 0.1, having a relatively small impact; the sizes of Ci and C2 depend on the memory space occupied by the hash table and Runtime Filter respectively, and the sizes of both the hash table and Runtime Filter depend on the number of rows in the Build phase. In this embodiment, the hash table memory space occupies next_power_of_2(r / Load Factor) * 4 bytes. According to the blocked Bloom Filter algorithm, the runtime filter memory space occupies next_power_of_2(r / 8) * 8 bytes. Where next_power_of_2(x) represents the smallest integer power greater than x (2), r represents the number of rows in the build, and Load Factor is a hash table parameter, typically taken as 0.25-0.75. Ci depends on the hash table memory size and the CPU architecture information of the physical machine.It should be noted that if the hash table memory usage exceeds the size of the L3 / L2 / L1 cache, then Ci approximates the main memory access speed (250 cycles), L3 access speed (50 cycles), and L2 access speed (15 cycles), respectively, and so on. Similarly, C2 is calculated as follows: if the runtime filter memory usage exceeds the size of the L3 / L2 / L1 cache, then C2 approximates the main memory access speed (250 cycles), L3 access speed (50 cycles), and L2 access speed (15 cycles), respectively, and so on. Furthermore, based on the Blocked Bloom Filter algorithm implementation, since the memory access span after mapping each record is less than 512 bits, K can be approximated as 1, as described above. In summary, after constructing the hash table, the values ​​of Ci, C2, and Selectivity are calculated using existing information. These values ​​are then substituted into the performance gain formula to see if they are satisfied. If satisfied, the runtime filter can be enabled. oThis forms an adaptive runtime filter switching mechanism. Additionally, some further adjustments are needed for Broadcast and Local modes. For example, in Broadcast mode, the runtime filter is shared across all partitions, so only a single runtime filter needs to satisfy the performance benefit relationship. In Local mode, the runtime filter is divided into multiple local partial runtime filters within each partition. When calculating the C2 value in the performance benefit relationship, it's necessary to consider whether the total number of rows across all partitions on the build side can be accommodated in the L1 / L2 / L3 cache. In some embodiments, the cardinality and selectivity estimation of the join operator often exhibit significant deviations, which are difficult to avoid in databases. Therefore, when there is a large difference between the selectivity and the actual value, we need to calculate a rough boundary for the performance benefit model to minimize misjudgments. When the Selectivity estimate is inaccurate, we need to consider the following three scenarios to roughly define the boundary thresholds: Scenario 1: The number of rows on the build side is too large, and both Ci and C2 are equal to the main memory access latency. In this case, it is unnecessary to enable the Runtime Filter because it is close to 1, which cannot satisfy the size relationship in the benefit model. The memory space occupied by the Runtime Filter will exceed the L3 Cache size. We can work backward to estimate the number of rows on the build side at this time and use the number of rows on the build side as the threshold for judgment. For example, if we know that the physical machine L3 Cache Size = 16MB, then next_power_of_2(r / 8) * 8Bytes > 16 MB, where r is the number of rows on the build side, which is between 10 million and 20 million. Therefore, we can roughly agree that the total number of rows on the build side is between 1 and 20 million as one of the conditions for enabling the Runtime Filter. Scenario 2: The number of rows on the build side is too small, and the differences between Ci and C2 are not significant, approaching 1. The memory access latency G is already very small, and the hash table can often be accommodated in the LI Cache. The size of the number of rows on the build side can be inferred from the hash table's memory usage.For example, if we know the physical machine's ID Cache Size = 32KB and hash table Load_Factor = 0.5, then next_power_of_2(r / Load Factor) * 4Bytes < 32KB, where r is the number of rows on the build side, and r < 4096. We roughly agree that the lower bound of the number of rows on the build side is 4096, as one of the conditions for enabling the runtime filter. Scenario 3: The filtering performance is very poor, i.e., the selectivity is much larger than the estimated value. In this case, it is impossible to constrain the filtering performance by the number of rows on the build side, because the filtering performance on the probe side is not yet known when the runtime filter is built on the build side. Therefore, we can introduce a dynamic sampling and checking mechanism on the scan side to check the filtering performance of the first 5 to 10 chunks in each batch; if the filtering performance of the first 5 to 10 chunks is very poor, then the runtime filter calculation for this batch is terminated. Of course, by this time the Runtime Filter has already been built, and the performance loss during the building process and processing of these 5-10 chunks cannot be avoided. Referring to Figure 12, which is a schematic diagram of a filter building apparatus provided in an embodiment of this disclosure, the filter building apparatus 1200 includes: a generation module 1201, used to generate an operator chain corresponding to a data query plan based on the data query plan. The operator chain includes multiple linearly connected operators, including connection operators and scan operators; an acquisition module 1202, used to acquire filter building information corresponding to the operator chain based on the connection operators and scan operators. The filter building information includes the correspondence between the construction channels of the connection operators and the filtering channels of the scan operators in the operator chain; a determination module 1203, used to determine the target connection key of the filter to be built from the connection keys included in the operator chain based on the performance gain and / or performance overhead of the filter corresponding to the filter building information; and a construction module 1204, used to build a filter at the construction end of the target connection key. The target connection key is used to filter data based on the filter when executing the data query plan.In some optional embodiments, the acquisition module 1202 is specifically configured to: acquire first identification information of the construction key and second identification information of the probe key of the connection operator in the operator chain; for each connection key on the connection operator, based on the first identification information of the construction key on the connection key, obtain the construction channel in the filter construction information from the metadata of the connection operator; and, based on the second identification information of the probe key on the connection key, obtain the filter channel corresponding to the construction channel from the metadata of the scan operator; and acquire the filter construction information corresponding to the operator chain according to the construction channel and filter channel corresponding to the connection key contained in the operator chain. In some optional embodiments, the determining module 1203 is specifically used for at least one of the following: For each filter corresponding to the filter construction information, obtaining the performance gain of the filter; if the performance gain is not less than a preset performance gain, then determining the connection key corresponding to the filter as the target connection key; For each filter corresponding to the filter construction information, obtaining the performance overhead of the filter; if the performance overhead is not greater than a preset performance overhead, then determining the connection key corresponding to the filter as the target connection key; For each filter corresponding to the filter construction information, obtaining the performance overhead and performance gain of the filter; if the performance overhead is less than the performance gain, then determining the connection key corresponding to the filter as the target connection key. In some optional embodiments, the determining module 1203 is specifically used for: For each filter corresponding to the filter construction information, obtaining the performance overhead and performance gain of the filter; obtaining the memory access efficiency ratio corresponding to the filter based on the performance overhead and performance gain; obtaining the screening rate of the connection operator after false positive correction; determining whether the connection key corresponding to the filter is the target connection key based on the memory access efficiency ratio and / or the screening rate. In some optional embodiments, the determining module 1203 is specifically used for at least one of the following: if the memory access efficiency ratio is less than the screening rate, determine that the join operator corresponding to the filter is a target join key that meets the preset requirements; if the memory access efficiency ratio is greater than or equal to the screening rate, determine that the join operator corresponding to the filter is not a target join key; if the screening rate is greater than or equal to the preset screening rate, determine that the join key corresponding to the filter is a target join key; if the screening rate is less than the preset screening rate, determine that the join operator corresponding to the filter is not a target join key; if the difference between the memory access efficiency ratio and the preset memory access efficiency ratio is less than a first preset value, determine whether the join key corresponding to the filter is a target join key based on the number of rows in the construction end.In some optional embodiments, the determining module 1203 is specifically used to: determine the connection key corresponding to the filter as the target connection key if the number of rows in the construction end is greater than a second preset value or less than a third preset value; determine the connection key corresponding to the filter as not the target connection key if the number of rows in the construction end is less than or equal to the second preset value, or greater than or equal to the third preset value. In some optional embodiments, the determining module 1203 is further used to: determine the parallelism of each operator chain; set the same number of threads as the parallelism of the operator chain according to the parallelism, and the threads are used to execute the connection operators in the operator chain. In some optional embodiments, the construction module 1204 is specifically used to: determine the data partitioning mode corresponding to the target connection key; construct a filter at the construction end of the target connection key according to the data partitioning mode of the target connection key; wherein, the data partitioning mode includes a first partitioning mode and a second partitioning mode, the first partitioning mode is used to characterize that each partition of the construction end of the target connection key corresponds to the same hash table, and the second partitioning mode is used to characterize that each partition of the construction end of the target connection key corresponds to a different hash table. In some optional embodiments, the construction module 1204 is specifically used for: if the target connection key is a first partition mode, constructing a filter component corresponding to each thread for the target connection key, and merging the filter components of the threads corresponding to the target connection key to obtain the filter corresponding to the target connection key; if the target connection key is a second partition mode, constructing a local filter corresponding to each partition for the target connection key to obtain the filter corresponding to the target connection key. In some optional embodiments, the filter is at least one of a Bloom filter and a blocking Bloom filter. It should be noted that the filter construction device provided in this disclosure is used to execute the technical solutions in the aforementioned corresponding method embodiments, and its implementation principle and technical effects are similar, and will not be repeated here. Referring to FIG13, FIG13 is a schematic diagram of the structure of a data query device provided in an embodiment of this disclosure. As shown in Figure 13, the data query apparatus 1300 provided in this embodiment includes: a filter construction apparatus 1200, used to construct a filter in the operator chain corresponding to the data query plan in response to the data query plan; and a query module 1301, used to execute the operator chain corresponding to the data query plan, and filter data based on the filter to obtain the data query result corresponding to the data query plan.In some optional embodiments, the query module 1301 is further configured to: for each filter constructed in the operator chain, obtain the filtering effect of the filter in the process of executing the data query plan; if the filtering effect is not less than a threshold, then continue to use the filter for data filtering in the process of executing the data query plan; if the filtering effect is less than the threshold, then stop using the filter for data filtering in the process of executing the data query plan. It should be noted that the functions performed by each module in the data query device provided in this disclosure correspond to the technical solutions in the aforementioned corresponding method embodiments, and their implementation principles and technical effects are similar, and will not be repeated here. Referring to FIG14, FIG14 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. As shown in FIG14, the electronic device 1400 includes: at least one processor 1401 (only one processor is shown in FIG14); and a memory 1402 communicatively connected to at least one processor. The memory 1402 stores instructions that can be executed by at least one processor 1401, and the instructions are executed by at least one processor 1401 to enable the electronic device 1400 to execute the technical solutions in any of the aforementioned method embodiments. Optionally, memory 1402 can be either standalone or integrated with processor 1401. Memory 1402 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Processor 1401, coupled to memory 1402, is used to execute the computer program in memory 1402 to perform the technical solutions of any of the filter construction method embodiments described above, and / or, to perform the technical solutions of any of the data query method embodiments described above. It should be understood that the processor 1401 mentioned above can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor.The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. Optionally, when the memory 1402 is a device independent of the processor 1401, the electronic device 1400 further includes: a bus 1403 for connecting the memory 1402 and the processor 1401. The bus 1403 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses in the accompanying drawings are not limited to only one bus or one type of bus. It should be noted that only some components are schematically shown in Figure 14, and it does not mean that the electronic device 1400 only includes the components shown in Figure 14. In some embodiments, the electronic device 1400 further includes: a firewall, a load balancer, a communication component, a power supply component, and other components. Optionally, the electronic device 1400 can be a mobile terminal, server, or other device. The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies. It should be noted that the electronic device 1400 provided in this disclosure embodiment can execute the technical solutions of any of the aforementioned filter construction method embodiments, and / or execute the technical solutions of any of the aforementioned data query method embodiments; their implementation principles and technical effects are similar and will not be repeated here. Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.Furthermore, this disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the technical solutions of any of the aforementioned filter construction method embodiments, and / or execute the technical solutions of any of the aforementioned data query method embodiments. One exemplary embodiment is that the storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device. Accordingly, embodiments of this disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the technical solutions of any of the aforementioned filter construction method embodiments, and / or executes the technical solutions of any of the aforementioned data query method embodiments. Accordingly, this disclosure also provides a chip, including: a processing module and a communication interface, wherein the processing module is capable of executing the technical solutions of any of the filter construction method embodiments in the foregoing method embodiments, and / or executing the technical solutions of any of the foregoing data query method embodiments. Further, the chip also includes a storage module (e.g., a memory), the storage module being used to store instructions, the processing module being used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module causing the processing module to execute the technical solutions in the foregoing method embodiments. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure. If the function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.Based on this understanding, the technical solution of this disclosure, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "including a..." does not exclude the presence of other identical elements in the process, method, product, or device that includes the element. It should be noted that some processes described in the above embodiments and figures include multiple operations appearing in a specific order; however, it should be clearly understood that these operations may not be executed in the order they appear herein or may be executed in parallel. The operation numbers, such as S201, S202, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

Claims

Claims 1. A filter construction method, wherein, include: A chain of operators corresponding to the data query plan is generated based on the data query plan. The chain of operators includes multiple linearly connected operators, including connection operators and scan operators. Based on the connection operators and the scan operators, filter construction information corresponding to the operator chain is obtained. The filter construction information includes the correspondence between the construction channels of the connection operators and the filtering channels of the scan operators in the operator chain. Based on the performance gains and / or performance costs of the filters corresponding to the filter construction information, the target connection key for building the filter is determined from the connection keys included in the operator chain. A filter is built at the construction end of the target join key, and the target join key is used to filter data based on the filter when executing the data query plan.

2. The filter construction method according to claim 1, wherein, The step of obtaining filter construction information corresponding to the operator chain based on the connection operator and the scan operator includes: obtaining first identification information of the construction key and second identification information of the probe key of the connection operator in the operator chain; for each connection key on the connection operator, obtaining the construction channel in the filter construction information from the metadata of the connection operator based on the first identification information of the construction key on the connection key; and obtaining the filter channel corresponding to the construction channel from the metadata of the scan operator based on the second identification information of the probe key on the connection key; and obtaining the filter construction information corresponding to the operator chain based on the construction channel and filter channel corresponding to the connection key included in the operator chain.

3. The filter construction method according to claim 1 or 2, wherein, The step of determining the target connection key for constructing a filter from the connection keys included in the operator chain based on the performance gains and / or performance costs of the filters corresponding to the filter construction information includes at least one of the following: For each filter corresponding to the filter construction information, obtaining the performance gain of the filter; if the performance gain is not less than a preset performance gain, then determining the connection key corresponding to the filter as the target connection key; For each filter corresponding to the filter construction information, obtaining the performance cost of the filter; if the performance cost is not greater than a preset performance cost, then determining the connection key corresponding to the filter as the target connection key; For each filter corresponding to the filter construction information, obtaining the corresponding performance cost and performance gain of the filter; if the performance cost is less than the performance gain, then determining the connection key corresponding to the filter as the target connection key.

4. The filter construction method according to claim 1 or 2, wherein, The step of determining the target connection key for constructing a filter in the connection keys contained in the operator chain based on the performance gains and / or performance costs of the filters corresponding to the filter construction information includes: obtaining the corresponding performance costs and performance gains for each filter corresponding to the filter construction information; and obtaining the memory access efficiency ratio corresponding to the filter based on the performance costs and performance gains. 22 Obtain the filtering rate of the join operator after false positive correction; determine whether the join key corresponding to the filter is the target join key based on the memory access efficiency ratio and / or the filtering rate.

5. The filter construction method according to claim 4, wherein, The step of determining whether the join key corresponding to the filter is the target join key based on the memory access efficiency ratio and / or the screening rate includes at least one of the following: if the memory access efficiency ratio is less than the screening rate, the join operator corresponding to the filter is determined to be the target join key; if the memory access efficiency ratio is greater than or equal to the screening rate, the join operator corresponding to the filter is determined not to be the target join key; if the screening rate is greater than or equal to a preset screening rate, the join key corresponding to the filter is determined to be the target join key; if the screening rate is less than the preset screening rate, the join operator corresponding to the filter is determined not to be the target join key. If the difference between the memory access efficiency ratio and the preset memory access efficiency ratio is less than a first preset value, the connection key corresponding to the filter is determined to be the target connection key based on the number of rows in the construction end.

6. The filter construction method according to claim 5, wherein, The step of determining whether the connection key corresponding to the filter is the target connection key based on the number of rows in the construction end includes: if the number of rows in the construction end is greater than a second preset value or less than a third preset value, then the connection key corresponding to the filter is determined to be the target connection key; if the number of rows in the construction end is less than or equal to the second preset value, or if the number of rows in the construction end is greater than or equal to the third preset value, then the connection key corresponding to the filter is determined not to be the target connection key.

7. The filter construction method according to any one of claims 1 to 6, wherein, After generating at least one operator chain corresponding to the data query plan based on the data query plan, the method further includes: determining the parallelism of each operator chain; setting up a number of threads equal to the parallelism of the operator chain based on the parallelism, wherein the threads are used to execute the connection operators in the operator chain.

8. The filter construction method according to any one of claims 1 to 7, wherein, The step of building a filter at the construction end of the target join key includes: determining the data partitioning pattern corresponding to the target join key; and building a filter at the construction end of the target join key according to the data partitioning pattern of the target join key; wherein the data partitioning pattern includes a first partitioning pattern and a second partitioning pattern, the first partitioning pattern is used to characterize that each partition at the construction end of the target join key corresponds to the same hash table, and the second partitioning pattern is used to characterize that each partition at the construction end of the target join key corresponds to a different hash table.

9. The filter construction method according to claim 8, wherein, The step of constructing a filter corresponding to the target connection key based on the data partitioning pattern of the target connection key includes: if the target connection key is the first partitioning pattern, constructing a filter component corresponding to each thread for the target connection key, and merging the filter components of the threads corresponding to the target connection key to obtain the filter corresponding to the target connection key; If the target join key is the second partition pattern, for each partition corresponding to the target join key, a local filter corresponding to the partition is constructed to obtain the filter corresponding to the target join key.

10. The filter construction method according to any one of claims 1 to 9, wherein, The filter is at least one of a Bloom filter and a blockage Bloom filter.

11. A data query method, wherein, include: In response to a data query plan, a filter is constructed in the operator chain corresponding to the data query plan based on the filter construction method of any one of claims 1 to 10; Execute the operator chain corresponding to the data query plan, and filter the data based on the filter to obtain the data query result corresponding to the data query plan.

12. The data query method according to claim 11, wherein, It also includes: for each filter constructed in the operator chain, obtaining the filtering effect of the filter on data filtering during the execution of the data query plan; if the filtering effect is not less than a threshold, then the filter continues to be used for data filtering during the execution of the data query plan; if the filtering effect is less than the threshold, then the filter is stopped from being used for data filtering during the execution of the data query plan.

13. A filter construction apparatus, wherein, include: A generation module generates an operator chain corresponding to the data query plan based on the data query plan. The operator chain includes multiple linearly connected operators, including connection operators and scan operators. An acquisition module is used to acquire filter construction information corresponding to the operator chain based on the connection operators and the scan operators. The filter construction information includes the correspondence between the construction channels of the connection operators and the filter channels of the scan operators in the operator chain. A determination module is used to determine the target connection key for building a filter in the connection keys contained in the operator chain, based on the performance benefits and / or performance costs of the filter corresponding to the filter construction information; a construction module is used to build a filter at the construction end of the target connection key, wherein the target connection key is used to filter data based on the filter when executing the data query plan.

14. A data query device, wherein, include: The filter construction apparatus of claim 13 is configured to construct a filter in the operator chain corresponding to the data query plan in response to the data query plan; The query module is used to filter data based on the filter when executing the operator chain corresponding to the data query plan.

15. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the filter construction method as described in any one of claims 1 to 10, and / or implement the data query method as described in claims 11 to 12.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the filter construction method of any one of claims 1 to 10, and / or implements the data query method of claims 11 to 12. 25

Images