Method and apparatus for dynamic filtering of distributed database that performs hash join operations on data distributed and stored on multiple servers

Abstract

A distributed database dynamic filtering method according to various embodiments of the present invention may include the steps of: identifying a hash join operation node by traversing a query execution plan tree; identifying a filter target node by searching the right subtree of the hash join operation node; generating a filter identifier corresponding to the filter target node; recording the filter identifier in the left input table of the hash join operation node; distributing the execution plan tree to a plurality of servers; generating or updating a hash value-based dynamic filter based on records in the left input table; applying the generated dynamic filter to the filter target node by progressively propagating it downward along the right subtree of the hash join operation; configuring a global filter by mutually transmitting and merging filters generated at each server when the filter target node includes a communication node that performs inter-server communication; and removing join target records by applying the dynamic filter prior to performing a physical data scan on the right input table of the hash join operation node.

Description

Technology Field

[0001] The present invention relates to a distributed database dynamic filtering technology. More specifically, the present invention relates to a technology for dynamic filtering of a distributed database by performing a hash join operation on data distributed and stored on a plurality of servers. Background Technology

[0002] With the recent rapid increase in the scale of data processing, distributed database systems that store data across multiple servers and process queries in parallel are being widely used. In distributed database environments, join operations between large-scale tables are performed frequently; in particular, join methods such as hash joins require loading massive amounts of data into memory to perform matching, so processing efficiency significantly impacts overall system performance. However, in distributed environments, data to be joined is often stored on different servers, leading to the problem of having to transmit large volumes of data over a network prior to the join. This is a major cause of increased network load and overall query delays.

[0003] To mitigate these issues, filtering techniques have been proposed to remove unnecessary data in advance of the join stage. However, in conventional technologies, the decision to create filters was often fixed during the execution planning stage, and the merging and application structures of filters between distributed servers were frequently not sufficiently considered. Particularly in execution plans involving aggregation operations, statistical prediction errors could reduce the effectiveness of filters or even cause performance degradation. Furthermore, the lack of dynamic control structures based on filter application results presented a limitation, making it difficult to properly assess and adjust the validity of filters during actual execution. The problem to be solved

[0004] The technical problem to be solved through some embodiments of the present invention is to provide a distributed database dynamic filtering method and apparatus for performing a hash join operation on data distributed across multiple servers, wherein unnecessary records are removed in advance at a stage prior to the join operation, thereby reducing the amount of data to be joined and reducing the processing time and memory usage required for the join operation.

[0005] The technical problem to be solved through some embodiments of the present invention is to provide a distributed database dynamic filtering method and apparatus that performs hash join operations on data distributed and stored across multiple servers, which can reduce network transmission volume between servers and improve overall distributed query processing efficiency by performing more sophisticated filtering before data redistribution between servers in a distributed environment.

[0006] The technical problem to be solved through some embodiments of the present invention is to provide a distributed database dynamic filtering method and apparatus that performs hash join operations on data distributed and stored on multiple servers, which rationally controls whether to apply a filter according to the characteristics of an execution plan, thereby preventing inefficient application of the filter due to statistical prediction errors and maintaining stable performance.

[0007] The technical problem to be solved through some embodiments of the present invention is to provide a distributed database dynamic filtering method and apparatus that performs hash join operations on data distributed and stored in multiple servers that are sequentially input, which can dynamically adjust the filter strategy according to the actual execution results, thereby reducing unnecessary filter operations when the filter effect is negligible and improving the utilization efficiency of overall system resources.

[0008] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0009] A distributed database dynamic filtering method for performing a hash join operation on data distributed and stored on a plurality of servers according to a few embodiments of the present invention to solve the above technical problem comprises: a step of generating a query execution plan tree of a distributed database and then traversing the query execution plan tree to identify a hash join operation node; a step of recursively searching the right subtree of the hash join operation node to identify a node to which a filter is applied; a step of generating a filter identifier corresponding to the node to which a filter is applied; a step of recording the generated filter identifier in the left input table of the hash join operation node; a step of distributing the execution plan tree with the filter identifier inserted to the plurality of servers; a step of generating or updating a hash value-based dynamic filter based on a record in the left input table during the hash table construction step of the hash join operation; a step of applying the generated dynamic filter to the node to which a filter is applied by stepwise downward propagating it along the right subtree of the hash join operation; and, if the node to which a filter is applied includes a communication node that performs inter-server communication, a step of configuring a global filter by mutually transmitting and merging filters generated on each server. and may include a step of pre-removing join target records by applying the dynamic filter prior to performing a physical data scan on the right input table of the hash join operation node.

[0010] In some embodiments, the step of generating the filter identifier may include generating the filter identifier when the ratio of the number of output rows of the right subtree to the number of expected join result rows exceeds a preset first threshold.

[0011] In some embodiments, the distributed database dynamic filtering method may further include the step of adding an intent identifier to the filter identifier when the filter-applied target node includes an aggregation node; and the step of invalidating the dynamic filter applied to the aggregation node when the cumulative number of rows processed at the aggregation node exceeds a second threshold.

[0012] In some embodiments, the dynamic filter may be generated based on the hash value of each record of the left input table during the hash table construction step of the hash join operation.

[0013] In some embodiments, the dynamic filter may include at least one of a Bloom filter, a minimum-maximum filter, or a set inclusion condition-based filter.

[0014] In some embodiments, the step of configuring the global filter may include receiving filters generated at each server and merging them by logical operation.

[0015] In some embodiments, to configure the global filter, the server generating the filter may pre-establish a communication channel with the server receiving the filter and transmit the filter synchronously through the communication channel.

[0016] In some embodiments, if the data reduction rate after applying the dynamic filter is less than a preset threshold, the step of disabling the dynamic filter for subsequent join operations may be further included.

[0017] A distributed database dynamic filtering device that performs a hash join operation on data distributed and stored on a plurality of servers according to some embodiments of the present invention includes a memory that stores one or more instructions; A distributed database dynamic filtering device comprising at least one processor, wherein the at least one processor executes one or more instructions, generates a query execution plan tree of the distributed database, traverses the execution plan tree to identify a hash join operation node, recursively searches the right subtree of the hash join operation node to identify a filter application target node, generates a filter identifier corresponding to the filter application target node, records the generated filter identifier in the left input table of the hash join operation node, distributes the execution plan tree with the inserted filter identifier to the plurality of servers, generates or updates a hash value-based dynamic filter based on the records of the left input table during the hash table construction step of the hash join operation, applies the generated dynamic filter to the filter application target node by stepwise downward propagation along the right subtree of the hash join operation, and if the filter application target node includes a communication node that performs inter-server communication, constructs a global filter by mutually transmitting and merging the filters generated at each server, and applies the dynamic filter prior to performing a physical data scan on the right input table of the hash join operation node. You can apply this to pre-remove the join target records.

[0018] In some embodiments, the distributed database dynamic filtering device can generate a filter identifier when the ratio of the number of output rows of the right subtree to the number of expected join result rows exceeds a preset first threshold by the at least one processor executing the one or more instructions. Effects of the invention

[0019] A distributed database dynamic filtering method and apparatus for performing a hash join operation on data distributed and stored on a plurality of servers according to some embodiments of the present invention can reduce the processing time and memory usage required for the join operation by removing unnecessary records in advance at a stage prior to performing the join, thereby reducing the amount of data to be joined.

[0020] A distributed database dynamic filtering method and apparatus for performing hash join operations on data distributed and stored on a plurality of servers according to some embodiments of the present invention can reduce network transmission volume between servers and improve overall distributed query processing efficiency by performing more sophisticated filtering before data redistribution between servers in a distributed environment.

[0021] A distributed database dynamic filtering method and apparatus for performing hash join operations on data distributed and stored on a plurality of servers according to some embodiments of the present invention can reasonably control whether to apply a filter according to the characteristics of an execution plan, and accordingly, prevent inefficient application of a filter due to statistical prediction errors and maintain stable performance.

[0022] A distributed database dynamic filtering method and apparatus for performing hash join operations on data distributed and stored on a plurality of servers according to some embodiments of the present invention can dynamically adjust the filter strategy according to the actual execution result, and accordingly, can reduce unnecessary filter operations when the filter effect is negligible and can improve the utilization efficiency of the entire system resources.

[0023] The effects according to some embodiments of the present invention are not limited to those exemplified above, and a wider variety of effects are included in the present invention. Brief explanation of the drawing

[0024] FIG. 1 is a diagram showing an exemplary environment in which a distributed database dynamic filtering method and apparatus according to one embodiment of the present invention may be applied. FIG. 2 is a schematic diagram showing the location of the filter identifier generation and application target node in a query execution plan tree including a hash join operation according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating a distributed database dynamic filtering method according to one embodiment of the present invention. FIG. 4 is a diagram showing a specific operation for generating a filter identifier according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the operation of invalidating a dynamic filter applied to an aggregation node when the number of accumulated rows exceeds a threshold according to one embodiment of the present invention. FIG. 6 is a diagram illustrating an operation to disable a dynamic filter for a subsequent join operation when the data reduction rate after applying a dynamic filter according to an embodiment of the present invention is less than a preset standard. FIG. 7 is a block diagram illustrating the configuration of a device according to embodiments of the present invention. Specific details for implementing the invention

[0025] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.

[0026] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or substitutions that fall within the technical concept.

[0027] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0028] Furthermore, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in this invention must be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0029] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0030] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] Throughout this specification, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, the singular form used in this specification includes the plural form unless specifically stated otherwise in the text. Additionally, the expression “at least one of a, b, and c” described throughout this specification may encompass ‘a alone,’ ‘b alone,’ ‘c alone,’ ‘a and b,’ ‘a and c,’ ‘b and c,’ or ‘a, b, and c all.’

[0032] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0033] Additionally, terms such as “…part,” “…module,” etc., as described in this specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. Furthermore, embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, embodiments of the present invention may employ integrated circuit configurations such as memory, processing, logic, and look-up tables, which can execute various functions under the control of one or more microprocessors or other control devices.

[0034] Each block of the process flow diagrams attached to this specification and combinations of the flow diagrams may be executed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in the flow diagram block(s).

[0035] These computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, and the instructions stored in said computer-available or computer-readable memory may also produce a manufactured item containing instruction means that performs the function described in the flowchart block(s).

[0036] Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0037] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Furthermore, in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0038] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.

[0039] In one embodiment, a distributed database system can process user queries in parallel while data is distributed and stored across multiple servers. The multiple servers may be able to communicate with each other, and each server may perform scan processing, join processing, and filter generation processing on locally stored data. The query may be a query based on a structured query language, and a query execution plan tree may be generated for query processing. The query execution plan tree may represent the relationships between operation nodes in a tree structure and may include, for example, operation nodes including join operations, aggregation operations, scan operations, data exchange operations, or combinations thereof.

[0040] In one embodiment, the query execution plan tree may include a hash join operation node, and the hash join operation node may be defined in a form having a left input and a right input. Here, the left input may be used as an input for constructing a hash table, and the right input may be used as an input for performing a match using the constructed hash table. The left input or the right input may each be represented by one or more subtrees, and for convenience of explanation, the subtree corresponding to the left input connected to the hash join operation node may be referred to as the “left input subtree,” and the subtree corresponding to the right input may be referred to as the “right subtree.” Additionally, “scan” may include a physical data reading operation that reads table data from a storage medium or memory.

[0041] In one embodiment, the present invention may utilize a dynamic filter to reduce the data throughput associated with a hash join operation. The dynamic filter may be generated or updated based on a join key obtained from a left input or a corresponding hash value during the execution of the hash join operation. The dynamic filter may include condition information for determining in advance whether a record scanned from the right input side has the potential to be joined, and the condition information may be implemented in the form of a probability-based expression, a range-based expression, a set-based expression, or a combination thereof. For example, the dynamic filter may include a bit sequence-based expression for determining whether a join key is included, may include an expression using a range of minimum and maximum values ​​of the join key, or may include an expression using an inclusion condition for a specific set of keys.

[0042] In one embodiment, a filter identifier may be used to link the creation and application of a dynamic filter with an execution plan. The filter identifier can function as identification information indicating a specific hash join operation node and the filter application location within its right subtree. For example, after identifying the hash join operation node by traversing the query execution plan tree, the node to which the filter is applied can be identified by recursively searching the right subtree. The node to which the filter is applied may be a node containing a scan operation within the right subtree, a node containing an operation that combines multiple paths, or a node containing a communication-related operation that performs data exchange between servers. The filter identifier may be recorded as metadata related to the left input so as to be linked with the processing of the left input side, and accordingly, the dynamic filter created during the hash table construction phase can be transmitted to and applied at the corresponding location in the right subtree.

[0043] In one embodiment, a dynamic filter can be propagated downward stepwise along the right subtree and applied prior to the physical data scan of the right input table. Accordingly, records scanned from the right input that have a low probability of forming a join can be removed before the join is performed, thereby reducing join throughput, memory access volume, and data movement between servers. Additionally, if a node performing inter-server communication is included within the right subtree, a global filter can be constructed by mutually transmitting and merging filters generated from each server. The global filter can function as an integrated filter reflecting observation results from multiple servers, and by applying the global filter again to the scan path on the right input side of each server, filtering precision in a distributed environment can be improved.

[0044] In one embodiment, the creation or application of a dynamic filter may be conditionally controlled based on the characteristics of the execution plan. For example, a filter identifier may be created only when a preset threshold condition is satisfied based on the ratio of the number of output rows in the right subtree to the number of expected join result rows. Additionally, if the node to which the filter is applied includes an aggregation operation, an intent identifier corresponding to the aggregation operation may be added to distinguish the operation for that path. Furthermore, if the cumulative number of rows processed in the aggregation path exceeds a preset threshold condition, the dynamic filter applied to the aggregation path may be disabled to suppress inefficiency caused by the filter application. Additionally, if the data reduction rate after applying the dynamic filter falls short of a preset standard, the dynamic filter may be disabled for subsequent join operations to reduce unnecessary filter processing overhead.

[0045] The above functions may be implemented in a combined form of hardware and software. For example, a server may include a processor and memory, and by executing one or more instructions stored in memory by the processor, the generation and traversal of a query execution plan tree, the performance of hash join operations, the generation and recording of filter identifiers, the creation / update / propagation / merge / application of dynamic filters, and invalidation or deactivation operations based on critical conditions may be performed.

[0046] Below, under these premises, the specific configuration of a distributed database dynamic filtering method and apparatus according to one embodiment of the present invention will be described sequentially with reference to the attached drawings.

[0047] FIG. 1 is a diagram showing an exemplary environment in which a distributed database dynamic filtering method and apparatus according to one embodiment of the present invention may be applied.

[0048] Referring to FIG. 1, in one embodiment, a distributed database dynamic filtering method and apparatus may be implemented in an environment comprising a query plan generation module (10), a plurality of servers (20, 30, 40), a global filter merging module (50), and a communication channel (60). The query plan generation module (10) may receive a query input from a user and generate a query execution plan tree, and may analyze hash join operation nodes and right subtree structures included in the query execution plan tree. Based on the analysis results, the query plan generation module (10) may reflect filter application target nodes and filter identifier information in the execution plan and distribute the execution plan to a plurality of servers (20, 30, 40).

[0049] In one embodiment, Server A (20), Server B (30), and Server C (40) can each include a data storage space and distribute data in the form of a table or partition. Server A (20) can extract a join key or hash value from a record included in the left input table of a hash join through a filter generation function (21), and can generate or update a dynamic filter based on the join key or hash value. Server A (20) can perform hash table construction through a join execution function (22), and can generate a transmission trigger to propagate the dynamic filter generated during the hash table construction process to the right input side. Server A (20) can perform an actual data reading operation from the right input table or right subtree through a scan processing function (23), and can pre-exclude records with a low probability of joining by applying a dynamic filter before the data reading operation.

[0050] In one embodiment, Server B (30) and Server C (40) can each generate or update a dynamic filter based on the left input data or a divided join key set they are responsible for through a filter generation function (31, 41). Server B (30) and Server C (40) can perform some steps of a hash join in parallel through a join execution function (32, 42), and can apply a dynamic filter during the right input data scanning process through a scan processing function (33, 43). If communication-related operations are included in the right subtree, Server B (30) and Server C (40) can adjust the timing of transmission and application of the dynamic filter before and after the communication-related operations.

[0051] In one embodiment, the global filter merging module (50) can receive dynamic filters generated from a plurality of servers (20, 30, 40) and merge them into a single global filter. The global filter merging module (50) can perform merging by logical operation if the server-specific dynamic filter includes a bit sequence-based representation, and can perform merging according to update rules for minimum and maximum values ​​if the server-specific dynamic filter includes a range-based representation. The global filter merging module (50) can perform merging according to merging rules for a set of inclusion conditions if the server-specific dynamic filter includes a set-based representation. The global filter merging module (50) can retransmit the merged global filter to the plurality of servers (20, 30, 40) so that it is applied commonly prior to the right input data scan.

[0052] In one embodiment, the communication channel (60) may be used for distributing execution plans between the query plan generation module (10) and a plurality of servers (20, 30, 40), and may be used for transmitting filters between servers or between a server and a global filter merging module (50). The communication channel (60) may include a wired or wireless network, inter-process communication, a message queue, or a remote call interface, and may operate synchronously or asynchronously depending on the implementation environment. The communication channel (60) may guarantee the order of filter transmission or control the update order by including filter version information, thereby preventing old filters from being applied.

[0053] In one embodiment, the server (20, 30, 40) can determine the target for creation, propagation path, and application location of a dynamic filter by referring to filter identifier information included in the execution plan distributed from the query plan generation module (10). The server (20, 30, 40) can propagate the dynamic filter downward stepwise in the direction of the right subtree, and if there is a communication-related operation in the middle of downward propagation, it can continue downward propagation after configuring a global filter through the global filter merging module (50). The server (20, 30, 40) can reduce the amount of data in the stage prior to the join operation by applying a dynamic filter or a global filter before performing a physical data scan on the right input table. The server (20, 30, 40) can adjust whether to apply the filter in the subsequent join operation according to the result of the filter application, and can reduce overhead by disabling the filter if it is determined that the filter effect is low.

[0054] FIG. 2 is a schematic diagram showing the location of the filter identifier generation and application target node in a query execution plan tree including a hash join operation according to one embodiment of the present invention.

[0055] Referring to FIG. 2, in one embodiment, the query execution plan tree may include a hash join node (70), and the hash join node (70) may receive a left input node (71) and a right subtree (72) as inputs to produce a join result. The hash join node (70) may construct a hash table based on records input from the left input node (71) and may generate a join result by matching records input from the right subtree (72) with the hash table. The left input node (71) may provide an input including the join key of the join target records, and the right subtree (72) may include an operation path in which candidates for the join target records are scanned or preprocessed.

[0056] In one embodiment, the filter identifier (73) may be generated as information indicating the location where the dynamic filter is to be applied within the right subtree (72) of the hash join node (70) and the association regarding the creation / propagation / application of the dynamic filter. The filter identifier (73) may be generated, for example, by identifying the hash join node (70) during the query plan generation process and then recursively searching the right subtree (72) to determine the filter application target (74). The filter application target (74) may be the scan node (75) itself located below the right subtree (72), may be an intermediate operation node for transmitting the filter to the scan node (75), or, if communication-related operations for performing data exchange between servers are included, may be set as a location before or after the communication-related operations.

[0057] In one embodiment, the filter identifier (73) may be recorded to be associated with the left input node (71), and the “filter identifier recording” illustrated in FIG. 2 may include the operation of storing the filter identifier (73) in the left input node (71) or in the input table meta-information referenced by the left input node (71). Accordingly, in the process of the hash join node (70) constructing a hash table using the records of the left input node (71), the creation time and propagation path of the dynamic filter can be determined by referencing the filter identifier (73). Additionally, the filter identifier (73) may be separated by each hash join node when there are multiple hash join nodes, and may be separated by each filter application target (74) when there are multiple filter application targets (74) within the same right subtree (72).

[0058] In one embodiment, the right subtree (72) may include a flow of operations up to the scan node (75), and the scan node (75) may perform the operation of physically reading records from the right input table. The filter application target (74) may be defined as a location that restricts the flow of records delivered to the scan node (75), and the dynamic filter may be evaluated at the filter application target (74) to determine whether the records read by the scan node (75) are delivered to a subsequent stage. For example, the dynamic filter may pass or block records based on the hash value of the join key, and if the records are blocked, the records may be processed so that they are not delivered to the hash join node (70).

[0059] In one embodiment, the filter identifier (73) may also indicate parameters related to the type, version, scope, or merging method of the dynamic filter to be used in the filter application target (74). The filter identifier (73) may include a flag indicating whether the dynamic filter needs to be merged into a global filter, and if global filter merging is required, it may include settings related to the filter's transmission and reception targets, merging order, or synchronization method. Additionally, the filter identifier (73) may also include an intent identifier or a path identifier indicating if an aggregation node is included in the right subtree (72), and accordingly, an invalidation condition based on the cumulative number of rows may be applied to the aggregation path.

[0060] In one embodiment, the query execution plan tree can be updated during execution, and the filter identifier (73) can be reused during the process of updating or reusing the execution plan. For example, when the same or similar query is executed repeatedly, mapping information regarding the filter identifier (73) and the filter application target (74) can be cached so that it can be quickly referenced in subsequent executions. Additionally, if the data reduction rate after applying a dynamic filter in the filter application target (74) is evaluated to be below a threshold, the filter identifier (73) can be used to indicate that the dynamic filter for that path should be disabled, thereby reducing unnecessary filter evaluations in subsequent join operations.

[0061] FIG. 3 is a flowchart illustrating a distributed database dynamic filtering method (300) according to one embodiment of the present invention. Referring to FIG. 3, a distributed database dynamic filtering method (300) comprises the steps of: generating a query execution plan tree of a distributed database; traversing the query execution plan tree to identify a hash join operation node (S310); recursively searching the right subtree of the hash join operation node to identify a node to which a filter is applied (S320); generating a filter identifier corresponding to the node to which a filter is applied (S330); recording the generated filter identifier in the left input table of the hash join operation node (S340); distributing the execution plan tree with the filter identifier inserted to a plurality of servers (S350); generating or updating a hash value-based dynamic filter based on the records of the left input table in the hash table construction step of the hash join operation (S360); applying the generated dynamic filter to the node to which a filter is applied by stepwise downward propagating it along the right subtree of the hash join operation (S370); and, if the node to which a filter is applied includes a communication node that performs inter-server communication, configuring a global filter by mutually transmitting and merging the filters generated at each server. It may include a step (S380) and a step (S390) of applying a dynamic filter to pre-remove join target records before performing a physical data scan on the right input table of the hash join operation node.

[0062] In one embodiment, steps S310 through S390 may be processed cooperatively by a computing device performing a query plan generation function and a plurality of servers. For example, the device performing the query plan generation function may parse a query input by a user to generate a logical plan, convert the logical plan into a physical plan to generate a query execution plan tree, and serialize and distribute the query execution plan tree into a form executable by each server. The device performing the query plan generation function may be implemented in the form of a coordinator node, a master node, or a front-end process, and the plurality of servers may be implemented in the form of a worker node, a segment node, or a back-end process.

[0063] In one embodiment, step S310 may be performed by a device that performs a query plan generation function. For example, the device may traverse the generated query execution plan tree to verify type information of each operation node and identify operation nodes classified as hash join operations. If multiple hash join operation nodes exist, the device may assign unique identification information to each hash join operation node and configure mapping information for each hash join operation node to be associated with a filter identifier to be generated in a subsequent step. The device may determine which table, which partition, or which subplan the left input and right input of the hash join operation node correspond to, respectively, by referring to meta-information.

[0064] In one embodiment, step S320 may be performed by a device that performs a query plan generation function for the hash join operation node identified in step S310. For example, the device may recursively search for child nodes with the right subtree of the hash join operation node as the root, and during the search process, determine whether a scan operation node, a join operation node, an aggregation operation node, or a communication operation node exists. The device may select a location where the effect of applying a dynamic filter is expected as a filter application target node, for example, immediately before the scan operation or the scan operation itself as a filter application target node. If multiple scan paths exist within the right subtree, the device may select a separate filter application target node for each path, and if a communication operation is included within the right subtree, it may determine where to apply the filter, either before or after the communication operation.

[0065] In one embodiment, step S330 may be performed by a device that performs a query plan generation function based on the result of step S320. For example, the device may generate a filter identifier corresponding to a target node for filter application, and the filter identifier may include a hash join operation node identifier, a target node identifier for filter application, and a filter application path identifier. The filter identifier may include a setting value indicating the type of dynamic filter, whether merging is required, version control parameters, or whether synchronous transmission is required. The filter identifier may include an intent identifier or a path identifier to distinguish an aggregation path when an aggregation operation exists within the right subtree, and accordingly, an invalidation condition for the aggregation path may be optionally applied in a subsequent step.

[0066] In one embodiment, step S340 may be performed by a device that performs a query plan generation function or by a server that performs a hash join operation. For example, the device may record the generated filter identifier in a form associated with the left input table of the hash join operation node, and the recording may be performed by storing it in the attribute value of the execution plan node corresponding to the left input table, the meta-information of the left input table scan operation, or a runtime state object for the left input. By recording, the server that performs hash table construction may trigger the creation and propagation of dynamic filters by referencing the filter identifier during the process of processing the left input table. If multiple filter identifiers exist, the recording may be stored in the form of a list or a map, and may be queried separately by hash join operation node.

[0067] In one embodiment, step S350 may be performed by a device that performs a query plan generation function. For example, the device may transmit an execution plan tree with filter identifiers inserted to a plurality of servers, and may divide and distribute subplans of the execution plan according to the data partition, node role, or parallelism handled by each server. The device may distribute the execution plan so that the identifiers of the execution plan nodes, filter identifiers, and mapping information regarding the nodes to which the filter is applied are consistently maintained for each server. The device may establish a communication channel during the execution plan distribution process and may also transmit channel information necessary for dynamic filter transmission between servers.

[0068] In one embodiment, step S360 may be performed by a plurality of servers that actually perform the hash join operation. For example, each server may read records from the left input table under its charge to construct a hash table, and during the process of constructing the hash table, input the join key of the left input table records into a hash function to calculate a hash value. Each server may create or update a dynamic filter based on the calculated hash value or join key, and the dynamic filter may be managed to correspond to the filter application target node specified by the filter identifier. Each server may update the dynamic filter on a record-by-record basis, update it in a batch manner whenever a certain number of records are processed for performance, and adjust the update policy considering the size or collision rate of the dynamic filter.

[0069] In one embodiment, step S370 may be performed by a server that performs a hash join operation and a server that executes a right subtree. For example, a server processing a left input may transmit a dynamic filter toward the right subtree along a propagation path specified by a filter identifier at the time when the dynamic filter is created or updated. A server executing the right subtree may transmit the received dynamic filter to an intermediate operation node or a filter application target node of the right subtree, and may evaluate the dynamic filter at the filter application target node to select records to be transmitted to a subsequent scan or subsequent operation. Downward propagation may be performed stepwise according to the structure of the right subtree, and if multiple paths exist within the right subtree, the same dynamic filter or dynamic filters branched per path may be propagated for each path.

[0070] In one embodiment, step S380 may be performed by a plurality of servers and a device that performs a global filter merging function when the filter application target node includes communication operations. For example, each server may transmit a dynamic filter it has generated to another server or merging node, and the receiving side may merge the dynamic filters received from the plurality of servers to form a global filter. The global filter may be distributed to the plurality of servers again after merging, and each server may perform consistent filtering on the same or similar scan paths of the right subtree using the global filter. Each server may manage version information of the global filter to control the update order and prevent outdated global filters from being applied.

[0071] In one embodiment, step S390 may be performed by a server scanning the right input table. For example, prior to performing a physical data scan on the right input table, the scan node may apply a dynamic filter or a global filter to determine whether a record is joinable. The scan node may remove records that do not satisfy the dynamic filter so that they are not passed to the join operation, thereby reducing the number of records flowing into the hash join node. The scan node may evaluate the validity of the filter based on the result of applying the filter, and if the data reduction rate is determined to be below a threshold, it may update a status value to disable the dynamic filter in subsequent join operations.

[0072] In one embodiment, a dynamic filter can be generated based on the hash value of each record of the left input table during the hash table construction step of the hash join operation.

[0073] In one embodiment, the hash table construction step may include a process of sequentially reading records of the left input table to extract a join key and applying a hash function to the join key to calculate a hash value. The dynamic filter may be generated or updated incrementally based on the hash value or the join key itself. For example, filter information corresponding to the join key of each record of the left input table may be reflected in the dynamic filter whenever that record is processed, or the filter may be implemented by updating it in batches whenever a certain number of records are processed.

[0074] In one embodiment, the dynamic filter can be maintained in memory in parallel with the hash table, and can maintain a consistent standard by calculating the hash value of the join key using the same hash function as the hash table construction. The dynamic filter can be created from the initial stage of hash table construction and can be propagated toward the right subtree in a partially formed state even before the hash table is fully constructed. Accordingly, pre-filtering of the right input can be initiated even when only a portion of the left input has been processed.

[0075] In one embodiment, the creation of a dynamic filter may be controlled according to a policy specified by a filter identifier. For example, the filter may be activated when the number of left input records exceeds a certain threshold, and the creation of the filter may be allowed only when the distribution characteristics of the join key satisfy specific conditions. Additionally, the dynamic filter may reflect information about all records of the left input table, or it may be created individually for left inputs divided by partition or server.

[0076] In one embodiment, the dynamic filter can dynamically adjust its size, the number of hash functions, or its internal data structure at the time of creation. For example, if the expected number of records in the left input table increases, the size of the filter's bit array can be expanded, and if there are memory constraints, the filter's representation can be compressed or some information can be maintained in a summarized form. The dynamic filter can be continuously updated based on additional processing of left input records even after creation, and the updated state can be managed along with version information.

[0077] In one embodiment, the dynamic filter may be included in the execution context of a hash join operation and may be managed by the same execution thread or the same server process as the hash table. The dynamic filter may be maintained for a certain period even after the hash table construction is complete and may be reused if a subsequent join operation exists within the same execution plan. Additionally, the dynamic filter may be passed to a global filter merging stage after creation and may be combined with filters created on multiple servers to expand into a more comprehensive filter.

[0078] In one embodiment, the dynamic filter may include at least one of a Bloom filter, a minimum-maximum filter, or a set inclusion condition-based filter. In one embodiment, the dynamic filter may include at least one of a Bloom filter, a minimum-maximum filter, or a set inclusion condition-based filter, and may be selectively applied depending on the characteristics of the hash join operation, data distribution, memory constraints, or accuracy requirements. The type of the dynamic filter may be determined during the query execution plan generation phase and may be adaptively changed during execution.

[0079] In one embodiment, the Bloom filter may be implemented as a bit sequence-based structure capable of probabilistically determining the probability of a join key's existence using multiple hash functions. The join key included in each record of the left input table may be input into one or more hash functions to set multiple bit positions, and the set bit information may be reflected in the Bloom filter. When scanning records in the right input table, records with no possibility of joining can be quickly excluded by inputting the join key of the corresponding record into the same hash function to verify the corresponding bit positions. The Bloom filter can compress and represent a large amount of key information while maintaining memory usage below a certain level, and can adjust the number of hash functions and the size of the bit array to have an acceptable false positive probability.

[0080] In one embodiment, the minimum-maximum filter can generate range-based conditions by calculating the minimum and maximum values ​​of the join key included in the left input table. For example, if the join key of the left input table exists within a specific range, right input records having values ​​outside said range may be excluded from the join target. The minimum-maximum filter can be effectively applied when data is sorted or has distinct range characteristics, and can reduce computational costs since determination can be made solely through comparison operations. The minimum-maximum filter may be expressed as a single range or divided into multiple sub-ranges.

[0081] In one embodiment, a set inclusion condition-based filter can determine whether the join key of a right input record is included in the set by storing the join key value of the left input table in a set form. The set inclusion condition-based filter can be implemented as a hash table structure, a tree-based structure, or a compressed set representation structure. Since the set inclusion condition-based filter can determine accurate inclusion, false positives may not occur, and it can be applied efficiently when the number of keys in the left input is relatively small.

[0082] In one embodiment, the dynamic filter may use only one filter type or may use a combination of two or more filter types. For example, a minimum-maximum filter may be applied first to remove out-of-range data, and then a Bloom filter may be applied secondarily to perform fine filtering. Alternatively, most unnecessary records may be removed using a Bloom filter, and then accuracy may be supplemented through a set inclusion condition-based filter.

[0083] In one embodiment, the selection of the type of dynamic filter may be determined based on the number of records in the left input table, the distribution characteristics of the join key, redundancy, available memory, or the expected false positive rate. For example, if the number of records in the left input table is very large, a Bloom filter may be applied first, and if the number of records is relatively small, a set inclusion condition-based filter may be applied. Additionally, if the join key has a continuous range, a minimum-maximum value filter may be applied first.

[0084] In one embodiment, the dynamic filter may be incrementally created or updated during the hash table construction phase of the hash join operation and may be propagated to the right subtree at regular intervals. The dynamic filter may be resizable during execution and may reduce or reconstruct some filter information if memory usage exceeds a threshold. The dynamic filter may be designed as a mergeable structure considering merging between servers, and its internal representation may be configured to facilitate logical or set operations during merging.

[0085] In one embodiment, the dynamic filter may include version information or timestamp information and can maintain consistency based on the latest version when merged from multiple servers. The dynamic filter may include effectiveness evaluation information based on the application results and can change or disable the filter type based on the data reduction rate or false positive rate.

[0086] In one embodiment, the step of configuring a global filter (S380) may include receiving filters generated at each server and merging them by logical operation.

[0087] In one embodiment, the step of configuring a global filter (S380) may include a merging procedure for integrating dynamic filters generated or updated on a plurality of servers into a single global filter. The global filter may be configured in situations where it is difficult to secure a sufficient filtering effect using only filters partially generated per server, such as when a communication node is included in the right subtree or when right input data is distributed and scanned among servers. The step of configuring a global filter (S380) may be performed according to a merging policy specified by a filter identifier, a list of servers to be merged, or a time of merging.

[0088] In one embodiment, each server may create or update a dynamic filter based on the left input table records it has processed, and then serialize the said dynamic filter into a transmittable form. Each server may reduce the transmission size by compressing the internal representation of the dynamic filter (e.g., bit string, range value, set representation), and may transmit only incremental update information to reduce transmission overhead. Each server may add version information, creation time information, target path information, or validity period information to the dynamic filter so that the receiving side can determine the merging order and application order.

[0089] In one embodiment, the merging of global filters may be performed directly by each server to each other, or a separate merging node may perform the merging. For example, the merging node may correspond to the global filter merging module of FIG. 1, and may receive filters from multiple servers, merge them, and then distribute the merged results back to each server. Additionally, merging may be performed in a circular or hierarchical structure between servers without a merging node, and, for example, a specific server may be selected as a merge leader to collect filters from other servers and then broadcast the merged results.

[0090] In one embodiment, the merging of global filters may apply different logical operation rules depending on the type of dynamic filter. For example, if the dynamic filter is implemented based on bit sequences, such as a Bloom filter, merging may be achieved by performing a bitwise OR operation on bit sequences generated by multiple servers. In this case, bits reflecting the join key observed from the left input of each server may be integrated into the global filter, and the global filter may operate in a form representing the union of each server filter. Additionally, post-processing such as resizing the bit sequence or adjusting the number of hash functions may be performed after merging to control the increase in false positive rates.

[0091] In one embodiment, when the dynamic filter is a minimum-maximum filter, the merging of the global filter can be performed by selecting the minimum value among the server-specific minimum values ​​as the global minimum and selecting the maximum value among the server-specific maximum values ​​as the global maximum. The global minimum and global maximum values ​​merged in this way can be used as range-based filtering conditions in the right-hand input scan step. Additionally, when the minimum-maximum filter is implemented by maintaining multiple partial ranges per server, the merging can be performed according to a consolidation rule between sets of partial ranges, and overlapping ranges can be merged or adjacent ranges can be combined to reduce the representation size.

[0092] In one embodiment, if the dynamic filter is a set inclusion condition-based filter, the merging of the global filter may be performed by forming a union of server-specific sets. For example, if a key set in the form of a hash set is maintained for each server, the receiving side may generate a global key set by merging the key sets transmitted from each server. In this case, to reduce the amount of transmission, the server may transmit only newly added keys in an incremental form rather than transmitting the entire key set, and the receiving side may reflect only the incremental keys in the global key set. Additionally, if the size of the set becomes very large, the global filter may be converted from a set inclusion condition-based filter to a probability-based filter, or reduced to maintain only a representative set of a certain size or smaller.

[0093] In one embodiment, the global filter merging process may include synchronization control to prevent merge conflicts. For example, a merge node or merge leader may assign a merge round identifier to ensure that only filters of the same round are merged, thereby preventing filters of different rounds from being mixed. Additionally, when distributing the merge results, version information of the global filters may be transmitted along with the results to control each server to apply only the latest global filters. When each server detects that the version of a global filter has been updated, it may replace the existing global filter or apply it by reflecting only the updated parts.

[0094] In one embodiment, the global filter can be forwarded to the filter target node of the right subtree after merging and can be applied prior to the physical data scan of the right input table. Since the global filter can include more left input observation information than the server-specific partial filter, it can remove non-join records on the right input side at a higher rate. The global filter can be further updated even after the merging point and can be continuously kept up to date by re-merging at regular intervals.

[0095] In one embodiment, the distributed database dynamic filtering method (300) can configure a global filter by pre-establishing a communication channel with a server receiving the filter and transmitting the filter synchronously through the communication channel.

[0096] In one embodiment, filter transmission for configuring a global filter may include a procedure for pre-configuring a communication channel between servers, and consistency at the time of merging can be ensured by transmitting the filter synchronously through the communication channel. The pre-configuration of the communication channel may be performed before or together with the distribution of the query execution plan tree, and may be performed according to the communication configuration information included in the filter identifier. For example, a device performing a query plan generation function may determine whether a communication node is included in the right subtree, and if a communication node is included, it may generate information regarding the list of servers to participate in filter transmission, the transmission direction, and the merging method, and transmit it to each server.

[0097] In one embodiment, the communication channel may be configured as a direct communication channel between servers or as a relay channel passing through a merge node. A direct communication channel between servers may be configured by each server sharing the address, port, session identifier, or authentication information of another server in advance, and a channel passing through a merge node may be configured by each server registering the endpoint of the merge node in advance. The communication channel may be configured in a connection-oriented manner to ensure transmission reliability, or in a connectionless manner to minimize latency. Depending on the implementation environment, the communication channel may include a message queue, a remote call interface, streaming transmission, or inter-process communication.

[0098] In one embodiment, the preconfiguration of the communication channel may define synchronization conditions for performing a “synchronous method” of filter transmission. Synchronous transmission may be implemented, for example, by controlling transmission to begin after all participating servers have reached a filter transmission readiness state in each merge round. When each server processes a certain amount of left-hand input records during the hash table construction phase and detects that a dynamic filter has been formed above a certain level, it may transmit a transmission readiness signal through the communication channel. After receiving transmission readiness signals from multiple servers, the merge node or merge leader may broadcast a transmission start signal to control filters at the same time to be transmitted.

[0099] In one embodiment, synchronous transmission can control the filters to be merged to have the same version or the same round identifier. For example, each server may assign a round identifier, version number, or timestamp to dynamic filters, and the filters transmitted through the communication channel may include the identifier. The receiving side may select only filters having the same round identifier as merge targets and prevent filters from different rounds from being mixed and merged. If a filter from a specific server is delayed and in a non-received state, the receiving side may decide, according to a timeout policy, to exclude that server from the merge round or to use a filter from the previous round as a replacement.

[0100] In one embodiment, synchronous transmission can be used to match the transmission order and the application order. For example, since a global filter can be applied prior to the scan of the right input table, the server performing the scan may delay the initiation of the scan until the global filter is received and becomes available for application. Alternatively, the scan may be initiated immediately, but the application of the global filter may be switched to begin with the scan batch after the global filter has been received. In this case, by transmitting a control message indicating the start time of global filter application through the communication channel, the timing of global filter application can be consistently matched at each server.

[0101] In one embodiment, the communication channel may support differential transmission or incremental transmission to increase the efficiency of filter transmission. For example, if the dynamic filter is in the form of a Bloom filter, the server may transmit only the changed bit segment or the changed bit index instead of repeatedly transmitting the entire bit sequence. If the dynamic filter is in the form of a minimum-maximum filter, the server may transmit only the updated minimum or maximum value, and if the dynamic filter is in the form of a set inclusion condition-based filter, the server may transmit only the newly added key. The receiving side can reduce merging costs and network usage by reflecting the incremental information in the global filter.

[0102] In one embodiment, the communication channel may include retransmission, acknowledgment, or error detection functions to handle transmission errors. For example, if the server does not receive an acknowledgment after transmitting a filter, it may retransmit the filter of the same round, and the receiving side may detect corruption during transmission using the filter's checksum or hash verification value. The communication channel may include encryption or authentication procedures according to security requirements, and control may be exercised to allow filter exchange only when authorization between servers has been verified.

[0103] In one embodiment, synchronous transmission can improve the consistency of global filter merging and mitigate excessive false positives or filter omissions that may occur when filters generated at different times on each server are mixed. Synchronous transmission allows global filters to represent left-hand input observation information at a specific time and maintains the consistency of filters applied prior to right-hand input scans. Accordingly, the effectiveness of applying global filters can be reliably secured in a distributed environment.

[0104] FIG. 4 is a diagram illustrating a specific operation for generating a filter identifier according to an embodiment of the present invention. In one embodiment, the step of generating a filter identifier (S330) may include the step of generating a filter identifier (S401) when the ratio of the number of output rows of the right subtree to the number of expected join result rows exceeds a preset first threshold.

[0105] Referring to FIG. 4, in one embodiment, step S401 can control the generation of filter identifiers selectively when a filtering effect is expected, while preventing the unnecessary over-generation of filter identifiers. The number of output rows of the right subtree may represent the number of records predicted to flow into the right input of the hash join operation node, and the number of expected join result rows may represent the number of result records predicted to be generated after applying the join condition between the left input and the right input. Step S401 can indirectly evaluate the potential amount of records that can be removed from the right input side using the ratio of the two values, and accordingly determine whether the value of applying a dynamic filter is sufficient.

[0106] In one embodiment, the number of output rows in the right subtree can be estimated based on statistical information collected at the time of query execution plan generation. For example, the expected number of output rows after a scan can be calculated based on table cardinality, the number of records per partition, index selectivity, condition selectivity, or sampling results. If filtering conditions, sorting, constraints, or intermediate aggregation are included within the right subtree, the number of output rows can be adjusted by reflecting the selectivity and reduction rate of the operations. Additionally, if the right subtree includes multiple paths, the number of output rows per path can be summed, or the number of output rows per server can be calculated individually by considering parallelism and data distribution.

[0107] In one embodiment, the expected number of join result rows can be estimated based on the selectivity of the join condition. For example, the join selectivity can be calculated based on the number of different values ​​of the join key, the degree of bias of the key distribution, histograms, minimum and maximum values, or the ratio of null values ​​of the join key. If there are multiple join conditions, the number of result rows can be predicted by combining the selectivity of each condition, and the number of result rows can be adjusted by reflecting the prior application of filter conditions for the left and right inputs. Additionally, if the statistical information is not up to date, the prediction of the number of result rows can be dynamically corrected using the number of rows observed during execution.

[0108] In one embodiment, step S401 can calculate a ratio by dividing the number of output rows of the right subtree by the number of expected join result rows, and determine whether the ratio exceeds a first threshold. If the ratio exceeds the first threshold, it can be determined that there is a high probability that the right input side contains a relatively large number of records that do not contribute to the join, and it can be evaluated that there are sufficient records that can be removed when a dynamic filter is applied. Accordingly, step S401 can enable the creation and propagation of a dynamic filter by generating a filter identifier and inserting it into the execution plan. Conversely, if the ratio is below the first threshold, it can be determined that the overhead associated with applying the filter may outweigh the performance improvement effect, and the generation of the filter identifier can be omitted to simplify subsequent operations.

[0109] In one embodiment, the first threshold value may be pre-set as a system setting value or a policy value based on the query type. The first threshold value may be adjusted according to the server's network bandwidth, the storage format of the right input table, the filter evaluation cost, or the parallelism of the hash join operation. For example, in an environment with high network transmission costs, a relatively low first threshold value may be applied to actively activate the filter, and in an environment with high filter evaluation costs, a relatively high first threshold value may be applied to conservatively limit filter activation. Additionally, the first threshold value may be applied differentially depending on the data type or domain size of the join key, and the threshold value may be set differently in cases with a wide distribution, such as string keys.

[0110] In one embodiment, the filter identifier may include a hash join operation node identifier, a filter application target node identifier, and filter application path information. The filter identifier may additionally include parameters indicating the type of dynamic filter, whether merging is required, whether synchronous transmission is required, or a version control method. If multiple scan nodes exist within the right subtree, the filter identifier may be generated separately for each scan node, and if communication nodes exist within the right subtree, it may include information specifying the application location before and after the communication node. The filter identifier may be recorded in the left input table and referenced during the hash table construction step, thereby linking the creation of the dynamic filter with its application to the right subtree.

[0111] In one embodiment, step S401 may determine whether the first threshold is exceeded on a server-by-server basis or on a single criterion for the entire query execution plan tree. For example, if data is distributed in a biased manner by server, the ratio of the number of output rows per server and the number of expected join result rows per server may be calculated to generate a filter identifier only on a specific server. Additionally, if the number of output rows in the right subtree is extremely small, filter generation may be unnecessary even if the ratio is large; therefore, an auxiliary threshold for the absolute number of rows may be applied together to restrict the generation of filter identifiers.

[0112] FIG. 5 is a diagram illustrating an operation to invalidate a dynamic filter applied to an aggregation node when the number of accumulated rows exceeds a threshold according to an embodiment of the present invention. In one embodiment, the distributed database dynamic filtering method (300) may further include the step of adding an intent identifier to a filter identifier (S501) when the filter application target node includes an aggregation node, and the step of invalidating a dynamic filter applied to an aggregation node when the number of accumulated rows processed at the aggregation node exceeds a second threshold (S502).

[0113] Referring to FIG. 5, in one embodiment, steps S501 and S502 may be performed as control actions to mitigate inefficiencies that may arise from the application of a dynamic filter when an aggregation operation is included within the right subtree. An aggregation node may be defined as an operation node in an execution plan that performs an aggregation operation by combining a plurality of input rows into one or more groups, and may perform, for example, a grouping operation, a summing operation, a counting operation, an average operation, or a combination thereof. An aggregation node may be located in the middle or lower part of the right subtree, and a communication node or a scan node may be placed before or after the aggregation node. When an aggregation node is included, the number of output rows and selectivity estimated in the execution plan stage may vary significantly during the actual execution process; therefore, it may be necessary to separately control whether to apply a dynamic filter based on the characteristics of the aggregation path.

[0114] In one embodiment, step S501 determines whether the filter-applied target node includes an aggregation node, and then adds an intent identifier to the filter identifier to distinguish the processing of the aggregation path from other paths. The intent identifier may be included in the form of a flag, a path tag, or a code value indicating the type of aggregation operation within the filter identifier, thereby allowing subsequent invalidation conditions to be applied only to paths that include the aggregation node. The intent identifier may be set to different values ​​depending on the location of the aggregation node, the number of aggregation keys, or the type of aggregation operation, and if multiple aggregation nodes exist, different intent identifiers may be assigned to each aggregation node. The intent identifier may be transmitted together when the execution plan is distributed to multiple servers, and each server may separately manage the dynamic filter of the aggregation path by referencing the intent identifier.

[0115] In one embodiment, step S502 can monitor the number of accumulated rows processed by the aggregation node and can invalidate the dynamic filter applied to the aggregation node if the number of accumulated rows exceeds a second threshold. The number of accumulated rows may be defined as at least one of the number of rows input to the aggregation node, the number of updates to the intermediate state maintained for grouping processing in the aggregation node, or the number of result rows output by the aggregation node. The number of accumulated rows may be incremented by a runtime counter while the aggregation node is running, and each server may locally aggregate the number of accumulated rows within the aggregation processing scope it is responsible for. If the aggregation nodes are running in parallel, the number of accumulated rows may be converted into a global number of accumulated rows by summing the server-specific accumulated values, or it may be determined by whether the number of accumulated rows per server exceeds the second threshold.

[0116] In one embodiment, the second threshold may be pre-set as a reference value to control overhead that may occur due to the application of dynamic filters when an aggregation node is included. The second threshold may be determined based on the type of aggregation operation, the number of groups, expected memory usage, or the processing latency of the aggregation node, and may be a policy value set by a system administrator or a value based on the results of automatic tuning. The second threshold may be dynamically calculated by reflecting the expected cardinality and number of groups of the aggregation node at the time of query execution plan generation, and may be updated based on the throughput observed during execution. For example, if the number of groups of the aggregation node increases rapidly, the second threshold may be lowered to induce filter invalidation early, and if the processing burden of the aggregation node is low, the second threshold may be raised to increase the filter retention time.

[0117] In one embodiment, invalidating a dynamic filter may include an action of transitioning the state to stop the filter evaluation applied to the aggregation node. For example, the aggregation node or a filter target node above the aggregation node may mark the corresponding filter as inactive by referencing a filter identifier containing an intent identifier, and process it so that filter condition evaluation is not performed on subsequent input records. The invalidation of a dynamic filter may be limited to the aggregation path, and dynamic filters in other paths that do not include the aggregation node for the same hash join operation node may continue to be maintained. Additionally, filter creation itself may continue even after the invalidation of the dynamic filter, but only the transmission path or application location may be blocked so that it is not applied to the aggregation node.

[0118] In one embodiment, step S502 may determine the invalidation point by a single threshold comparison, or apply a hysteresis condition to prevent frequent switching. For example, invalidation may occur immediately when the accumulated number of rows exceeds a second threshold, or invalidation may occur only when the state of exceeding the second threshold persists for a certain period of time or longer. Additionally, even if the accumulated number of rows exceeds the second threshold, invalidation may be delayed or exceptionally maintained if the data reduction rate resulting from the application of a dynamic filter is sufficiently large. Conversely, even if the accumulated number of rows is below the second threshold, invalidation may be performed first if the processing delay of the aggregation node increases rapidly.

[0119] In one embodiment, by invalidating the dynamic filter applied to the aggregation node, the computational cost and state management overhead required for evaluating the dynamic filter in the aggregation path can be reduced. Accordingly, performance fluctuations caused by the dynamic filter in a query execution path that includes aggregation operations can be mitigated, and queries can be processed stably even for execution plans with large prediction errors in a distributed environment.

[0120] FIG. 6 is a diagram illustrating an operation to disable a dynamic filter for a subsequent join operation when the data reduction rate after applying a dynamic filter according to an embodiment of the present invention is less than a preset threshold. In one embodiment, the distributed database dynamic filtering method (300) may further include a step (S601) of disabling a dynamic filter for a subsequent join operation when the data reduction rate after applying a dynamic filter is less than a preset threshold.

[0121] Referring to FIG. 6, in one embodiment, step S601 may be performed as an adaptive control operation to reduce unnecessary filter processing overhead when the effect of applying a dynamic filter is insufficient. The dynamic filter may be applied before performing a physical data scan of the right input table or during an intermediate scan batch, and step S601 may quantify the result of applying the dynamic filter to determine whether to maintain the active state of the dynamic filter in a subsequent join operation. If the data reduction rate resulting from the filter application is below a threshold, step S601 may reduce the waste of overall system resources by stopping at least some of the dynamic filter creation, propagation, or evaluation operations in a subsequent join operation.

[0122] In one embodiment, the data reduction rate can be calculated by comparing the number of records before and after the application of a dynamic filter. For example, the data reduction rate can be calculated as (number of records to be scanned before filter application - number of records that passed after filter application) / number of records to be scanned before filter application. The number of records to be scanned before filter application can be defined as the number of records scheduled for or already scanned in the right input table, and the number of records that passed after filter application can be defined as the number of records that satisfied the dynamic filter and were passed to the hash join node. The data reduction rate can be calculated at the scan node level, and if multiple scan nodes exist, the reduction rate at the right subtree level can be calculated by weighting the reduction rates per scan node.

[0123] In one embodiment, the evaluation of the data reduction rate may be performed in intervals or batch units rather than at a single point in time. For example, the scan node may set a “verification interval” whenever a certain amount of records is processed and aggregate the number of filter-blocked records and the number of records that pass through for each interval. Since the hash table of the left input table may not be fully constructed in the initial interval and the filter may not be sufficiently formed, step S601 may evaluate the reduction rate in an interval after a certain amount of time or a certain number of accumulated records has elapsed. Step S601 may stabilize the evaluation value by applying a moving average, exponential smoothing, or hysteresis conditions to mitigate abrupt fluctuations in the reduction rate.

[0124] In one embodiment, a “pre-set criterion” may be set as a threshold value to determine whether the expected gain from maintaining a dynamic filter outweighs the filter processing cost. The criterion may be a policy value set by a system administrator, or it may be calculated by reflecting weights for network costs, disk read costs, filter evaluation costs, or join processing costs. For example, in an environment where network transmission is a bottleneck, a low criterion may be applied to maintain the filter even if the reduction rate is small, while in an environment where the CPU is a bottleneck, a high criterion may be applied to maintain the filter only when the reduction rate is sufficiently large. Additionally, the criterion may vary depending on the storage format or scan method of the right-hand input table, and the criterion may be set differently for column-oriented storage formats.

[0125] In one embodiment, the deactivation of the dynamic filter according to step S601 may be implemented in a form that partially or completely stops the filter function for subsequent join operations. For example, the deactivation may take the form of stopping filter evaluation at the filter application target node to allow all records to pass through. Additionally, the deactivation may take the form of stopping the creation or updating of the dynamic filter in the left input table, or it may take the form of creating the dynamic filter but stopping its propagation to the right subtree. Step (S601) may define the scope of the “subsequent join operations” within the execution plan at the level of hash join operation nodes, and may apply the deactivation state to subsequent iterative executions on the same hash join operation node, subsequent batch processing within the same query, or similar queries that are repeatedly executed within the same session.

[0126] In one embodiment, step S601 may record the disabled state in a runtime state object of the execution plan, and said state object may be managed in association with a hash join operation node identifier, a filter identifier, or a right subtree path identifier. Accordingly, when the same hash join operation is performed next, the system may look up the disabled state and skip processing related to dynamic filters. Additionally, the disabled state may be released under certain conditions, and may be reactivated, for example, when it is determined that the reduction rate will improve due to a change in data distribution or a change in join conditions. Reactivation may be performed by re-evaluating the reduction rate by applying the filter on a sampling basis at regular intervals.

[0127] In one embodiment, since the data reduction rate in a distributed environment may vary by server, step S601 may evaluate the reduction rate for each server individually and disable the dynamic filter for each server. Alternatively, the reduction rates for each server may be collected to calculate a global reduction rate, and then the decision to disable the filter may be made based on a global standard. If a server-specific standard is applied, the filter may be disabled on a specific server because the filter effect is low, while it may remain active on another server because the filter effect is sufficient. If a global standard is applied, a consistent decision may be made by considering the integrated reduction rate calculated in the global filter merging step or the false positive rate of the merged global filter.

[0128] In one embodiment, when the dynamic filter is disabled by step S601, CPU resources, memory resources, and communication resources required for filter creation / propagation / merging / evaluation can be reduced. Accordingly, performance degradation that may occur when the dynamic filter does not contribute to substantial data reduction can be mitigated, and the stability of the entire distributed query processing can be improved.

[0129] FIG. 7 is a block diagram illustrating the configuration of a device according to embodiments of the present invention.

[0130] In one embodiment, the device (2000) may include an input / output interface (2010), a memory (2020), a processor (2030), and a communication interface (2040). However, the present invention is not limited thereto. The device (2000) may be configured with some of the components shown in FIG. 7 omitted, or may be configured to include other components in addition to the components shown in FIG. 7.

[0131] In one embodiment, the input / output interface (2010), memory (2020), processor (2030), and communication interface (2040) may each be physically / electrically connected to each other.

[0132] In one embodiment, the device (2000) may be connected to various types of external devices through an input / output interface (2010). In one embodiment, the input / output interface (2010) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (Input / Output) port, or a video I / O (Input / Output) port. In one embodiment, the input / output interface (2010) may include a USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), or DVI (Digital Visual Interface), etc.

[0133] In one embodiment, the memory (2020) may store data used in the device (2000). In one embodiment, the memory (2020) may store instructions, programs, or modules for the operation of the processor (2030).

[0134] In one embodiment, the memory (2020) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD type (Solid State Disk type), an SSD type (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, and an optical disk.

[0135] In one embodiment, the processor (2030) may include a general-purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), DSP (Digital Signal Processor), or a neural network processing processor such as an NPU (Neural Processing Unit). In one embodiment, the processor (2030) may be divided by one or more processors to perform operations. In one embodiment, the processor (2030) may control the operation of the device (2000). The processor (2030) may control the operation of the device (2000) according to instructions stored in memory (2020).

[0136] In one embodiment, the processor (2030) may be implemented as a memory that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the device (2000) of the present invention, and as a processor that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0137] In addition, the processor may control one or a combination of the components described above in order to implement various embodiments according to the present invention through the device (2000).

[0138] In one embodiment, the communication interface (2040) may include one or more components that enable communication between the device (2000) and an external server or external electronic device. In one embodiment, the communication interface (2040) may include at least one of a wired communication module or a wireless communication module.

[0139] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0140] The wireless communication module may include a wireless communication module that supports a wireless communication method including at least one of WiBro (Wireless broadband), GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, Bluetooth™, or Wi-Fi (Wireless-Fidelity).

[0141] In one embodiment, the processor (2030) can obtain information from an external electronic device or server or provide information through a communication interface (2040).

[0142] Using the embodiments of the present invention described above, those skilled in the art will be able to easily make various changes and modifications within the scope of the essential characteristics of the present invention. The content of each claim of the patent claims may be combined with other claims that are not related by reference within the scope of what can be understood from this specification.

[0143] The effects according to the technical concept of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description in the specification.

[0144] The technical concept of the present invention, as described so far with reference to FIGS. 1 to 7, can be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). The computer program recorded on the computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed on the other computing device, thereby being used on the other computing device.

[0145] Although it has been described above that all components constituting the embodiments of the present invention are combined as one or operate in combination, the technical concept of the present invention is not necessarily limited to such embodiments. That is, within the scope of the objectives of the present invention, all components may be selectively combined in one or more ways to operate.

[0146] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0147] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concept defined by the present invention. Explanation of the symbols

[0148] 2000: Device 2010: Input / Output Interface 2020: Memory 2030: Processor 2040: Communication Interface

Claims

Claim 1 A distributed database dynamic filtering method for performing a hash join operation on data distributed and stored on multiple servers, wherein the distributed database dynamic filtering method is performed by at least one processor, and the at least one processor performs the steps of: generating a query execution plan tree of the distributed database and then traversing the query execution plan tree to identify a hash join operation node; the at least one processor performs the step of recursively searching the right subtree of the hash join operation node to identify a node to which a filter is applied; the at least one processor performs the step of generating a filter identifier corresponding to the node to which a filter is applied; the at least one processor performs the step of recording the generated filter identifier in the left input table of the hash join operation node; the at least one processor performs the step of distributing the execution plan tree with the filter identifier inserted to the multiple servers; the at least one processor performs the step of generating or updating a hash value-based dynamic filter based on a record of the left input table in the hash table construction step of the hash join operation; and the at least one processor performs the step of progressively propagating the generated dynamic filter downward along the right subtree of the hash join operation. A distributed database dynamic filtering method comprising: a step of applying a filter to a target node; a step in which, if the target node for the filter includes a communication node that performs inter-server communication, the at least one processor constructs a global filter by mutually transmitting and merging filters generated at each server; and a step in which the at least one processor applies the dynamic filter to pre-remove join target records prior to performing a physical data scan on the right input table of the hash join operation node. Claim 2 A distributed database dynamic filtering method according to claim 1, wherein the step of generating the filter identifier includes the step of generating the filter identifier when the ratio of the number of output rows of the right subtree to the number of expected join result rows exceeds a preset first threshold. Claim 3 A distributed database dynamic filtering method according to claim 1, further comprising: a step of adding an intent identifier to the filter identifier when the filter application target node includes an aggregation node; and a step of invalidating the dynamic filter applied to the aggregation node when the cumulative number of rows processed in the aggregation node exceeds a second threshold. Claim 4 A distributed database dynamic filtering method according to claim 1, wherein the dynamic filter is generated based on the hash value of each record of the left input table during the hash table construction step of the hash join operation. Claim 5 A distributed database dynamic filtering method according to claim 1, wherein the dynamic filter comprises at least one of a Bloom filter, a minimum-maximum filter, or a set inclusion condition-based filter. Claim 6 A distributed database dynamic filtering method according to claim 1, wherein the step of configuring the global filter includes the step of receiving filters generated at each server and merging them by logical operation. Claim 7 A distributed database dynamic filtering method according to claim 1, wherein, in order to configure the global filter, a server generating the filter pre-establishes a communication channel with a server receiving the filter and transmits the filter synchronously through the communication channel. Claim 8 A distributed database dynamic filtering method according to claim 1, further comprising the step of disabling the dynamic filter for subsequent join operations when the data reduction rate after applying the dynamic filter is less than a preset standard. Claim 9 A distributed database dynamic filtering device that performs a hash join operation on data distributed and stored on multiple servers, comprising: a memory for storing one or more instructions; A distributed database dynamic filtering device comprising at least one processor, wherein the at least one processor executes one or more instructions, generates a query execution plan tree of the distributed database, traverses the execution plan tree to identify a hash join operation node, recursively searches the right subtree of the hash join operation node to identify a filter application target node, generates a filter identifier corresponding to the filter application target node, records the generated filter identifier in the left input table of the hash join operation node, distributes the execution plan tree with the inserted filter identifier to the plurality of servers, generates or updates a hash value-based dynamic filter based on the records of the left input table in the hash table construction step of the hash join operation, applies the generated dynamic filter to the filter application target node by stepwise downward propagation along the right subtree of the hash join operation, and if the filter application target node includes a communication node that performs inter-server communication, constructs a global filter by mutually transmitting and merging the filters generated at each server, and prior to performing a physical data scan on the right input table of the hash join operation node, the dynamic A distributed database dynamic filtering device that pre-removes join target records by applying filters. Claim 10 A distributed database dynamic filtering device according to claim 9, wherein the distributed database dynamic filtering device generates a filter identifier when the ratio of the number of output rows of the right subtree to the number of expected join result rows exceeds a preset first threshold by the at least one processor executing the one or more instructions.

Images