Data association query method and related device

Abstract

Provided are a data association query method and a device. The method comprises: in response to a data association query for a first data table and a second data table, determining an association key related to the data association query, the volume of data comprised in the first data table being less than that in the second data table (S201); determining the number of data rows comprised in the first data table, and creating a filter the size of which matches the number of the data rows (S202); determining in the first data table a target data column corresponding to the association key, and adding into the filter at least one association key value comprised in the target data column that corresponds to the association key (S203); and, on the basis of the filter, filtering data rows comprised in the second data table which correspond to the at least one association key value, and performing data association query on the basis of the first data table and the filtered second data table (S204).

Description

A data association query method and related equipment Technical Field

[0001] This disclosure relates to the field of data query technology, and more specifically, to a data association query method and related equipment. Background Technology

[0002] When performing data join queries across multiple tables, filters (such as Bloom filters) can often be used to optimize the query. For example, you can first add several join keys from the smaller table that are relevant to the current join query to a filter. Then, use this filter to pre-filter data from the larger table that does not match the join keys. Finally, you can perform a small-scale join query based on the smaller table and the filtered larger table, avoiding unnecessary join operations and thus improving query efficiency.

[0003] However, on the one hand, filters usually have a certain false positive rate. If the filter is small and the number of elements actually added to it (such as the related key values ​​mentioned above) is too large, the false positive rate of the filter will be greatly increased, which will make it impossible to accurately filter the data in the large table and seriously affect the query performance. On the other hand, if an excessively large filter is created in order to reduce the false positive rate, it will occupy a lot of memory space, thereby reducing the query performance. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data association query method and related equipment.

[0005] Firstly, this specification provides a data association query method, the method comprising:

[0006] In response to a data association query targeting a first data table and a second data table, a association key related to the data association query is determined; wherein the first data table contains less data than the second data table;

[0007] Determine the number of data rows contained in the first data table, and create a filter whose specifications match the number of data rows;

[0008] Determine the target data column in the first data table that corresponds to the association key, and add at least one association key value in the target data column that corresponds to the association key to the filter;

[0009] Based on the filter, data rows in the second data table that correspond to the at least one associated key value are filtered, and data association queries are performed based on the first data table and the filtered second data table.

[0010] Secondly, this specification provides a data association query device, the device comprising:

[0011] The association key determination unit is used to determine the association key related to the data association query in response to a data association query for a first data table and a second data table; wherein the first data table contains less data than the second data table;

[0012] A filter creation unit is used to determine the number of data rows contained in the first data table and create a filter whose specifications match the number of data rows;

[0013] An adding unit is used to determine the target data column in the first data table that corresponds to the association key, and to add at least one association key value in the target data column that corresponds to the association key to the filter;

[0014] The filtering unit is used to filter data rows in the second data table that correspond to the at least one associated key value based on the filter, and to perform data association queries based on the first data table and the filtered second data table.

[0015] Accordingly, this specification also provides a computing device, including: a memory and a processor; the memory stores a computer program / instructions executable by the processor; when the processor executes the computer program / instructions, it performs the data association query method described in the first aspect above.

[0016] Accordingly, this specification also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, performs the data association query method as described in the first aspect above.

[0017] Accordingly, this specification also provides a computer program product, which includes a computer program / instruction that, when executed by a processor, performs the data association query method as described in the first aspect above.

[0018] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0019] This manual first determines the number of all data rows contained in the first data table (smaller table) participating in the data association query, and then creates a filter whose specifications match the number of data rows. This allows the filter to effectively reduce the false positive rate of the filter when filtering the second data table (larger table) based on the filter, ensuring the filtering effect of the filter on the second data table, and also avoids the filter occupying a large amount of memory space, thereby improving the performance of the data association query. Attached Figure Description

[0020] Figure 1 is a schematic diagram of the architecture of a data association query system provided in an exemplary embodiment.

[0021] Figure 2 is a flowchart illustrating a data association query method provided in an exemplary embodiment.

[0022] Figure 3 is a schematic diagram of the execution flow of a distributed query plan provided in an exemplary embodiment.

[0023] Figure 4 is a schematic diagram illustrating the number of data rows provided in an exemplary embodiment.

[0024] Figure 5 is a schematic diagram of a distributed cluster system architecture provided by an exemplary embodiment.

[0025] Figure 6 is a schematic diagram of the structure of a data association query device provided in an exemplary embodiment.

[0026] Figure 7 is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0029] It should be noted that "multiple" as used in this application refers to two or more.

[0030] Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0031] The following explanations of some of the technical terms used in this specification are provided to facilitate understanding by those skilled in the art.

[0032] (1) Bloom Filter:

[0033] A Bloom filter typically consists of a binary vector (often called a bit array) and several random mapping functions (such as hash functions), and can be used to retrieve whether an element exists in a set.

[0034] Due to the characteristics of Bloom filters, they may produce false positives, meaning they incorrectly report that an element exists in the set when it does not. However, Bloom filters will never produce false negatives; if a Bloom filter detects that an element is not in the set, then that element is indeed not in the set.

[0035] Suppose a Bloom filter contains a binary vector of length m (initially all zeros) and k different hash functions (each function maps an element to a position in a bit array).

[0036] When using a Bloom filter to check if an element exists in a set, the element in the set can be inserted into the Bloom filter first. Then, the element can be mapped to a position in the binary vector using the k hash functions mentioned above, resulting in k positions. Finally, the bit values ​​corresponding to these k positions in the binary vector can be set to 1.

[0037] When querying whether an element exists in the set inserted into the Bloom filter, the k-hash function described above can be used to map the element to a position in the binary vector, resulting in k positions. If the values ​​of the bits corresponding to these k positions are all 1, then the element is considered to exist in the set inserted into the Bloom filter; if the value of any bit among the bits corresponding to these k positions is 0, then the element is considered not to exist in the set inserted into the Bloom filter.

[0038] Therefore, if a Bloom filter is small, that is, if the binary vector it contains is short, then the probability that a small number of bits in the vector will be shared by multiple elements will be higher. This may increase the likelihood of false positives from the Bloom filter, thus increasing the false positive rate of the Bloom filter.

[0039] (2) Hash join:

[0040] Hash join is a join algorithm used in relational databases that can combine multiple relational tables to perform data association queries.

[0041] Hash joins are mainly divided into two phases: the build phase and the probe phase. In the build phase, a smaller table from multiple data tables can be selected as the build table. The related key values ​​contained in the data columns corresponding to the related key (or join key) in the build table are hashed, and the resulting hash values ​​are stored in a hash table.

[0042] During the probing phase, a large table from multiple data tables can be used as the probe table. This large table is scanned, and for each data row scanned, the hash value corresponding to its associated key value is calculated and searched in the hash table. If a hash value with the same value is found, the corresponding data rows are retrieved from the two data tables, and the data join is performed, and the result of the data join is output.

[0043] In practical applications, when using hash join algorithms to perform data association queries on multiple relational data tables, Bloom filters can be used to reduce unnecessary hash join operations.

[0044] Specifically, several related key values ​​from the smaller table that are relevant to the current data join query can be pre-added to the Bloom filter. Subsequently, each row in the larger table can be scanned, and the presence of the related key value in that row can be checked against the Bloom filter. If the related key value is not present in the Bloom filter, that row can be directly filtered out from the larger table. Only the remaining rows that are not filtered out will be used for the complete hash join operation with the smaller table. This narrows the scope of data lookup in the hash table, thereby reducing unnecessary computational overhead.

[0045] However, in practical applications, on the one hand, if the created Bloom filter is small, and the number of elements actually added to the Bloom filter (such as the related key values ​​contained in the small table mentioned above) is too large, it will greatly increase the false positive rate of the Bloom filter, thus failing to accurately filter out the data in the large table and seriously affecting query performance.

[0046] On the other hand, if an excessively large Bloom filter is created in order to reduce the false positive rate, it will consume a lot of memory space, which will also reduce query performance.

[0047] Based on this, this specification provides a technical solution for creating a Bloom filter in real time by statistically analyzing the actual number of data rows contained in a constructed table (smaller table) in an application scenario that uses hash join technology to perform data association queries on multiple relational data tables.

[0048] In implementation, in response to a data join query targeting the first and second data tables, the join key related to that query can be determined first. The first data table can contain less data than the second data table.

[0049] Then, the number of data rows contained in the first data table can be determined, and a filter with specifications matching the number of data rows can be created.

[0050] Furthermore, the target data column corresponding to the association key contained in the first data table can be determined, and at least one association key value corresponding to the association key contained in the target data column can be added to the created filter.

[0051] Finally, based on this filter, data rows in the second data table that correspond to at least one of the aforementioned associated key values ​​can be filtered, and data association queries can be performed based on the first data table and the filtered second data table.

[0052] In the above technical solution, by first determining the number of all data rows contained in the first data table (smaller table) participating in the data association query, and then creating a filter whose specifications match the number of data rows, the false judgment rate of the filter can be effectively reduced when filtering the second data table (larger table) based on the filter, ensuring the filtering effect of the filter on the second data table, and avoiding the filter occupying a large amount of memory space, thereby improving the performance of the data association query.

[0053] Please refer to Figure 1, which is a schematic diagram of the architecture of a data association query system provided in an exemplary embodiment.

[0054] One or more embodiments provided in this specification can be implemented in the system architecture shown in Figure 1 or a similar system architecture. As shown in Figure 1, the data association query system may include computing device 100 and computing device 200, wherein computing device 100 and computing device 200 can establish a communication connection in any possible way, which is not specifically limited in this specification.

[0055] As shown in Figure 1, a client related to data association queries runs on computing device 100, and a server related to data association queries runs on computing device 200. As shown in Figure 1, users can initiate data association queries targeting the first and second data tables in the database through the client running on computing device 100 to the server running on computing device 200.

[0056] In one illustrated embodiment, the database may be a relational database, and correspondingly, the first data table and the second data table may be relational data tables consisting of multiple data columns and multiple data rows (or multiple rows of data records).

[0057] For example, the first data table could be a merchant information table, which could contain multiple data columns (or field columns) such as merchant ID, registration time, positive review rate, and number of followers. Each data row could be used to record information about one merchant. For example, the second data table could be an e-commerce transaction order table, which could contain multiple data columns such as transaction number, payment time, merchant ID, buyer ID, and traded products. Each data row could be used to record one product transaction.

[0058] Accordingly, in response to a data join query targeting the first and second data tables, the computing device 200 can first determine the join key related to this data join query. Specifically, the computing device 200 can receive a data join query statement targeting the first and second data tables sent by the user through the computing device 100, parse the data join query statement, and obtain the join key contained in the query conditions of the statement. For example, the data join query statement can be the following SQL statement: "EXPLAIN SELECT FROM t1 JOIN t2 ON t1.c1=t2.c1", where t1 is the first data table, t2 is the second data table, and c1 is the join key.

[0059] It should be understood that a join key (or connection key) is a data column field used to link data with other data tables. For example, assuming the first data table is a merchant information table and the second data table is an e-commerce transaction order table, the join key could be "Merchant ID". Thus, using the join key "Merchant ID", one can query the merchant information contained in the first data table and some transaction records related to that merchant contained in the second data table, and so on. Further details will not be elaborated here.

[0060] In one illustrated embodiment, the amount of data contained in the first data table may be smaller than that in the second data table. For example, the size of the first data table may be 64MB, the size of the second data table may be 1GB, and so on; this specification does not specifically limit the size in this regard.

[0061] Furthermore, after determining the association key related to the current data association query, the computing device 200 can scan the first data table to determine the number of data rows contained in the first data table and create a filter whose specifications match the number of data rows. For details, please refer to the description of the embodiment corresponding to Figure 2 below, which will not be elaborated here.

[0062] Furthermore, the computing device 200 can determine the target data column in the first data table that corresponds to the association key, and add at least one association key value contained in the target data column as a filtering element to the filter. For details, please refer to the description of the embodiment corresponding to Figure 2 below, which will not be elaborated further here. For example, taking the first data table as a merchant information table and the association key as "Merchant ID" as an example, the target data column corresponding to the association key is the Merchant ID column, which may include multiple association key values ​​such as "id001", "id002", "id003", and "id004".

[0063] Furthermore, after creating the filter and adding the filter elements, the computing device 200 can filter data rows in the second data table that correspond to at least one of the aforementioned associated key values ​​based on the filter. For example, the computing device 200 can scan each data row in the second data table and determine whether the associated key value in the target data column corresponding to the aforementioned associated key exists in the filter; if not, the data row is filtered out from the second data table; if so, the data row is retained.

[0064] Furthermore, after the computing device 200 completes the filtering of the second data table based on the filter, it can perform data association queries based on the first data table and the filtered second data table.

[0065] As described above, after receiving a data association query initiated by the client, the computing device 200 in this application first determines the number of all data rows contained in the first data table participating in the data association query, and then creates a filter whose specifications match the number of data rows. In this way, when filtering the second data table (larger table) based on the filter, it can not only effectively reduce the false judgment rate of the filter and ensure the filtering effect of the filter on the second data table, but also avoid the filter occupying a large amount of memory space, thereby improving the performance of the data association query.

[0066] Please refer to Figure 2, which is a flowchart illustrating a data association query method provided in an exemplary embodiment. This method can be applied to the system architecture shown in Figure 1, specifically to the computing device 200 within the system architecture shown in Figure 1. As shown in Figure 2, the method may specifically include the following steps S201-S204.

[0067] Step S201: In response to a data association query targeting the first data table and the second data table, determine the association key related to the data association query; wherein the first data table contains less data than the second data table.

[0068] First, in response to a data association query targeting the first data table and the second data table, the computing device 200 can determine the association key related to the current data association query. For details, please refer to the description of the embodiment corresponding to Figure 1 above, which will not be repeated here.

[0069] In one illustrated embodiment, the aforementioned data association query can be a data association query based on a hash join algorithm, combining the first data table and the second data table; this specification does not specifically limit this.

[0070] The first data table can contain less data than the second data table. That is, the first data table is the smaller table among the multiple data tables in this data association query, and the second data table is the larger table among the multiple data tables.

[0071] Step S202: Determine the number of data rows contained in the first data table, and create a filter whose specifications match the number of data rows.

[0072] Furthermore, the computing device 200 can scan the first data table to determine the number of data rows contained in the first data table, and create a filter in memory space whose specifications match the number of data rows. In one illustrated embodiment, the filter can be a Bloom filter. A Bloom filter typically contains a binary vector for adding filter elements and several hash functions. Accordingly, the length (i.e., the number of bits) of the binary vector contained in the created Bloom filter can match the number of data rows contained in the first data table. This ensures that the specifications of the created Bloom filter match the number of data rows.

[0073] For example, assuming the first data table contains 10 data rows, the binary vector of the created Bloom filter can be 10 bits long, that is, the binary vector is a bit array consisting of 10 bits; assuming the first data table contains 50 data rows, the binary vector of the created Bloom filter can be 50 bits, that is, the binary vector is a bit array consisting of 10 bits; assuming the first data table contains 200 data rows, the binary vector of the created Bloom filter can be 200 bits, that is, the binary vector is a bit array consisting of 200 bits, and so on. This specification does not make specific limitations in this regard.

[0074] In one illustrated implementation, taking a distributed query scenario as an example, the first data table may contain multiple shards. Based on this, the computing device 200 may enable multiple parallel threads corresponding one-to-one with the multiple shards to perform data association queries in parallel on the multiple shards of the first data table.

[0075] Specifically, after receiving a data association query statement targeting the first and second data tables, the computing device 200 performs a series of processing steps, including syntax parsing and semantic analysis, on the statement. Finally, the optimizer generates a distributed query plan corresponding to the data association query statement. This distributed query plan may include multiple sub-plans (Data Flow Operations, DFOs) corresponding one-to-one with the multiple shards, and each sub-plan may include at least one operator executed serially.

[0076] In one illustrated implementation, the entire distributed query plan can be managed by a scheduling thread; wherein, as a coordinator of parallel execution (PX), the scheduling thread is responsible for starting multiple computing threads and scheduling each sub-plan in the distributed query plan to the corresponding computing thread among the multiple parallel computing threads for execution.

[0077] Based on this, taking a distributed query scenario as an example, when determining the number of data rows contained in the first data table and creating a filter that matches the number of data rows, the specific steps may include:

[0078] First, the scheduling thread can start multiple first computing threads corresponding to the multiple shards contained in the first data table and multiple second computing threads corresponding to the multiple shards contained in the first data table, and send the first sub-plan contained in the distributed query plan to the multiple first computing threads corresponding to the multiple shards in the first data table, so that each first computing thread executes the first sub-plan.

[0079] In one illustrated embodiment, each first computing thread, when executing the first sub-plan, may include: scanning the corresponding shard among the plurality of shards contained in the first data table, and sending the scanned shard to the second computing thread among the plurality of second computing threads corresponding to the shard.

[0080] In addition, the scheduling thread can also send the second sub-plan contained in the distributed query plan to multiple second computing threads that correspond one-to-one with multiple data shards in the first data table, so that each second computing thread can execute the second sub-plan.

[0081] In one illustrated embodiment, each second computing thread, when executing the second sub-plan, may include: receiving a fragment of the first data table sent by the corresponding first computing thread, determining the number of data rows contained in the received fragment, and sending the determined number of data rows contained in the fragment to the scheduling thread via a message. Correspondingly, the scheduling thread may collect the number of data rows contained in each of the multiple fragments of the first data table determined by the multiple second computing threads, and calculate the total number of data rows contained in the entire first data table. In one illustrated embodiment, when each first computing thread sends its scanned fragment to the corresponding second computing thread, in addition to sending the entire fragment to one corresponding second computing thread, it may also perform a hash calculation and modulo operation on the associated key value contained in each row of data in the scanned fragment, and then send each row of data to the corresponding second computing thread based on the modulo result.

[0082] For example, if the modulo result of the hash value corresponding to the associated key value in any data row is 1, then the data row can be sent to the second calculation thread-1; if the modulo result of the hash value corresponding to the associated key value in any data row is 2, then the data row can be sent to the second calculation thread-2, and so on. This specification does not specifically limit this. Accordingly, each second calculation thread can receive data sent by multiple first calculation threads, count the number of data rows received, and send the counted number of data rows to the scheduling thread, which then summarizes the total number of data rows contained in the entire first data table, and so on. This specification does not specifically limit this.

[0083] Furthermore, the scheduling thread can also send the total number of data rows contained in the first data table obtained from the above statistics to each of the multiple second calculation threads. Correspondingly, when executing the second sub-plan, each second calculation thread can receive the total number of data rows contained in the first data table sent by the scheduling thread and create a filter fragment with a specification that matches the total number of data rows contained in the first data table.

[0084] In one illustrated implementation, the filter shard can be a shard of a distributed Bloom filter. For example, a Bloom filter shard can be created whose length (i.e., number of bits) of the contained binary vector matches the number of data rows contained in the first data table.

[0085] Please refer to Figure 3, which is a schematic diagram of the execution flow of a distributed query plan provided by an exemplary embodiment.

[0086] As shown in Figure 3, the distributed query plan may include a first subplan DFO1, a second subplan DFO2, and a third subplan DFO3. The distributed query plan shown in Figure 3 can be managed by the Px Coordinator operator (not shown in Figure 3) in the scheduling thread.

[0087] As shown in Figure 3, the first sub-plan DFO1 includes a table scan t1 operator that scans a first data table t1, and an exchange out operator. In one illustrated embodiment, the first sub-plan DFO1 can be scheduled by the Px Coordinator operator to be executed in parallel across multiple first computation threads.

[0088] As shown in Figure 3, the second sub-plan DFO2 includes a data exchange-in operator, a join-filter-create operator, and a hash-join operator. In one illustrated embodiment, the second sub-plan DFO2 can also be scheduled by the Px Coordinator operator to be executed in parallel across multiple second computation threads.

[0089] As shown in Figure 3, the third sub-plan DFO3 includes a table scan t2 operator for scanning the second data table t2, an execution filter (join filter use) operator for filtering the second data table t2 using a filter, and an exchange out operator. In one illustrated embodiment, the third sub-plan DFO3 can also be scheduled by the Px Coordinator operator to be executed in parallel across multiple third computation threads.

[0090] In one illustrated implementation, as shown in FIG3, on each first computation thread, the table scan t1 operator in the first sub-plan DFO1 can scan the shards of the first data table that match it, and transmit each data row in the scanned shard to the exchange out operator connected to it. Further, as shown in FIG3, through the exchange in operator in DFO2 connected to the exchange out operator, each data row in the shards of the first data table scanned by the table scan t1 operator can be transmitted to the join filter create operator in DFO2 on the corresponding second computation thread.

[0091] Furthermore, on each second computation thread, the join filter create operator in DFO2 can materialize (i.e. cache) the received data and determine the number of received data rows, i.e., the partial row count of the first data table.

[0092] Furthermore, the join filter create operator on each second calculation thread can send its determined partial row count to the Px Coordinator operator. Correspondingly, the Px Coordinator operator can collect the partial row counts sent by multiple join filter create operators and calculate the total number of data rows contained in the entire first data table, i.e., the total row count of the first data table.

[0093] Furthermore, the Px Coordinator operator can send the total number of rows of the first data table obtained from the statistics to the join filter create operator on each second calculation thread. Correspondingly, after receiving the total number of rows of the first data table sent by the Px Coordinator operator, the join filter create operator on each second calculation thread can create filter fragments with specifications matching the total number of rows of the first data table.

[0094] For example, please refer to Figure 4, which is a schematic diagram illustrating the number of data rows provided in an exemplary embodiment. As shown in Figure 4, taking multiple second calculation threads including second calculation thread-1, second calculation thread-2, and second calculation thread-3 as an example, the join filter create operator on second calculation thread-1 can send its determined partial row count (e.g., partial row count-1) to the Px Coordinator operator; the join filter create operator on second calculation thread-2 can send its determined partial row count (e.g., partial row count-2) to the Px Coordinator operator; and the join filter create operator on second calculation thread-3 can send its determined partial row count (e.g., partial row count-3) to the Px Coordinator operator. Accordingly, the Px Coordinator operator can summarize the received partial row counts-1,-2, and-3 to obtain the total number of rows in the first data table. For example, taking the above partial row count -1 as 10 rows, partial row count -2 as 20 rows, and partial row count -3 as 20 rows as an example, the total number of rows in the first data table obtained by the Px Coordinator operator is 50 rows.

[0095] Furthermore, the Px Coordinator operator can send the total number of rows of the first data table obtained from the statistics to the join filter create operator on the second calculation thread-1, the second calculation thread-2, and the second calculation thread-3. Correspondingly, after receiving the total number of rows of the first data table sent by the Px Coordinator operator, each of the join filter create operators on the second calculation thread-1, the second calculation thread-2, and the second calculation thread-3 can create filter fragments with specifications matching the total number of rows of the first data table.

[0096] For example, taking a first data table with a total of 50 rows, the length of the binary vector contained in the filter slices created by the join filter create operator on the second calculation thread-1, the second calculation thread-2, and the second calculation thread-3 can all be 50 bits.

[0097] Step S203: Determine the target data column in the first data table that corresponds to the association key, and add at least one association key value in the target data column that corresponds to the association key to the filter.

[0098] Furthermore, the computing device 200 can determine the target data column in the first data table that corresponds to the association key of the current data association query, and add at least one association key value in the target data column that corresponds to the association key as a filter element to the filter created above.

[0099] For example, taking the filter as a Bloom filter, the computing device 200 can calculate at least one hash value that corresponds one-to-one with at least one association key value contained in the target data column that corresponds to the association key, and add the at least one hash value as a filter element to the Bloom filter.

[0100] In one of the illustrated implementations, taking a distributed query scenario as an example, the process of adding at least one associated key value corresponding to the associated key contained in the target data column of the first data table to the filter can be specifically executed by the aforementioned multiple second computing threads.

[0101] Specifically, after receiving the shard of the first data table sent by the corresponding first computing thread, each second computing thread can determine the target data column contained in the shard that corresponds to the aforementioned association key, and add at least one association key value contained in the target data column that corresponds to the aforementioned association key to the filter shard it created.

[0102] Please refer to Figure 3. On each second computation thread, after receiving several data rows from the corresponding table scan t1 operator, the join filter create operator can determine the target data column corresponding to the aforementioned association key in those data rows, and add the association key values ​​contained in that target data column to the filter shard it created. For example, the join filter create operator can first calculate at least one hash value that corresponds one-to-one with at least one association key value contained in the target data column that corresponds to the aforementioned association key, and then add that at least one hash value as a filter element to the Bloom filter shard it created, and so on. This specification does not specifically limit this.

[0103] Step S204: Based on the filter, filter the data rows in the second data table that correspond to the at least one associated key value, and perform a data association query based on the first data table and the filtered second data table.

[0104] Furthermore, the computing device 200 can filter data rows in the second data table that correspond to the aforementioned at least associated key values ​​based on the filter. For example, the computing device 200 can scan the second data table and, for any data row in the scanned second data table, determine whether the associated key value contained in the target data column of that data row exists in the filter; if not, filter out that data row from the second data table; if so, retain that data row.

[0105] Furthermore, after filtering the second data table based on the filter, the computing device 200 can perform a data association query based on the first data table and the filtered second data table to obtain the query result. For example, a hash join algorithm can be used to perform the data association query by combining the first data table and the filtered second data table. The specific implementation of this hash join algorithm for combining the first data table and the filtered second data table can be found in the explanation section above, and will not be elaborated further here.

[0106] In one illustrated implementation, taking a distributed query scenario as an example, the second data table can also include multiple shards. Thus, when filtering the second data table based on a filter and performing a data association query based on the first data table and the filtered second data table, multiple third computing threads corresponding one-to-one with the multiple shards in the second data table can be initiated. Each second computing thread can send the filter shard with at least one association key value added to it to each of the multiple third computing threads corresponding to the multiple shards of the second data table. That is, for each second computing thread, the filter shard with the association key value added needs to be sent once to each of the multiple third computing threads, so that each third computing thread can receive multiple filter shards with the association key value added from the multiple second computing threads. Correspondingly, each third computing thread can merge the multiple filter shards received from the multiple second computing threads to obtain a complete filter.

[0107] Furthermore, each third computation thread can also perform attentional scanning on the corresponding shards among the multiple shards of the second data table, and for the data rows in the scanned shard, determine whether the association key value contained in the target data column corresponding to the above association key exists in the filter; if not, the data row can be filtered out from the shard; if so, the data row is retained.

[0108] Furthermore, each third computing thread can also send the shards contained in its filtered second data table to the corresponding second computing threads among multiple second computing threads. Accordingly, each second computing thread can perform data association queries based on the shards contained in the received first data table and the shards contained in the filtered second data table.

[0109] In one illustrated embodiment, when each third computing thread sends a shard of its filtered second data table to its corresponding second computing thread, it can send the entire shard to a single second computing thread, or it can perform a hash calculation and modulo operation on the associated key values ​​contained in each row of data within the shard, and then send each row of data to its corresponding second computing thread based on the modulo result. For example, if the modulo result of the hash value corresponding to the associated key value in any data row is 1, then the data row can be sent to second computing thread-1; if the modulo result of the hash value corresponding to the associated key value in any data row is 2, then the data row can be sent to second computing thread-2, and so on. This specification does not specifically limit this. Correspondingly, each second computing thread can perform efficient data association queries based on a portion of data from the first data table with the same modulo operation and a portion of data from the filtered second data table, and so on. This specification does not specifically limit this.

[0110] Finally, the scheduling thread can obtain the query results obtained by each second computing thread in performing data association queries, obtain multiple query results, and summarize the multiple query results to obtain the final query result of data association queries on the first data table and the filtered second data table.

[0111] Please refer to Figure 3. The join filter create operator on each second computation thread can send its filter fragments to the join filter use operators on multiple third computation threads. Correspondingly, the join filter use operator can receive multiple filter fragments sent by the join filter create operators on multiple second computation threads, and merge the received multiple filter fragments to obtain a complete filter containing all the associated key values ​​contained in the target data column of the first data table.

[0112] Furthermore, the join filter use operator on each third thread can scan the matching shards in multiple shards of the second data table. For any data row in the scanned shard, it determines whether the associated key value contained in the target data column of the data row is in the filter. If not, the data row is filtered out from the shard; if so, the data row is retained.

[0113] Furthermore, as shown in Figure 3, the join filter use operator on each third thread can send each row of data in the shards of the filtered second data table to the hash join operator in the corresponding second calculation thread through the exchange out operator and the exchange in operator.

[0114] Furthermore, the hash join operator on each second computing thread can perform data association queries by combining a portion of the data it receives from the first data table and a portion of the filtered data from the second data table based on the hash join algorithm.

[0115] Finally, the Px Coordinator operator can collect the data association query results (e.g., hash join results) calculated by the hash join operator on each second computation thread, obtain multiple data association query results, and then summarize these multiple data association query results to obtain the final association query result.

[0116] In one illustrated embodiment, in a distributed query scenario, the first data table and the second data table can be partitioned tables stored in a single node (i.e., computing device 200).

[0117] Accordingly, the multiple shards of the first data table can be multiple partitions obtained by dividing the data in the first data table based on the first partition key; the multiple first computing threads can be computing threads running in the single node (i.e., computing device 200).

[0118] Accordingly, the multiple shards of the second data table can be multiple partitions obtained by dividing the data in the second data table based on the second partition key; the multiple three computing threads can also be computing threads running in the single node (i.e., computing device 200).

[0119] Furthermore, the aforementioned scheduling thread and multiple second computing threads can all be computing threads running within this single node (i.e., computing device 200).

[0120] In one illustrated embodiment, the first partition key and the second partition key can be the same as the aforementioned association key. For example, if the first data table is a merchant information table and the second data table is an e-commerce transaction order table, the first partition key, the second partition key, and the association key can all be "Merchant ID".

[0121] In one illustrated embodiment, the first partition key, the second partition key, and the association key can also be different. For example, assuming the first data table is a merchant information table and the second data table is an e-commerce transaction order table, the association key could be "Merchant ID," the first partition key could be "Registration Time," and the second partition key could be "Transaction Product"; alternatively, both the first partition key and the association key could be "Merchant ID," and the second partition key could be "Transaction Product," etc. This specification does not impose specific limitations on these possibilities.

[0122] In one of the illustrated embodiments, in a distributed query scenario, the first and second data tables can be distributed across multiple nodes in a distributed cluster, i.e., the database shown in Figure 1 can be a distributed database.

[0123] For example, the first data table can be distributed and stored in multiple first computing nodes included in the distributed cluster. Correspondingly, the multiple first computing threads can include first computing threads running on each of the multiple first computing nodes, that is, a first computing thread is started in each first computing node.

[0124] For example, the second data table can also be distributed and stored in multiple second computing nodes included in the distributed cluster. Correspondingly, the multiple third computing threads can include third computing threads running on each of the multiple second computing nodes, that is, a third computing thread is started in each second computing node.

[0125] For example, please refer to Figure 5, which is a schematic diagram of a system architecture for a distributed cluster provided in an exemplary embodiment. As shown in Figure 5, the distributed cluster may include a scheduling node, multiple first computing nodes, and multiple second computing nodes, such as first computing node 100a, first computing node 100b, second computing node 200a, and second computing node 200b. In one illustrated embodiment, the scheduling node, first computing node 100a, first computing node 100b, second computing node 200a, and second computing node 200b can establish communication connections through any possible means, such as wired or wireless networks; this specification does not specifically limit this.

[0126] As shown in Figure 5, the scheduling node can run the aforementioned scheduling thread. This scheduling node can interface with the client, responsible for receiving data association query requests initiated by the user, scheduling multiple computing nodes in the distributed cluster to execute the data association query in parallel, and returning the corresponding query results to the user, etc. It is easy to understand that in the architecture of the distributed cluster, the aforementioned computing device 200 can be the scheduling node shown in Figure 5.

[0127] In one embodiment shown, the scheduling node may also be one of the plurality of first computing nodes and plurality of second computing nodes shown in FIG5, and this specification does not specifically limit it.

[0128] As shown in Figure 5, the first computing node 100a and the first computing node 100b, etc., distribute and store the first data table, that is, the first computing node 100a and the first computing node 100b each store a fragment of the first data table. The first computing thread runs in the first computing node 100a and the first computing node 100b.

[0129] Please refer to Figure 3. When the first computing thread running in the first computing node 100a executes the first sub-plan DFO1, the table scan t1 operator is used to scan the shards of the first data table stored locally in the first computing node 100a. When the first computing thread running in the first computing node 100b executes the first sub-plan DFO1, the table scan t1 operator is used to scan the shards of the first data table stored locally in the first computing node 100b.

[0130] As shown in Figure 5, the second computing node 200a and the second computing node 200b, respectively, store the second data table in a distributed manner; that is, the second computing node 200a and the second computing node 200b each store a fragment of the second data table. The third computing thread runs on each of the second computing nodes 200a and 200b.

[0131] Please refer to Figure 3. When the third computing thread running in the second computing node 200a executes the third sub-plan DFO3, the table scan t2 operator is used to scan the shards of the second data table stored locally in the second computing node 200a; when the third computing thread running in the second computing node 200b executes the third sub-plan DFO3, the table scan t2 operator is used to scan the shards of the second data table stored locally in the second computing node 200b.

[0132] It should be noted that this application does not specifically limit the storage location of the first data table and the second data table. In one illustrated embodiment, the first data table can be distributed and stored in the external storage devices (e.g., disks, solid-state drives, etc.) of the first computing node 100a and the first computing node 100b, and the second data table can also be distributed and stored in the external storage devices (e.g., disks, solid-state drives) of the second computing node 200a and the second computing node 200b, etc. This specification does not specifically limit this.

[0133] In one illustrated embodiment, the scheduling node, the first computing node 100a, the first computing node 100b, the second computing node 200a, and the second computing node 100b may be a laptop computer, a desktop computer, a server, or a server cluster consisting of multiple servers, etc., which are not specifically limited in this specification.

[0134] It should be understood that the system architecture of the distributed cluster shown in Figure 5 is only illustrative. In some possible implementations, the distributed cluster may include more or fewer devices than shown in Figure 1. For example, it may include multiple computing nodes running the second computing thread mentioned above, etc. This specification does not make any specific limitations in this regard.

[0135] In one of the illustrated embodiments, in a non-distributed query scenario, for example, if both the first data table and the second data table are completely stored in the computing device 200, and the first data table and the second data table are not divided into multiple partitions, then the computing device 200 does not need to start multiple parallel threads, but can implement the above method steps through conventional serial threads, etc. This specification does not make specific limitations in this regard.

[0136] Additionally, in some possible implementations, if the data join query targets three or more tables, the above-described process can also be used to filter multiple larger tables. First, determine the number of rows in the smaller table and construct a filter matching that number. Then, add several join key values ​​from the target rows in the smaller table that correspond to the join key to the filter. Finally, use this filter to filter the remaining larger tables, and so on.

[0137] In some possible implementations, if the data association query targets multiple association keys, multiple filters corresponding to the multiple association keys can be constructed through the above method flow, etc., which are not specifically limited in this specification.

[0138] In summary, this application first determines the number of all data rows contained in the first data table (smaller table) participating in the data association query, and then creates a filter whose specifications match the number of data rows. This allows for effective reduction of the filter's false positive rate and ensures the filter's filtering effect on the second data table when filtering the second data table (larger table) based on this filter. It also avoids the filter occupying a large amount of memory space, thereby improving the performance of the data association query.

[0139] Corresponding to the above method flow, embodiments of this specification also provide a data association query device. Please refer to Figure 6, which is a schematic diagram of the structure of a data association query device provided in an exemplary embodiment. This device 60 can be applied to the computing device 200 in the system architecture shown in Figure 1. As shown in Figure 6, the device 60 includes:

[0140] Association key determination unit 601 is used to determine the association key related to the data association query in response to the data association query for the first data table and the second data table; wherein the first data table contains less data than the second data table; filter creation unit 602 is used to determine the number of data rows contained in the first data table and create a filter with specifications matching the number of data rows;

[0141] Adding unit 603 is used to determine the target data column in the first data table that corresponds to the association key, and add at least one association key value in the target data column that corresponds to the association key to the filter;

[0142] The filtering unit 604 is used to filter the data rows in the second data table that correspond to the at least one associated key value based on the filter, and to perform data association queries based on the first data table and the filtered second data table.

[0143] In one illustrated embodiment, the filter unit 604 is specifically used for:

[0144] Scan each data row in the second data table and determine whether the associated key value contained in the target data column corresponding to the associated key in the scanned data row exists in the filter; if not, filter out the data row from the second data table.

[0145] In one illustrated embodiment, the first data table comprises multiple shards; the filter creation unit 602 is specifically used for:

[0146] Start multiple first computing threads corresponding one-to-one with multiple shards contained in the first data table and multiple second computing threads corresponding one-to-one with multiple shards contained in the first data table. Each first computing thread scans the shards corresponding to it in the multiple shards contained in the first data table and sends the scanned shards to the second computing thread corresponding to the shards in the multiple second computing threads.

[0147] Each second computation thread determines the number of data rows contained in the received shard in the first data table;

[0148] The number of data rows contained in the shards of the first data table received by each second computing thread is obtained, and the total number of data rows contained in the first data table is further calculated based on the number determined by each second computing thread.

[0149] In one illustrated embodiment, the filter creation unit 602 is specifically used for:

[0150] The total number of data rows in the first data table, obtained from the statistics, is sent to each of the plurality of second calculation threads, causing each second calculation thread to create a filter shard with a specification matching the total number of data rows. In one illustrated embodiment, the adding unit 603 is specifically used for:

[0151] Each second computing thread determines the target data column corresponding to the association key contained in the received shard of the first data table, and adds at least one association key value corresponding to the association key contained in the target data column to the filter shard.

[0152] In one illustrated embodiment, the second data table includes multiple shards; the filtering unit 604 is specifically used for:

[0153] Start multiple third computing threads that correspond one-to-one with multiple shards in the second data table, and have each second computing thread send the filter shard with the at least one associated key value added to each of the multiple third computing threads respectively;

[0154] Each third computing thread merges the multiple filter fragments received from the multiple second computing threads to obtain a complete filter; and scans each data row in the corresponding fragment of the multiple fragments contained in the second data table, and determines whether the association key value contained in the target data column corresponding to the association key in the scanned data row exists in the filter; if not, the data row is filtered out from the fragment. In one illustrated embodiment, the filtering unit 604 is specifically used for:

[0155] Each third computing thread sends the shard contained in the filtered second data table to the second computing thread corresponding to the shard among the plurality of second computing threads;

[0156] Each second computing thread performs a data association query based on the shards contained in the first data table and the filtered shards contained in the second data table;

[0157] The query results obtained by the multiple second computing threads performing data association queries are obtained, and the multiple query results are summarized to obtain the query results of data association queries performed on the first data table and the filtered second data table.

[0158] In one illustrated embodiment, the first data table is distributed across multiple first computing nodes in a distributed cluster; the multiple first computing threads include first computing threads running on each of the multiple first computing nodes.

[0159] In one illustrated embodiment, the second data table is distributed across multiple second computing nodes contained in the distributed cluster; the multiple third computing threads include third computing threads that run on each of the multiple second computing nodes.

[0160] In one illustrated embodiment, the first data table is a partitioned table stored in a single node, and the multiple shards of the first data table include multiple partitions obtained by dividing the data in the first data table based on a first partition key; the multiple first computing threads include multiple first computing threads running in the single node.

[0161] In one illustrated embodiment, the second data table is a partitioned table stored in a single node, and the multiple shards of the second data table include multiple partitions obtained by dividing the data in the second data table based on a second partition key; the multiple third computing threads include multiple third computing threads running in the single node.

[0162] In one illustrated embodiment, the filter is a Bloom filter.

[0163] In one illustrated embodiment, the Bloom filter includes a binary vector for adding filter elements, the length of which matches the number of data rows contained in the first data table.

[0164] In one illustrated embodiment, the adding unit 603 is specifically used for:

[0165] Adding at least one associated key value corresponding to the associated key contained in the target data column to the filter includes:

[0166] Calculate at least one hash value that corresponds one-to-one with at least one association key value contained in the target data column that corresponds to the association key, and add the at least one hash value as a filter element to the Bloom filter.

[0167] In one illustrated embodiment, the filter unit 604 is specifically used for:

[0168] A data association query is performed on the first data table and the filtered second data table based on the hash join algorithm.

[0169] The specific implementation process of the functions and roles of each unit in the aforementioned device 60 is detailed in the description of the above embodiments and will not be repeated here. It should be understood that the aforementioned device 60 can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor (CPU) of the device loading the corresponding computer program instructions into memory for execution. From a hardware perspective, in addition to the CPU and memory, the device typically includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards for implementing network communication functions.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of the solution described in this specification, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0171] The devices, units, and modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, in-vehicle computer, or any combination of these devices.

[0172] Corresponding to the above-described method embodiments, this specification also provides a computing device. Please refer to FIG7, which is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. The computing device shown in FIG7 can be the computing device 200 in the system architecture shown in FIG1 above. As shown in FIG7, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (e.g., a keyboard) and an output device 1005 (e.g., a display). The processor 1001, memory 1002, input device 1004, and output device 1005 can be connected via a bus or other means. As shown in FIG7, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program executable by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit used to control the execution of the above method embodiments. When the processor 1001 runs the stored computer program, it can execute various steps of the data association query method in the embodiments of this specification, including: in response to a data association query for a first data table and a second data table, determining an association key related to the data association query; determining the number of data rows contained in the first data table and creating a filter that matches the number of data rows; determining a target data column contained in the first data table corresponding to the association key, and adding a plurality of association key values ​​contained in the target data column to the filter; filtering the data rows contained in the second data table corresponding to the plurality of association key values ​​based on the filter, and performing a data association query based on the first data table and the filtered second data table, etc.

[0173] For a detailed description of each step of the above data association query method, please refer to the previous content, which will not be repeated here.

[0174] Corresponding to the above method embodiments, embodiments of this specification also provide a computer-readable storage medium storing computer programs that, when run by a processor, execute the various steps of the data association query method in the embodiments of this specification. Please refer to the description of the above embodiments for details, which will not be repeated here.

[0175] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0176] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0177] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0178] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data.

[0179] Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0181] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A data association query method, characterized in that, The method includes: In response to a data association query targeting a first data table and a second data table, a association key related to the data association query is determined; wherein the first data table contains less data than the second data table; Determine the number of data rows contained in the first data table, and create a filter whose specifications match the number of data rows; Determine the target data column in the first data table that corresponds to the association key, and add at least one association key value in the target data column that corresponds to the association key to the filter; Based on the filter, data rows in the second data table that correspond to the at least one associated key value are filtered, and data association queries are performed based on the first data table and the filtered second data table.

2. The method according to claim 1, characterized in that, The step of filtering data rows in the second data table corresponding to the at least one associated key value based on the filter includes: Scan each data row in the second data table and determine whether the associated key value contained in the target data column corresponding to the associated key in the scanned data row exists in the filter; if not, filter out the data row from the second data table.

3. The method according to claim 2, characterized in that, The first data table contains multiple shards; Determining the number of data rows contained in the first data table includes: Start multiple first computing threads corresponding one-to-one with multiple shards contained in the first data table and multiple second computing threads corresponding one-to-one with multiple shards contained in the first data table. Each first computing thread scans the shards corresponding to it in the multiple shards contained in the first data table and sends the scanned shards to the second computing thread corresponding to the shards in the multiple second computing threads. Each second computation thread determines the number of data rows contained in the received shard in the first data table; The number of data rows contained in the shards of the first data table received by each second computing thread is obtained, and the total number of data rows contained in the first data table is further calculated based on the number determined by each second computing thread.

4. The method according to claim 3, characterized in that, The filter that creates a specification that matches the number of data rows includes: The total number of data rows in the first data table obtained from the statistics is sent to each of the plurality of second computing threads, so that each second computing thread creates a filter fragment with a specification that matches the total number of data rows.

5. The method according to claim 4, characterized in that, The step of determining the target data column in the first data table that corresponds to the association key, and adding at least one association key value in the target data column that corresponds to the association key to the filter, includes: Each second computing thread determines the target data column corresponding to the association key contained in the received shard of the first data table, and adds at least one association key value corresponding to the association key contained in the target data column to the filter shard.

6. The method according to claim 5, characterized in that, The second data table includes multiple shards; The step of scanning each data row in the second data table and determining whether the associated key value contained in the target data column corresponding to the associated key in the scanned data row exists in the filter, and if not, filtering out the data row from the second data table, includes: Start multiple third computing threads that correspond one-to-one with multiple shards in the second data table, and have each second computing thread send the filter shard with the at least one associated key value added to each of the multiple third computing threads respectively; Each third computing thread merges the multiple filter fragments received from the multiple second computing threads to obtain a complete filter; and scans each data row in the multiple fragments contained in the second data table and the corresponding fragment, and determines whether the association key value contained in the target data column corresponding to the association key in the scanned data row exists in the filter; if not, the data row is filtered out from the fragment.

7. The method according to claim 6, characterized in that, The data association query based on the first data table and the filtered second data table includes: Each third computing thread sends the shard contained in the filtered second data table to the second computing thread corresponding to the shard among the plurality of second computing threads; Each second computing thread performs a data association query based on the shards contained in the first data table and the filtered shards contained in the second data table; The query results obtained by the multiple second computing threads performing data association queries are obtained, and the multiple query results are summarized to obtain the query results of data association queries performed on the first data table and the filtered second data table.

8. The method according to claim 3, characterized in that, The first data table is distributed and stored in multiple first computing nodes contained in the distributed cluster; the multiple first computing threads include first computing threads that run on each of the multiple first computing nodes respectively.

9. The method according to claim 6, characterized in that, The second data table is distributed and stored in multiple second computing nodes contained in the distributed cluster; the multiple third computing threads include third computing threads that run on each of the multiple second computing nodes respectively.

10. The method according to claim 3, characterized in that, The first data table is a partitioned table stored in a single node, and the multiple shards of the first data table include multiple partitions obtained by dividing the data in the first data table based on the first partition key; the multiple first computing threads include multiple first computing threads running in the single node.

11. The method according to claim 6, characterized in that, The second data table is a partitioned table stored in a single node. The multiple shards of the second data table include multiple partitions obtained by dividing the data in the second data table based on the second partition key. The multiple third computing threads include multiple third computing threads running in the single node.

12. The method according to claim 1, characterized in that, The filter is a Bloom filter.

13. The method according to claim 12, characterized in that, The Bloom filter contains a binary vector for adding filter elements, the length of which matches the number of data rows contained in the first data table.

14. The method according to claim 13, characterized in that, Adding at least one associated key value corresponding to the associated key contained in the target data column to the filter includes: Calculate at least one hash value that corresponds one-to-one with at least one association key value contained in the target data column that corresponds to the association key, and add the at least one hash value as a filter element to the Bloom filter.

15. The method according to any one of claims 1-14, characterized in that, The data association query based on the first data table and the filtered second data table includes: A data association query is performed on the first data table and the filtered second data table based on the hash join algorithm.

16. A data association query device, characterized in that, The device includes: The association key determination unit is used to determine the association key related to the data association query in response to a data association query for a first data table and a second data table; wherein the first data table contains less data than the second data table; A filter creation unit is used to determine the number of data rows contained in the first data table and create a filter whose specifications match the number of data rows; An adding unit is used to determine the target data column in the first data table that corresponds to the association key, and to add at least one association key value in the target data column that corresponds to the association key to the filter; The filtering unit is used to filter data rows in the second data table that correspond to the at least one associated key value based on the filter, and to perform data association queries based on the first data table and the filtered second data table.

17. A computing device, characterized in that, include: Memory and processor; The memory stores computer programs / instructions that can be executed by the processor; When the processor runs the computer program / instructions, it performs the method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, It stores computer programs / instructions thereon, which, when executed by a processor, implement the method as described in any one of claims 1-15.

19. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-15.

Images