An implementation method of cross-model query pushdown, electronic equipment, storage medium and product

Abstract

The application discloses an implementation method of cross-model query pushdown, electronic equipment, a storage medium and a product, a first data model is scanned according to a query condition of the first data model, a query result set meeting the query condition is returned, and the query result set is put into an intermediate result set; assuming that a query result set of a current data model and a query result set of a next data model are connected according to an ID key, a Bloom filter is constructed based on a key value of the ID key; a storage engine filters the query result of the next data model by using the Bloom filter, and obtains a query result meeting a condition of the Bloom filter; each query result of the query result set of the next data model and the intermediate result set is connected by an associated ID key through hash connection, the query result of successful connection is added to a new intermediate result set, the new intermediate result set is used to sequentially execute the next data model; and the intermediate result set after all query operators are executed is the final result set.

Description

Technical Field

[0001] This invention belongs to the field of database technology, specifically relating to a method, electronic device, storage medium, and product for implementing cross-model query pushdown. Background Technology

[0002] Generally speaking, multi-model databases support mixed queries of multiple models. This means that a single query statement may contain queries on multiple data models at the same time, and there is a specific order to these queries. Therefore, how to efficiently select and merge the query results of each data model is a crucial issue that needs to be addressed when multi-model databases perform cross-model queries.

[0003] Multi-model databases can simultaneously support read and write operations for various data models, including relational models, document models, vector models, graph models, and key-value pair models. However, when performing cross-model mixed queries, the common approach is to push the query conditions of each data model down to the storage engine, where data filtering is performed. The query results that meet the conditions are then returned to the computation engine, where data joins, merges, and filters. While this approach reduces network transmission overhead and computational costs by pushing down the query conditions of each model, the results from each model may not satisfy the requirements of other models. This is especially true when dealing with large amounts of data, inevitably leading to significant unnecessary network transmission overhead. Furthermore, the computation engine still needs to join, merge, and filter large amounts of data from each model, resulting in low disk I / O efficiency, high memory usage, and high CPU usage, ultimately leading to poor query efficiency and performance for cross-model queries. Summary of the Invention

[0004] Purpose of the invention: To address the problems of existing scheduling methods not making precise use of hardware resources and lacking the ability to detect and process hot data, this invention proposes an implementation method, electronic device, storage medium, and product for cross-model query pushdown in multi-model databases, particularly an implementation method based on Bloom filters, which greatly reduces the amount of data transmitted over the network and calculated by the computing engine in cross-model queries, thereby improving the query performance and efficiency of cross-model queries in multi-model databases.

[0005] Technical solution: Firstly, a method for implementing cross-model query pushdown, including the following steps:

[0006] Step 1: Based on the query statement that includes mixed queries on multiple data models, determine the order of the query operators corresponding to each data model;

[0007] Step 2: Execute the corresponding query operators in sequence to obtain the final query result set;

[0008] Specifically, when executing the query operator of the first data model in the computing engine, the first data model is scanned according to the query conditions of the first data model, the query result set that satisfies the query conditions is returned, and it is placed into the intermediate result set.

[0009] For each remaining query operator, perform the query as follows:

[0010] After obtaining the query result set of the current data model, assuming that the query result set of the current data model and the query result set of the next data model are connected according to a certain ID key, a Bloom filter is constructed based on the key value of the ID key; the constructed Bloom filter is then pushed down to the storage engine.

[0011] After receiving a query request for the next data model, the storage engine uses a Bloom filter to filter the query results of the next data model, obtains the query results that meet the conditions of the Bloom filter, and adds the query results to the query result set of the next data model.

[0012] After receiving the query result set of the next data model, the computing engine performs a hash join with each query result in the intermediate result set using the associated ID key. Query results that are successfully joined are added to the new intermediate result set, while query results that fail to join are discarded. Once all query results have been joined, the computing engine releases the query result set of the next data model it received.

[0013] Using the new intermediate result set, execute the next data model sequentially;

[0014] The intermediate result set after all query operators have been executed is the final result set.

[0015] Further, construct a Bloom filter by following these steps:

[0016] Calculate the optimal bit array size n and the number of hash functions for the Bloom filter;

[0017] Choose the hash function seed and the hash function;

[0018] Based on the optimal bit array size n, initialize a bit array of size n, with a value of 0 and a data type of bit;

[0019] Based on the key value of the ID key in the query result set of the previous data model and the query result set of the current data model, obtain the bit array after inserting the ID key;

[0020] The bit array after the inserted ID key, the hash function name, the number of hash functions, and the hash function seed are serialized and encoded in a unified manner to complete the construction of the Bloom filter.

[0021] Furthermore, the step of obtaining the bit array after inserting the ID key based on the key value of the ID key connecting the query result set of the previous data model and the query result set of the current data model includes:

[0022] Using the selected hash function seed and hash function, the key value of the ID key that connects the query result sets of the two data models in the query result set of the previous data model is hashed. The number of hash calculations is determined by the number of hash functions, and the calculation method depends on the number of hash function seeds. Finally, k hash values ​​are obtained, where k represents the number of hash functions.

[0023] Perform a modulo operation on each hash value to obtain the bit index;

[0024] Perform division and modulo operations on the bit indices to obtain the byte index and bit offset; the byte index represents the index value in the bit array, and the bit offset represents the offset position in a byte.

[0025] Set the values ​​corresponding to the byte index and bit offset to 1.

[0026] Furthermore, the step of using a Bloom filter to filter the query results of the next data model to obtain query results that meet the conditions of the Bloom filter includes:

[0027] Retrieve the key value of the ID key that is responsible for connecting the query result set of the current data model to the query result set of the next data model;

[0028] Based on the hash function, the number of hash functions, and the hash function seed, the ID key value is hashed k times to obtain k hash values, where k represents the number of hash functions;

[0029] Calculate the positions of k hash values ​​in the bit array after inserting the ID key;

[0030] The algorithm checks whether the bit position of the hash value in the bit array is 1. Only when all the bits corresponding to the k hash values ​​of an ID key are 1, it means that the conditions of the Bloom filter are met; otherwise, it means that the conditions of the Bloom filter are not met.

[0031] Secondly, the present invention provides an electronic device, the electronic device comprising:

[0032] At least one processor;

[0033] and a memory communicatively connected to the at least one processor;

[0034] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform a method for implementing cross-model query pushdown.

[0035] Thirdly, the present invention proposes a computer-readable storage medium storing computer instructions for causing a processor to implement a method for cross-model query pushdown.

[0036] Fourthly, the present invention proposes a computer program product, which includes a computer program that, when executed by a processor, implements a method for cross-model query pushdown.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0038] (1) The method of the present invention effectively reduces network transmission overhead, memory usage and computational latency by using query pushdown based on Bloom filter, thereby improving query performance and efficiency when querying across models;

[0039] (2) The method of the present invention is designed for the multi-data model characteristics of multi-model databases, which greatly improves the query efficiency and performance of multi-model database services when querying across models;

[0040] (3) The method of the present invention uses relatively low cost to quickly eliminate a large amount of irrelevant data in the storage engine closest to the data source. Attached Figure Description

[0041] Figure 1 A flowchart illustrating the construction process of a Bloom filter;

[0042] Figure 2 This is a flowchart illustrating the usage of a Bloom filter.

[0043] Figure 3 This is a flowchart of the execution process for cross-model query pushdown based on Bloom filters. Detailed Implementation

[0044] Example 1:

[0045] When a query statement in a multi-model database contains mixed queries on multiple data models, the computing engine will execute the query operators corresponding to each data model sequentially, according to the order of the query operators generated in the executor, using a query pushdown implementation method based on Bloom filters proposed in this embodiment of the invention. This enables cross-model queries in the multi-model database. Figure 3 As shown, the specific execution process is as follows:

[0046] Step 1: Execute the query operator for the first data model in the computing engine. At this time, there is no need to build a Bloom filter. After receiving the query request, the storage engine scans and filters the current data model at the data source according to the pushed-down query conditions and returns a query result set that meets the conditions. After receiving the result set, the computing engine puts it into the intermediate result set.

[0047] Step 2: After obtaining the query result set of the current data model (denoted as A), prepare to execute the query operator of the next data model (denoted as B) in the computing engine. Since the query result set of A and the query result set of B are joined based on a certain ID key, a Bloom filter is constructed for the key value of this ID key. After the Bloom filter is constructed, the filtering conditions of B are pushed down to the storage engine together with the Bloom filter.

[0048] Step 3: After receiving the query request from B, the storage engine uses the Bloom filter to filter the query results for B. After filtering, the result set that may meet the conditions is returned to the computing engine. Although the returned result set is not an exact result set, the number of result sets has been greatly reduced.

[0049] Step 4: After receiving the result set of B that may meet the conditions, the calculation engine performs a hash connection with each result set of A through the associated ID key. If the connection is successful, it means that the data fully meets the conditions and is put into the intermediate result set; if the connection fails, it means that the data is false data generated by the Bloom filter and does not meet the conditions, and is discarded directly.

[0050] Step 5: Repeat steps 2 to 4 to continue executing the remaining query operators, and finally obtain the final result set of the cross-model hybrid query.

[0051] In this embodiment of the invention, a Bloom filter is constructed according to the following steps. A Bloom filter is an efficient probabilistic data structure implemented using a bit array. It is used to quickly determine whether an element absolutely does not exist in a set, or may exist in that set. The construction process of the Bloom filter is completed in the query operator of the computation engine. Figure 1 As shown, it includes:

[0052] S100: Calculate the optimal bit array size and the number of hash functions.

[0053] As one implementation method, the optimal bit array size (denoted as n) and the number of hash functions (denoted as k) can be calculated using the classic theoretical formula of the Bloom filter. This classic theoretical formula has four core parameters: the number of keys, the false positive rate, the bit array size, and the number of hash functions. The number of keys comes from the actual size of the query results of the previous data model; the false positive rate is a set acceptable error probability. The higher the false positive rate, the less space is used and the lower the accuracy; the lower the false positive rate, the more space is used and the higher the accuracy; the bit array size can be calculated based on the number of keys and the false positive rate; the number of hash functions can also be calculated based on the number of keys and the false positive rate.

[0054] S110: Select the hash function seed and hash function.

[0055] As one implementation method, the choice of hash function seed can be determined according to the actual situation. A fixed constant value can be selected as the seed, a dynamically generated random number can be used as the seed, or an ID or sequence number with practical meaning can be used as the seed (set as hash_seed); and the hash function can be a mainstream algorithm such as MurmurHash.

[0056] S120: Initialization of the bit array. Based on the optimal bit array size (n) calculated in S100, initialize an array of size n, with values ​​of 0 and data type bit, called the bit array (denoted as bit_array).

[0057] S130: Insertion of a bit set.

[0058] As one implementation method, the specific operation is as follows:

[0059] S130_1: Calculate the hash value. Using the hash function seed and hash function selected in S120, perform a hash calculation on the ID key value of the data model result set that is responsible for connecting the two data model result sets in the query result set of the previous data model query operator. The number of hash calculations is determined by the number of hash functions (k) obtained in S100, and the number of hash function seeds is determined according to the calculation method. That is, each connection ID key value needs to be hashed k times, and each hash calculation uses a hash function, finally obtaining k hash values ​​(hash_key). For example: if the current query operator queries the relational model, and the next query operator queries the document model, then it is necessary to perform a hash calculation on the connection ID key value in the relevant dataset of the document model to be queried.

[0060] S130_2: Hash value modulo operation. To ensure that the bit index is between [0, n-1], perform a modulo operation (hash_key%n) on each hash value (hash_key) calculated in S131 to obtain the bit index (position);

[0061] S130_3: Byte index and bit offset operations. The bit index (position) calculated in S132 is divided (position / 8) and moduloed (position%8) to obtain the byte index (byte_index, the index value in the bit array, starting from 0) and bit offset (bit_offset, the offset position in a byte, starting from 0).

[0062] S130_4: Sets the value of the specified bit to 1. For example, sets the value of the bit_offset bit of the byte_index byte in the bit array to 1.

[0063] S140: Filter serialization. The bit array (bit_array), hash function name, number of hash functions (k), and hash function seed (hash_seed) are serialized and encoded in a unified manner, to be pushed down to the storage engine.

[0064] In this embodiment of the invention, for the Bloom filter function to take effect, the construction of the computing engine and the corresponding storage engine filtering must use exactly the same hash function, number of hash functions, and hash function seed. In the storage engine, the Bloom filter is used to filter out data that absolutely does not meet the requirements, returning only data that may meet the requirements. The usage process is completed within the storage engine, such as... Figure 2 As shown, the following is the process of using a Bloom filter in cross-model queries:

[0065] S200: Deserialize and decode the Bloom filter to obtain a bit array (bit_array), hash function name, number of hash functions (k), and hash function seed (hash_seed).

[0066] S210: Data filtering. Specific operations include:

[0067] S210_1: Scan the data source, filter the relevant data of the current query model using the current query conditions, and obtain the data that meets the conditions of the current query model. For example: if the current query is a relational model, it is necessary to use the relational model's filtering conditions to filter out the relational model data that meets the requirements;

[0068] S210_2: For each piece of data that meets the conditions of the current query model, obtain the ID key value that is responsible for connecting with the query result set of the previous data model;

[0069] S210_3: Based on the hash function, the number of hash functions (k), and the hash function seed, perform k hash calculations on the ID key value to obtain k hash values;

[0070] S210_4: Calculate the position of the k hash values ​​in the bit array, that is, the bit_offset position of the byte_index byte in the bit array;

[0071] S210_5: Check if the corresponding bit in S210_4 is 1. Only when all the bits corresponding to the k hash values ​​of an ID key are 1 can it be considered that the condition may be met; otherwise, it means that the condition is not met.

[0072] S210_6: Put the data that may meet the conditions of S210_5 into the result set, and wait to return it to the calculation engine.

[0073] Example 2:

[0074] This invention proposes a method for implementing cross-model query pushdown based on Bloom filters in a multi-model database, comprising the following steps:

[0075] Suppose there is a query request that wants to recommend a restaurant. The restaurant needs to meet the following requirements: (1) the restaurant's menu or customer reviews mention "rich flavor" or "fresh, fragrant and spicy", (2) it has been liked or recommended by friends or friends of friends, and (3) there are seats available in the evening and the average cost per person is less than 100 yuan.

[0076] Among them, (1) document query is used, in which unstructured data such as menus and reviews are stored in the document model; (2) graph query is used, in which highly related, semi-structured data such as social networks of friends are stored in the graph model; and (3) relation query is used, in which structured data such as basic restaurant information are stored in the relation model. The queries of the three different data models are connected through the restaurant ID (set as 'rid'). The execution order of the query operators in the calculation engine executor is as follows: document query -> graph query -> relation query.

[0077] Assume that the amount of data in the current multi-model database that satisfies the query conditions for each data model is shown in Table 1, and the values ​​of rid in the query result sets for each data model are shown in Table 2:

[0078] Table 1 Data volume of each data model

[0079]

[0080] The rid in the query results of each data model in Table 2

[0081]

[0082] Execute the query operator of the first data model, namely the query operator of the document model, and obtain restaurant data that meet the document filtering conditions (menus mention "rich flavor" or "fresh, fragrant and spicy") as shown in Tables 1 and 2. Put the query results of the document query operator into the intermediate result set.

[0083] The second data model's query operator, the graph model's query operator, is executed, using the intermediate result set returned by the document model to construct the graph model's Bloom filter. This includes:

[0084] The optimal bit array size (assumed to be 128) and the number of hash functions (assumed to be 2) are calculated based on the classic theoretical formula of the Bloom filter.

[0085] Choose the Bloom filter hash function seed (assuming random numbers 84354565 and 76854767), and choose the hash function (assuming MurmurHash); initialize the bit array (bit_array) as shown in Table 3 below.

[0086] Table 3 Initialize bit array

[0087]

[0088] The rids of the intermediate result set of the document model query are obtained, as shown in Table 2. There are a total of 20 records. For each rid, the hash value is calculated twice using hash function seeds 84354565 and 76854767. The calculation results are shown in Table 4 below.

[0089] The calculated hash value is moduloed (%128) with the size of the bit array to obtain the bit index. Then, the bit index is subjected to byte index operation ( / 8) and offset operation (%8) to obtain the byte index and the offset in the byte. The results are shown in Table 4 below.

[0090] Table 4. Document RID hash calculation results

[0091]

[0092]

[0093] Based on the bit index, byte index, and offset in Table 4, modify the corresponding positions in the bit array to 1. The modified bit array is shown in Table 5 below:

[0094] Table 5: Bit array after inserting the connection ID

[0095]

[0096] The bit array, hash function name (MurmurHash), number of hash functions (2), and hash function seeds (84354565, 76854767) in Table 5 are serialized and encoded in a unified manner, and will be pushed down to the storage engine.

[0097] The query identifies restaurants in the graph model that meet the following criteria: "liked or recommended by a friend or a friend of a friend".

[0098] The deserialized Bloom filter is pushed down to obtain the bit array, hash function name (MurmurHash), number of hash functions (2) and hash function seed (84354565, 76854767) in Table 5.

[0099] Based on the filtering conditions of the graph model, restaurants that meet the filtering conditions are scanned and filtered out, as shown in Tables 1 and 2. A total of 10 records that meet the filtering conditions are retrieved.

[0100] The `rid` values ​​from the 10 data entries that meet the filtering criteria are obtained. For each `rid`, the hash value is calculated twice using hash function seeds 84354565 and 76854767. The calculation results are shown in Table 6 below.

[0101] Table 6. Graph RID hash calculation results

[0102]

[0103] The bit index, byte index, and offset corresponding to 'rid' in the graph model query results obtained above (as shown in Table 6) are compared and verified with the Bloom filter bit array pushed down from the document model (as shown in Table 5). If the values ​​of the two hash values ​​of the graph model 'rid' at the corresponding positions in the bit array are both 1, then the conditions of the Bloom filter are met, and the data corresponding to this graph model 'rid' is added to the returned result set; otherwise, it is considered invalid data. The bit verification results are shown in Table 7 below:

[0104] Table 7 Bit verification results

[0105]

[0106] The query operator of the graph model returns a result set from the storage engine that may meet the matching requirements after being filtered by the graph model Bloom filter. This batch of returned results is then precisely joined (hash join) with the intermediate result set, and filtered again to select the data that perfectly matches. Data that fails to join is discarded, and data that successfully joins is added to a new intermediate result set. After all data has been joined, the intermediate result set obtained from the previous query operator is released.

[0107] Execute the query operator for the third data model, namely the query operator for the relational model, using the new intermediate result set;

[0108] All query operators for the relevant data models in the query request have been executed, and the result set will be returned as the final query result set. Only 4 restaurants in the final result set can satisfy the conditions of all data models, and their RIDs are shown in Table 8 below:

[0109] Table 8. Final query result set (rid)

[0110] .

Claims

1. A method for implementing cross-model query pushdown, characterized in that: Includes the following steps: Step 1: Based on the query statement that includes mixed queries on multiple data models, determine the order of the query operators corresponding to each data model; Step 2: Execute the corresponding query operators in sequence to obtain the final query result set; Specifically, when executing the query operator of the first data model in the computing engine, the first data model is scanned according to the query conditions of the first data model, the query result set that satisfies the query conditions is returned, and it is placed into the intermediate result set. For each remaining query operator, perform the query as follows: After obtaining the query result set of the current data model, assuming that the query result set of the current data model and the query result set of the next data model are connected according to a certain ID key, a Bloom filter is constructed based on the key value of the ID key; the constructed Bloom filter is then pushed down to the storage engine. After receiving a query request for the next data model, the storage engine uses a Bloom filter to filter the query results of the next data model, obtains the query results that meet the conditions of the Bloom filter, and adds the query results to the query result set of the next data model. After receiving the query result set of the next data model, the computing engine performs a hash join with each query result in the intermediate result set using the associated ID key. Query results that are successfully joined are added to the new intermediate result set, while query results that fail to join are discarded. Once all query results have been joined, the computing engine releases the query result set of the next data model it received. Using the new intermediate result set, execute the next data model sequentially; The intermediate result set after all query operators have been executed is the final result set.

2. The method for implementing cross-model query pushdown according to claim 1, characterized in that: To construct a Bloom filter, follow these steps: Calculate the optimal bit array size n and the number of hash functions for the Bloom filter; Choose the hash function seed and the hash function; Based on the optimal bit array size n, initialize a bit array of size n, with a value of 0 and a data type of bit; Based on the key value of the ID key in the query result set of the previous data model and the query result set of the current data model, obtain the bit array after inserting the ID key; The bit array after the inserted ID key, the hash function name, the number of hash functions, and the hash function seed are serialized and encoded in a unified manner to complete the construction of the Bloom filter.

3. The method for implementing cross-model query pushdown according to claim 2, characterized in that: The step of obtaining the bit array after inserting the ID key based on the key value of the ID key connecting the query result set of the previous data model and the query result set of the current data model includes: Using the selected hash function seed and hash function, the key value of the ID key that connects the query result sets of the two data models in the query result set of the previous data model is hashed. The number of hash calculations is determined by the number of hash functions, and the calculation method depends on the number of hash function seeds. Finally, k hash values ​​are obtained, where k represents the number of hash functions. Perform a modulo operation on each hash value to obtain the bit index; Perform division and modulo operations on the bit indices to obtain the byte index and bit offset; the byte index represents the index value in the bit array, and the bit offset represents the offset position in a byte. Set the values ​​corresponding to the byte index and bit offset to 1.

4. The method for implementing cross-model query pushdown according to claim 3, characterized in that: The step of using a Bloom filter to filter the query results of the next data model to obtain query results that meet the conditions of the Bloom filter includes: Retrieve the key value of the ID key that is responsible for connecting the query result set of the current data model to the query result set of the next data model; Based on the hash function, the number of hash functions, and the hash function seed, the ID key value is hashed k times to obtain k hash values, where k represents the number of hash functions; Calculate the positions of k hash values ​​in the bit array after inserting the ID key; The algorithm checks whether the bit position of the hash value in the bit array is 1. Only when all the bits corresponding to the k hash values ​​of an ID key are 1, it means that the conditions of the Bloom filter are met; otherwise, it means that the conditions of the Bloom filter are not met.

5. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the implementation method of cross-model query pushdown as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the implementation method of cross-model query pushdown as described in any one of claims 1-4.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the cross-model query pushdown method as described in any one of claims 1-4.

Images