Hash connection method, device, equipment and medium

Abstract

The present application discloses a hash join method, apparatus, device, and medium, relating to the field of hardware acceleration of database query operations, including: obtaining first data of a first data tuple to be joined and second data of a second data tuple to be joined, respectively calculating the first data and the second data using a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data; determining a first target hash table corresponding to the hash result of the first data and a second target hash table corresponding to the hash result of the second data; dividing the first data and the second data into a plurality of groups of first sub-data and second sub-data, respectively storing the first sub-data and the second sub-data in the first target hash table and the second target hash table; reading the first sub-data and the second sub-data from the first target hash table and the second target hash table, and performing a merge join on the first sub-data and the second sub-data that meet an equal value condition to obtain the joined data. The present application also discloses a hash join method, apparatus, device, and medium, relating to the field of hardware acceleration of database query operations, including: obtaining first data of a first data tuple to be joined and second data of a second data tuple to be joined, respectively using a cuckoo algorithm to calculate the first data and the second data to obtain a hash result of the first data and the second data; determining a first target hash table corresponding to the hash result of the first data and a second target hash table corresponding to the hash result of the second data; dividing the first data and the second data into a plurality of groups of first sub-data and second sub-data, respectively storing the first sub-data and the second sub-data in the first target hash table and the second target hash table; reading the first sub-data and the second sub-data from the first target hash table and the second target hash table, and performing a merge join on the first sub-data and the second sub-data that meet an equal value condition to obtain the joined data. The present application also discloses a hash join method, apparatus, and medium, relating to the field of hardware acceleration of database query operations, including obtaining first data of a first data tuple to be joined and second data to obtain a hash result of the first data and the second data to

Description

Technical Field

[0001] The present invention relates to the field of hardware acceleration of database query operations, and in particular to a hash connection method, device, equipment and medium. Background Art

[0002] The hash join algorithm is one of the most widely used join algorithms in database queries, achieving low-complexity and low-power join calculations. The hash join algorithm consists of two phases: construction and detection. After all data calculations are completed in the construction phase, the data calculations in the detection phase are performed.

[0003] Common FPGA (Field Programmable Gate Array)-based hash join acceleration methods primarily achieve superior acceleration by resolving hash conflicts. However, these methods are unable to achieve parallel processing of large amounts of data during the detection phase. Existing parallel data processing during the detection phase can only be achieved through partitioning hash joins, which is computationally complex and has poor parallelism and scalability, resulting in slow hash join speeds.

[0004] From the above, it can be seen that how to improve the speed of hash connection is a problem to be solved in this field. Summary of the Invention

[0005] In view of this, the present invention aims to provide a hash join method, apparatus, device, and medium that can improve the speed of hash join. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a hash connection method, comprising:

[0007] Obtaining first data of a first data tuple to be connected and second data of a second data tuple to be connected, and respectively calculating the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data;

[0008] Determine a first target hash table corresponding to a hash result of the first data and a second target hash table corresponding to a hash result of the second data;

[0009] Respectively dividing the first data and the second data into a plurality of groups of first sub-data and second sub-data, and storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively;

[0010] The first sub-data and the second sub-data are read from the first target hash table and the second target hash table, and the first sub-data and the second sub-data that meet the equal value condition are merged and connected to obtain connected data.

[0011] Optionally, obtaining the first data of the first data tuple to be connected and the second data of the second data tuple to be connected includes:

[0012] Reading a first data tuple to be connected and a second data tuple to be connected in an external memory to obtain first data of the first data tuple to be connected and second data of the second data tuple to be connected; wherein the external memory is any one of a double data rate synchronous dynamic random access memory and a synchronous dynamic random access memory.

[0013] Optionally, the using a first hash function and a second hash function in a cuckoo algorithm to calculate the first data and the second data respectively to obtain a hash result of the first data and a hash result of the second data includes:

[0014] Calculating the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm, respectively, to obtain a first hash result and a second hash result of the first data and a first hash result and a second hash result of the second data;

[0015] Accordingly, determining a first target hash table corresponding to the hash result of the first data and a second target hash table corresponding to the hash result of the second data includes:

[0016] Determine a first hash table address and a second hash table address corresponding to the first hash result and the second hash result of the first data, respectively, and a third hash table address and a fourth hash table address corresponding to the first hash result and the second hash result of the second data, respectively;

[0017] Determine a first target hash table address from the first hash table address and the second hash table address, and determine a second target hash table address from the third hash table address and the fourth hash table address;

[0018] Optionally, storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively includes:

[0019] The first sub-data is stored in a hash table corresponding to the first target hash table address, and the second sub-data is stored in a hash table corresponding to the second target hash table address.

[0020] Optionally, determining the first target hash table address from the first hash table address and the second hash table address, and determining the second target hash table address from the third hash table address and the fourth hash table address includes:

[0021] Determine whether there are free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address respectively; if so, determine the address of the hash table with the largest number of free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address respectively as the first target hash table address, and determine the address of the other hash table as the first target hash table backup address;

[0022] Determine whether there are free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address respectively, and if so, determine the hash table with the largest number of free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address respectively as the second target hash table address, and determine the address of another hash table as the second target hash table backup address;

[0023] Accordingly, storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively includes:

[0024] The first sub-data and the first target hash table spare address are stored in an idle hash bucket of the first target hash table, and the second sub-data and the second target hash table spare address are stored in an idle hash bucket of the second target hash table.

[0025] Optionally, after determining whether there are free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, the method further includes:

[0026] If not, searching for a spare address for storing data in a hash bucket in the hash table corresponding to the first hash table address and the second hash table address respectively, and determining whether there is a hash bucket that meets the preset conditions based on the spare address;

[0027] If it exists, the data stored in the hash bucket that meets the preset conditions is transferred to the hash bucket in the hash table corresponding to the backup address, so that the first sub-data is stored in the hash bucket that meets the preset conditions; if it does not exist, the first sub-data is stored in the linked list corresponding to the first target hash table.

[0028] Optionally, after storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively, the method further includes:

[0029] The total amount of data stored in the first target hash table and the second target hash table is recorded.

[0030] In a second aspect, the present application discloses a hash connection device, comprising:

[0031] a hash result acquisition module, configured to acquire first data of a first data tuple to be connected and second data of a second data tuple to be connected, and respectively calculate the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data;

[0032] a hash table determining module, configured to determine a first target hash table corresponding to a hash result of the first data and a second target hash table corresponding to a hash result of the second data;

[0033] a storage module, configured to divide the first data and the second data into a plurality of groups of first sub-data and second sub-data, and store the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively;

[0034] The merge join module is used to read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge join the first sub-data and the second sub-data that meet the equal value condition to obtain joined data.

[0035] In a third aspect, the present application discloses an electronic device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed hash connection method.

[0038] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed hash connection method are implemented.

[0039] It can be seen that the present application obtains the first data of the first data tuple to be connected and the second data of the second data tuple to be connected, and uses the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain the hash result of the first data and the hash result of the second data; determines the first target hash table corresponding to the hash result of the first data and the second target hash table corresponding to the hash result of the second data; divides the first data and the second data into several groups of first sub-data and second sub-data, respectively, and stores the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively; reads the first sub-data and the second sub-data from the first hash table and the second target hash table, and merges and connects the first sub-data and the second sub-data that meet the equal value condition to obtain the connected data. It can be seen that the present application implements hash calculation and storage of the first data of the first data tuple to be connected and the second data of the second data tuple to be connected during the construction phase, so that the first sub-data and the second sub-data that meet the equivalence condition can be merged and connected subsequently to obtain the connected data, eliminating the limitation that only one data tuple connection can be processed at a time during the detection phase, that is, parallel hash calculation, storage and merge connection can be realized, thereby improving the speed of hash connection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0041] Figure 1 This is a flow chart of a hash connection method disclosed in this application;

[0042] Figure 2 This is a flowchart of a specific hash connection method disclosed in this application;

[0043] Figure 3 A specific data storage diagram disclosed in this application;

[0044] Figure 4 This is a specific data merging and connection diagram disclosed in this application;

[0045] Figure 5 This is a schematic diagram of the structure of a hash connection device disclosed in this application;

[0046] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] The hash join algorithm is one of the most widely used join algorithms in database queries, achieving low-complexity and low-power join calculations. The hash join algorithm consists of two phases: construction and detection. After all data calculations are completed in the construction phase, the data calculations in the detection phase are performed.

[0049] Common FPGA-based hash join acceleration methods primarily achieve superior acceleration by resolving hash conflicts. However, these methods are unable to achieve parallel processing of large amounts of data during the detection phase. Existing parallel data processing during the detection phase can only be achieved by partitioning the hash join, which is computationally complex and has poor parallelism and scalability, resulting in slow hash join speeds.

[0050] To this end, this application provides a hash connection solution that can improve the speed of hash connection.

[0051] See also Figure 1 As shown, the embodiment of the present application discloses a hash connection method, including:

[0052] Step S11: Obtain the first data of the first data tuple to be connected and the second data of the second data tuple to be connected, and use the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain the hash result of the first data and the hash result of the second data.

[0053] In this embodiment, the obtaining of the first data of the first data tuple to be connected and the second data of the second data tuple to be connected includes: reading the first data tuple to be connected and the second data tuple to be connected from an external memory to obtain the first data of the first data tuple to be connected and the second data of the second data tuple to be connected; wherein the external memory is any one of a double data rate synchronous dynamic random access memory and a synchronous dynamic random access memory. It is understood that when the external memory is a DDR (Double Data Rate), the first data tuple to be connected and the second data tuple to be connected are read from the DDR to obtain the first data of the first data tuple to be connected and the second data of the second data tuple to be connected; when the external memory is an SDRAM (Synchronous Dynamic Random Access Memory), the first data tuple to be connected and the second data tuple to be connected are read from the SDRAM to obtain the first data of the first data tuple to be connected and the second data of the second data tuple to be connected.

[0054] It should be noted that, because the first data tuple to be connected and the second data tuple to be connected have a large amount of data, the first data tuple to be connected and the second data tuple to be connected are generally divided into several data units respectively, and the first data and the second data are one of the data units. For example, the first data tuple to be connected and the second data tuple to be connected can be divided into 10 data units respectively, and the various data units of the first data tuple to be connected and the second data tuple to be connected can be hashed, stored and merged in sequence.

[0055] Step S12: Determine a first target hash table corresponding to the hash result of the first data and a second target hash table corresponding to the hash result of the second data.

[0056] In this embodiment, the address of the first target hash table corresponding to the hash result of the first data is determined, that is, the corresponding first target hash table can be determined based on this. Similarly, the address of the second target hash table corresponding to the hash result of the second data can be determined, that is, the corresponding second target hash table can be determined based on this.

[0057] Step S13: Divide the first data and the second data into a plurality of groups of first sub-data and second sub-data, and store the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively.

[0058] It should be noted that the first data and the second data may respectively include several groups of sub-data. For example, if the first data includes 1000 sub-data, the 1000 sub-data are stored in the first target hash table in sequence.

[0059] It can be understood that, after storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively, the method further includes: recording the total amount of data stored in the first target hash table and the second target hash table.

[0060] Step S14: Read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge-connect the first sub-data and the second sub-data that meet the equal value condition to obtain connected data.

[0061] In this embodiment, performing a merge join is the detection phase of a hash join. Since the first data and the second data are respectively divided into several groups of sub-data in advance, the first sub-data of the first data and the second sub-data of the second data need to be merged and joined in sequence until all the sub-data are connected. It should be noted that before performing a merge join, it is necessary to determine whether the current first sub-data and the current second sub-data meet the equivalence condition. If so, a merge join can be performed. If not, the current first sub-data and the current second sub-data are skipped, and it is determined whether the next first sub-data and the next second sub-data meet the equivalence condition. The merge calculation of a single-row hash table data element is used for illustration:

[0062] 1) Detect the number of data elements in the same hash table row corresponding to the first sub-data and the second sub-data respectively. If the total amount of data in the hash table row corresponding to one sub-data is 0, skip the data merging calculation of that row and detect the next hash row data;

[0063] 2) When both hash rows have sub-data, read the data in the single-row hash bucket from the corresponding hash table. If a linked list exists, read all the data in the corresponding linked list. After reading all the data, cache it. Each hash bucket corresponds to a linked list storage module to improve the reading speed of the linked list.

[0064] 3) When the data in the first sub-data cache is greater than or equal to the second sub-data, the data is sequentially written into the M merge-join units based on the first sub-data cache. The data in the second sub-data cache is written into the merge-join units in a pipelined manner and sequentially transferred to the merge-join units M to complete the merge-join calculation.

[0065] 4) In the merge join unit, when two data elements exist at the same time, the two data elements are compared to see whether they meet the join condition. If they meet the join condition, the join result is output; otherwise, it is skipped; the above operation is repeated to complete the merge join of the data in all hash table rows.

[0066] It is understandable that after obtaining the connected data, it is necessary to transfer the connected data to a CPU (central processing unit) processing module outside the acceleration structure through DMA (Direct Memory Access).

[0067] It can be seen that the present application obtains the first data of the first data tuple to be connected and the second data of the second data tuple to be connected, and uses the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain the hash result of the first data and the hash result of the second data; determines the first target hash table corresponding to the hash result of the first data and the second target hash table corresponding to the hash result of the second data; divides the first data and the second data into several groups of first sub-data and second sub-data, respectively, and stores the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively; reads the first sub-data and the second sub-data from the first hash table and the second target hash table, and merges and connects the first sub-data and the second sub-data that meet the equal value condition to obtain the connected data. It can be seen that the present application implements hash calculation and storage of the first data of the first data tuple to be connected and the second data of the second data tuple to be connected during the construction phase, so that the first sub-data and the second sub-data that meet the equivalence condition can be merged and connected subsequently to obtain the connected data, eliminating the limitation that only one data tuple connection can be processed at a time during the detection phase, that is, parallel hash calculation, storage and merge connection can be realized, thereby improving the speed of hash connection.

[0068] See also Figure 2 As shown, the embodiment of the present application discloses a specific hash connection method, including:

[0069] Step S21: Acquire first data of a first data tuple to be connected and second data of a second data tuple to be connected.

[0070] Step S22: using the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain the first hash result and the second hash result of the first data and the first hash result and the second hash result of the second data.

[0071] Step S23: Determine a first hash table address and a second hash table address corresponding to the first hash result and the second hash result of the first data, respectively, and a third hash table address and a fourth hash table address corresponding to the first hash result and the second hash result of the second data, respectively.

[0072] Step S24: determining a first target hash table address from the first hash table address and the second hash table address, and determining a second target hash table address from the third hash table address and the fourth hash table address.

[0073] In this embodiment, determining the first target hash table address from the first hash table address and the second hash table address, and determining the second target hash table address from the third hash table address and the fourth hash table address, includes: determining whether there are free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, respectively; if so, determining the address of the hash table with the largest number of free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address as the first target hash table address, and determining the address of another hash table as the first target hash table backup address; determining whether there are free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address, respectively; if so, determining the hash table with the largest number of free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address as the second target hash table address, and determining the address of another hash table as the second target hash table backup address. Wherein, if the empty or full status item of the hash bucket is empty, the hash bucket is determined to be an idle hash bucket. It can be understood that the hash table contains N (N≥2) hash bucket slots, and the hash bucket contains flag signals such as data element, linked list address, empty and full status, and spare hash bucket address. Each slot stores data in the format of {data element; empty and full flag; linked list address; spare hash bucket address}.

[0074] In this embodiment, after determining whether there is an idle hash bucket in the hash table corresponding to the first hash table address and the second hash table address, the method further includes: if not, searching for a spare address for storing data in the hash bucket in the hash table corresponding to the first hash table address and the second hash table address, and determining whether there is a hash bucket that meets the preset conditions based on the spare address; if so, transferring the data stored in the hash bucket that meets the preset conditions to the hash bucket in the hash table corresponding to the spare address, so that the first sub-data is stored in the hash bucket that meets the preset conditions; if not, storing the first sub-data in the linked list corresponding to the first target hash table.

[0075] It can be understood that if there is no free hash bucket, it is detected whether the spare hash bucket is free. If there is an free spare hash bucket, the hash table with the most free spare hash buckets is screened out from the hash tables corresponding to the first hash table address and the second hash table address respectively, and it is determined as the first target hash table. The data in hash bucket 1 is then transferred to the spare hash bucket 1 so that when the sub-data of the first data is subsequently stored, the sub-data will be stored in the hash bucket 1.

[0076] Step S25: Divide the first data and the second data into several groups of first sub-data and second sub-data respectively, store the first sub-data in the hash table corresponding to the first target hash table address, and store the second sub-data in the hash table corresponding to the second target hash table address.

[0077] In this embodiment, the first sub-data and the first target hash table spare address are stored in a free hash bucket of the first target hash table, and the second sub-data and the second target hash table spare address are stored in a free hash bucket of the second target hash table.

[0078] Step S26: Read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge and connect the first sub-data and the second sub-data that meet the equal value condition to obtain connected data.

[0079] It can be seen that the present application calculates and stores two data tuples to be connected in parallel during the construction phase, so that the two sub-data can be connected in parallel in the subsequent detection phase, thereby improving the speed of hash connection; wherein, the two hash functions in the cuckoo algorithm are used for calculation, the calculation results of the first data and the second data increase, and the hash table that can be stored also increases accordingly, thereby increasing the probability of storing the first data and the second data.

[0080] The technical solution of this application is further described below:

[0081] 1) Reading a first data tuple to be connected and a second data tuple to be connected in an external memory to obtain first data of the first data tuple to be connected and second data of the second data tuple to be connected.

[0082] 2) Using the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively, to obtain hash results 11 and 12 of the first data and hash results 21 and 22 of the second data.

[0083] 3) Determine a first hash table address 11 corresponding to the hash result 11 and a second hash table address 12 corresponding to the hash result 12, and determine a first target hash table address from the first hash table address 11 and the second hash table address 12; the determination process is specifically as follows:

[0084] Determine the number of free hash buckets in hash table 11 and hash table 12 corresponding to the first hash table address 11 and the second hash table address 12, respectively, and determine whether there are free hash buckets in hash table 11 and hash table 12;

[0085] If it exists, the hash table with the largest number of free hash buckets is used as the first target hash table, and the address of the other hash table is used as the backup address of the first target hash table;

[0086] If not, determine the number of free standby hash buckets in hash table 11 and hash table 12. First, determine whether there are free standby hash buckets in hash table 11 and hash table 12. If so, use the hash table with the largest number of free standby hash buckets as the first target hash table. If not, determine the linked list address slots of the hash buckets, and determine the hash table with the largest number of empty linked list address slots as the first target hash table.

[0087] It is understandable that the process of determining the second target hash table is the same as the process of determining the first target hash table. It should be noted that if one of the hash tables is determined as the target hash table, the other hash table is determined as the backup hash table.

[0088] 4) The first data and the second data are respectively divided into a plurality of first sub-data and second sub-data groups, and the first sub-data and the second sub-data are respectively stored in a first target hash table and a second target hash table. Since the process of determining the target hash table is different, the storage process is also different. For example, taking the storage of the first sub-data as an example:

[0089] If there is an idle hash bucket in the determined first target hash table, the first subdata of the first data is stored in the data element item of the idle hash bucket 1, and the first target hash table spare address is stored in the spare hash bucket address item of hash bucket 1;

[0090] If there is no free hash bucket in the determined first target hash table, but there is an free spare hash bucket, then the data in the data element of hash bucket 1 is transferred to the spare hash bucket corresponding to hash bucket 1, and then the first subdata of the first data is stored in hash bucket 1;

[0091] If there is no free hash bucket or free spare hash bucket in the determined first target hash table, then for example Figure 3In a specific data storage schematic diagram shown, a first group of first sub-data of first data is stored in linked list 1-1, and the address of linked list 1-1 is stored in the linked list address item; a second group of first sub-data of the first data is stored in linked list 1-2, and the address of linked list 1-2 is stored in the linked list address item, until the first data is completely stored;

[0092] It is understandable that in the above process, there will be data writing and / or data transfer, so it is necessary to record the total amount of data elements written to the current hash table. Among them, the data format of the hash bucket can be adjusted, deleted, and added to a certain extent. When the spare address slot is not idle, the spare address of the spare slot can continue to be detected; multi-level spare slot detection is performed to improve the success rate of data element insertion.

[0093] 5) Read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge and connect the first sub-data and the second sub-data that meet the equal value condition to obtain the connected data; Figure 4 Taking a specific data merge and join diagram as an example, the first sub-data is written into the merge and join unit in the order of the data groups, and the first group of second sub-data is read. If the first group of first sub-data and the first group of second sub-data meet the equivalence condition, the first group of second sub-data is written into the merge and join unit 1 and merge and join is performed. This process is repeated until all data has been merged and joined, and all joined data is output to the join result cache module. According to the control instruction, the data result is input to the CPU processing unit. In this process, the merge calculation of multiple rows of hash table data elements is extended, and each hash table merge calculation module reads and calculates the data elements of a single row; after the calculation is completed, the next valid row is read and calculated. By extending the data element merge module, parallel merge calculation of data in multiple rows of hash tables can be achieved, fully utilizing the parallel computing capabilities of FPGA, and having strong scalability and portability.

[0094] See also Figure 5 As shown, the embodiment of the present application discloses a hash connection device, including:

[0095] a hash result acquisition module 11, configured to acquire first data of a first data tuple to be connected and second data of a second data tuple to be connected, and respectively calculate the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data;

[0096] a hash table determination module 12, configured to determine a first target hash table corresponding to a hash result of the first data and a second target hash table corresponding to a hash result of the second data;

[0097] a storage module 13, configured to divide the first data and the second data into a plurality of groups of first sub-data and second sub-data, and store the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively;

[0098] The merge join module 14 is configured to read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge join the first sub-data and the second sub-data that meet the equal value condition to obtain joined data.

[0099] It can be seen that the present application obtains the first data of the first data tuple to be connected and the second data of the second data tuple to be connected, and uses the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain the hash result of the first data and the hash result of the second data; determines the first target hash table corresponding to the hash result of the first data and the second target hash table corresponding to the hash result of the second data; divides the first data and the second data into several groups of first sub-data and second sub-data, respectively, and stores the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively; reads the first sub-data and the second sub-data from the first hash table and the second target hash table, and merges and connects the first sub-data and the second sub-data that meet the equal value condition to obtain the connected data. It can be seen that the present application implements hash calculation and storage of the first data of the first data tuple to be connected and the second data of the second data tuple to be connected during the construction phase, so that the first sub-data and the second sub-data that meet the equivalence condition can be merged and connected subsequently to obtain the connected data, eliminating the limitation that only one data tuple connection can be processed at a time during the detection phase, that is, parallel hash calculation, storage and merge connection can be realized, thereby improving the speed of hash connection.

[0100] Furthermore, an embodiment of the present application also provides an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0101] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, the device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the hash join method performed by the electronic device as disclosed in any of the aforementioned embodiments.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0103] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0104] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0105] Among them, the operating system 221 is used to manage and control the various hardware devices and computer programs 222 on the electronic device to enable the processor 21 to calculate and process the massive data 223 in the memory 22. It can be Windows, Unix, Linux, etc. In addition to including computer programs that can be used to complete the hash connection method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks. In addition to including data transmitted by external devices received by the electronic device, the data 223 can also include data collected by its own input and output interface 25.

[0106] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps performed in the hash connection process disclosed in any of the aforementioned embodiments are implemented.

[0107] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0108] The above is a detailed introduction to a hash connection method, device, equipment and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A hash connection method, characterized in that: include: Obtaining first data of a first data tuple to be connected and second data of a second data tuple to be connected, and respectively calculating the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data; Determine a first target hash table corresponding to the hash result of the first data and a second target hash table corresponding to the hash result of the second data; the hash table includes multiple hash bucket slots, and the hash bucket includes a data element, a linked list address, an empty or full status, and a spare hash bucket address; Respectively dividing the first data and the second data into a plurality of groups of first sub-data and second sub-data, and storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively; Reading the first sub-data and the second sub-data from the first target hash table and the second target hash table, and performing a merge-join on the first sub-data and the second sub-data that satisfy an equal value condition to obtain joined data; The determining of a first target hash table corresponding to a hash result of the first data and a second target hash table corresponding to a hash result of the second data includes: Determine a first hash table address and a second hash table address corresponding to the first hash result and the second hash result of the first data, respectively, and a third hash table address and a fourth hash table address corresponding to the first hash result and the second hash result of the second data, respectively; determine whether there are free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, respectively; if so, determine the address of the hash table with the largest number of free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, respectively, as the first target hash table address, and determine the address of another hash table as the first target hash table backup address; determine whether there are free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address, respectively,; if so, determine the hash table with the largest number of free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address, respectively, as the second target hash table address, and determine the address of the other hash table as the second target hash table backup address; The storing the first sub-data and the second sub-data into the first target hash table and the second target hash table respectively includes: The first sub-data and the first target hash table spare address are stored in an idle hash bucket of the first target hash table, and the second sub-data and the second target hash table spare address are stored in an idle hash bucket of the second target hash table.

2. The hash connection method according to claim 1, wherein: The acquiring of first data of the first data tuple to be connected and second data of the second data tuple to be connected comprises: Reading a first data tuple to be connected and a second data tuple to be connected in an external memory to obtain first data of the first data tuple to be connected and second data of the second data tuple to be connected; wherein the external memory is any one of a double data rate synchronous dynamic random access memory and a synchronous dynamic random access memory.

3. The hash connection method according to claim 1, wherein: The using the first hash function and the second hash function in the cuckoo algorithm to calculate the first data and the second data respectively to obtain a hash result of the first data and a hash result of the second data includes: The first data and the second data are calculated respectively using a first hash function and a second hash function in a cuckoo algorithm to obtain a first hash result and a second hash result of the first data and a first hash result and a second hash result of the second data.

4. The hash connection method according to claim 3, characterized in that: The storing the first sub-data and the second sub-data into the first target hash table and the second target hash table respectively includes: The first sub-data is stored in a hash table corresponding to the first target hash table address, and the second sub-data is stored in a hash table corresponding to the second target hash table address.

5. The hash connection method according to claim 1, wherein: After determining whether there is an idle hash bucket in the hash table corresponding to the first hash table address and the second hash table address respectively, the method further includes: If not, searching for a spare address for storing data in a hash bucket in the hash table corresponding to the first hash table address and the second hash table address respectively, and determining whether there is a hash bucket that meets the preset conditions based on the spare address; If it exists, the data stored in the hash bucket that meets the preset conditions is transferred to the hash bucket in the hash table corresponding to the backup address, so that the first sub-data is stored in the hash bucket that meets the preset conditions; if it does not exist, the first sub-data is stored in the linked list corresponding to the first target hash table.

6. The hash connection method according to any one of claims 1 to 5, characterized in that: After storing the first sub-data and the second sub-data in the first target hash table and the second target hash table respectively, the method further includes: The total amount of data stored in the first target hash table and the second target hash table is recorded.

7. A hash connection device, characterized in that: include: a hash result acquisition module, configured to acquire first data of a first data tuple to be connected and second data of a second data tuple to be connected, and respectively calculate the first data and the second data using a first hash function and a second hash function in a cuckoo algorithm to obtain a hash result of the first data and a hash result of the second data; a hash table determination module, configured to determine a first target hash table corresponding to a hash result of the first data and a second target hash table corresponding to a hash result of the second data; the hash table comprising a plurality of hash bucket slots, each hash bucket comprising a data element, a linked list address, an empty or full status, and a spare hash bucket address; a storage module, configured to divide the first data and the second data into a plurality of groups of first sub-data and second sub-data, and store the first sub-data and the second sub-data in the first target hash table and the second target hash table, respectively; a merge join module, configured to read the first sub-data and the second sub-data from the first target hash table and the second target hash table, and merge join the first sub-data and the second sub-data that satisfy an equal value condition to obtain joined data; The hash table determination module is specifically used to: Determine a first hash table address and a second hash table address corresponding to the first hash result and the second hash result of the first data, respectively, and a third hash table address and a fourth hash table address corresponding to the first hash result and the second hash result of the second data, respectively; determine whether there are free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, respectively; if so, determine the address of the hash table with the largest number of free hash buckets in the hash tables corresponding to the first hash table address and the second hash table address, respectively, as the first target hash table address, and determine the address of another hash table as the first target hash table backup address; determine whether there are free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address, respectively,; if so, determine the hash table with the largest number of free hash buckets in the hash tables corresponding to the third hash table address and the fourth hash table address, respectively, as the second target hash table address, and determine the address of the other hash table as the second target hash table backup address; The hash connection device is specifically used for: The first sub-data and the first target hash table spare address are stored in an idle hash bucket of the first target hash table, and the second sub-data and the second target hash table spare address are stored in an idle hash bucket of the second target hash table.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the hash connection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the hash connection method according to any one of claims 1 to 6 are implemented.

Images