A method for accessing hash tables in OpenGauss hash joins

Abstract

The present invention relates to an optimized access method and access system for a hash table in an OpenGauss hash connection. The method comprises the steps of constructing a hash table, completing the filling of a hash bucket array and a hash collision array; traversing the hash bucket array to detect whether a hash collision has occurred in the data stored at each location; and filling prompt information with the free space of the location information stored in the hash bucket array and the hash collision array based on the detection result. The method combines the advantages of the chain address method and the open addressing method, optimizes the implementation of the hash table from multiple perspectives, and distinguishes whether each conflicting data uses the chain address method or the open addressing method by filling in prompt information, thereby greatly improving the access performance of the hash table. In addition, the prompt information provided by the method can be used to determine whether a hash collision has occurred in the data without accessing the HashNext array, thereby improving the memory access and hash connection performance of the hash table.

Description

Technical Field

[0001] The present invention belongs to the technical field of hash table access methods, and in particular relates to an optimized access method and access system for a hash table in an OpenGauss hash connection. Background Art

[0002] In database systems, a hash join is a common method for connecting data from two tables. This method calculates the hash value of the data in the smaller table and constructs a hash table based on the hash value. The hash value of the data in the larger table is then calculated and used to detect the hash value in the hash table. The same data records are then found in the table for the join. As can be seen, the performance of a hash join depends largely on the access performance of the hash table. If the data corresponding to the hash value can be found in the hash table more quickly, the join operation can be completed more quickly. The access performance of a hash table depends on many factors, such as hash collision issues and the memory-friendliness of the data structure.

[0003] A hash collision occurs when two hash values ​​are modulo the size of the hash table. If the modulo results of the two hash values ​​are the same, a hash collision occurs. Methods for resolving hash collisions generally include open addressing, chain addressing (zipper method), rehashing, and establishing a common overflow area. Each of these methods has certain advantages and disadvantages. Developing more convenient and efficient hash collision resolution methods and hash table access methods is of great significance to improving the access performance of hash tables and the performance of hash joins. Summary of the Invention

[0004] In order to improve the access performance of hash tables and the performance of hash joins, the present invention proposes a new optimized access method for hash tables in OpenGauss hash joins.

[0005] The method of the present invention combines the advantages of the chain address method and the open addressing method by certain means, optimizes the implementation of the hash table from multiple angles, greatly improves the access performance of the hash table, and further improves the performance of the hash connection.

[0006] Explanation of terms

[0007] A hash table stores M data values. A hash value H is calculated for each data value and placed into the hash table. The hash table consists of the following three parts:

[0008] Raw data array DatumArray: an array structure of size M, storing raw data.

[0009] HashBucket: Calculates an array structure of size N based on the number of data M and stores the position of the original data in DatumArray.

[0010] Hash collision array HashNext: An array structure of size M, which stores the position of data in DatumArray that cannot be placed in HashBucket due to hash collision.

[0011] This invention optimizes the implementation of a hash table. The focus of the optimization is to utilize the free space of the location information stored in HashBucket and HashNext. Since the location information is represented by an integer, which can be of type int, the int type is generally represented by 32 bits in memory. However, representing the location information does not actually use 32 bits. For example, if the binary representation of position 4,4 is 100, only 3 bits are used. Therefore, we use this abundant space to create some hints to make hash table access more predictable, thereby improving access performance.

[0012] In the method of the present invention, the most significant 2 bits of the 32 bits in the int type are used as Hint hints, and the remaining 30 bits are used to represent location information. The amount of data that 30 bits can represent is 1,048,576, which is sufficient for most location information. If the amount of location information is too large, the long type can be used. The capacity of the long type can reach 64 bits. This article only discusses the implementation steps of the int type below.

[0013] The present invention optimizes the access performance of the hash table through the following improvements:

[0014] If data is stored in the HashBucket array and a hash conflict occurs with other data, the data is detected to see if the space at index + 1 of the HashBucket array is used. If it is used, a corresponding prompt message is added to indicate that the conflicting data is in HashNext. If it is not used, the data that should have been placed in the HashNext array is stored in the space at index + 1 of the HashBucket array, and a corresponding prompt message is added. Since continuous memory space is used, memory access performance is more friendly.

[0015] The amount of data that can be represented by 2 bits is 4, namely 00, 01, 10, and 11. The meanings of these 4 types of data are as follows:

[0016] 00 (or 0): indicates that the lower 30 bits are not used;

[0017] 01: Indicates that there is no hash conflict in the data stored in the lower 30 bits;

[0018] 11: Indicates that a hash conflict has occurred in the data stored in the lower 30 bits, and the conflicting data is stored in HashNext;

[0019] 10: Indicates that a hash conflict has occurred in the data stored in the lower 30 bits, and the conflicting data is in the space of the current HashBucket array index + 1.

[0020] Specifically, the present invention provides a method for accessing a hash table in an OpenGauss hash connection, the method comprising:

[0021] S1. Build a hash table and fill the hash bucket array and hash collision array;

[0022] S2. Traverse the hash bucket array starting from the index 0 of the hash bucket array to detect whether a hash conflict occurs in the data stored at that location;

[0023] S3 according to the detection results of step S2, the use of hash bucket array and hash collision array stored in the original data in the original data array position information free space, fill prompt information;

[0024] S4. Repeat steps S2 and S3 until the detection of the hash bucket array is completed.

[0025] Furthermore, according to some embodiments of the present invention, in step S1 of the method for accessing a hash table in an OpenGauss hash connection of the present invention, the hash bucket array and the hash collision array store position information of the original data in the original data array, and the position information is represented by an integer data of int type or long type.

[0026] Further, according to some embodiments of the present invention, the step S1 of the method for accessing a hash table in an OpenGauss hash connection of the present invention includes the following steps:

[0027] S11. Obtain M data from the data source and place them into the original data array;

[0028] S12. Calculate the value of N based on the value of M, where N is the next power of 2 starting from M, for example: 4, 8, 16, 32, 64, ..., 1024, 2048, 4096, ...;

[0029] S13. Traverse the original data array and calculate the hash value H of each data. The specific method of calculating the hash value is not limited;

[0030] S14. Modulo N based on the hash value H of each data, calculate its subscript in the hash bucket array based on the modulo result, and store the position of the new data in the original data array in the hash bucket array;

[0031] S15. Repeat steps S13 and S14 until all data are stored in the hash bucket array and the hash collision array.

[0032] Furthermore, in the above-mentioned method for accessing the hash table in the OpenGauss hash connection, the hash value H of each data is calculated in step S13. When the data is integer data, the hash value H is calculated by using the value of the data itself.

[0033] Furthermore, in step S14 of the method for accessing the hash table in the above-mentioned OpenGauss hash connection, the position of the data in the original data array is stored in the hash bucket array. The storage position may be in the following two situations:

[0034] (1) When the storage location space corresponding to the subscript in the hash bucket array is not used, the position of the new data in the original data array can be directly stored in the storage location;

[0035] (2) When the storage location space corresponding to the index in the hash bucket array has been used, the data in it is first taken out and stored in the hash collision array at the corresponding position according to the position information in the data, and then the position of the new data in the original data array is stored in the storage location.

[0036] Further, according to some embodiments of the present invention, the method for accessing a hash table in an OpenGauss hash connection of the present invention, when the position information of the original data stored in the hash bucket array and the hash collision array in the original data array is int type integer data, the step S3, according to the detection result of step S2, using the free space of the position information of the original data stored in the hash bucket array and the hash collision array in the original data array to fill the prompt information, includes:

[0037] (1) If there is no hash conflict between the data, the original position information is filled in the lower 30 bits of the int type, and the prompt information 01 is filled in the highest 2 bits;

[0038] (2) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is used, the original position information is filled in the lower 30 bits of the int type and the prompt information 11 is filled in the upper 2 bits;

[0039] (3) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is not used, the original position information is filled in the lower 30 bits of the int type, and the prompt information 10 is filled in the upper 2 bits. The position information that should be stored in the hash conflict array is stored in the position of the current index + 1 in the hash bucket array, and the position information is filled in the lower 30 bits of the int type, and the prompt information 10 is filled in the upper 2 bits. Repeat this step until all hash conflict data are processed or the space at the current index + 1 is used, and then stop and process according to step (2).

[0040] (4) The space of the current hash bucket array index is not used and no processing is performed (because after the highest 2 bits are filled with 00, the value of this data is also 0, so no additional processing is required).

[0041] Furthermore, the meaning of the prompt information in the above-mentioned method for accessing the hash table in the OpenGauss hash connection is as follows:

[0042] 0 (or 00): indicates that the lower 30 bits of the int type of the current subscript space in the hash bucket array are not used;

[0043] 01: Indicates that there is no hash conflict in the lower 30 bits of the int type data in the current index space in the hash bucket array;

[0044] 10: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the space of the current index + 1 in the hash bucket array.

[0045] 11: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the hash conflict array.

[0046] In addition, the present invention also provides a hash table access system in an OpenGauss hash connection, the system comprising:

[0047] Hash table construction module: used to build the hash table and complete the filling of the hash bucket array and hash collision array;

[0048] Hash collision detection module: used to detect whether hash collision occurs in the data stored in the hash bucket array;

[0049] Prompt information filling module: used to fill prompt information into the empty space of the position information stored in the hash bucket array and the hash conflict array;

[0050] Information recognition module: used to identify prompt information and make corresponding processing decisions.

[0051] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method for accessing a hash table in an OpenGauss hash connection when the program is executed by a processor.

[0052] Since the implementation of a hash table depends on its actual needs, although the processing ideas are the same, the implementation processes are different. The description of the present invention only focuses on the implementation of one hash table and optimizes it based on the implementation of this hash table. The implementation and optimization methods of other hash tables can refer to the processing ideas of the present invention and be obtained by appropriately adjusting and changing the method of the present invention in combination with the actual needs of the hash table implementation.

[0053] In summary, the hash table access method in the OpenGauss hash connection of the present invention has the following advantages:

[0054] (1) The method of the present invention combines the advantages of the chain address method and the open address method, and uses 2 bits of prompt information to distinguish whether each conflicting data uses the chain address method or the open address method, thereby improving the memory access performance of the data using the open address method.

[0055] (2) The method of the present invention also provides prompt information on whether a hash conflict occurs in the data, and a judgment can be made without accessing the HashNext array, which greatly improves the memory access performance of the hash table. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 4 is an implementation flow chart of the method of the present invention.

[0058] Figure 2 This is a flowchart of the hash table construction method of the present invention.

[0059] Figure 3 This is an example diagram of inserting data into a hash table in the method of the present invention. The figure shows that 14 data (elements) are inserted into a hash table with N being 16, where HashBucket and HashNext store the positions of the data in DatumArray.

[0060] Figure 4This is a schematic diagram of the functional division of the 32-bit int type in the method of the present invention. The int type shown in the figure is 32 bits, 2 bits of which are used to represent prompt information and 30 bits to represent location information.

[0061] Figure 5 This is an example diagram of the storage rules for hash collision data in the method of the present invention. As shown in the figure, the data 18 and 2 are calculated modulo the hash table with N being 16, and the calculated subscripts are both 2. Then, 2, which should be stored in HashNext, can be dumped to the space with subscript 3 in HashBucket because the space with subscript 3 (2+1) in HashBucket is unused, thereby obtaining better memory access performance.

[0062] Figure 6 It is a structural diagram of the system of the present invention. DETAILED DESCRIPTION

[0063] To make the objectives, technical solutions, and advantages of the present invention more clearly apparent, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention may also be implemented or applied through different specific implementation methods, and the details in this specification may be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0064] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention.

[0065] Example: A method for accessing a hash table in an OpenGauss hash connection (eg Figure 1 The method comprises the following steps:

[0066] (1) Build a hash table and complete the filling of the hash bucket array and the hash conflict array (such as Figure 2 As shown), the hash bucket array and the hash collision array store the position information of the original data in the original data array. The position information is represented by an integer data of int type.

[0067] 1. Get M data from the data source and put them into DatumArray.

[0068] 2. Calculate the N value based on the M value, where N is the next power of 2 starting from M, for example: 4, 8, 16, 32, 64, ..., 1024, 2048, 4096, ...

[0069] 3. Traverse DatumArray and calculate the hash value H of each data. The specific method of calculating the hash value is not limited.

[0070] 4. Take the modulus of N based on the hash value H of each data, calculate its subscript in the HashBucket based on the modulus result, and store the position of the new data in the DatumArray in the HashBucket. The storage location may be in the following two situations:

[0071] (1) When the storage location space corresponding to the subscript in HashBucket is not used, the position of the new data in DatumArray can be directly stored in the storage location;

[0072] (2) When the storage location space corresponding to the index in HashBucket is already in use, the data in it is first taken out and stored in the HashNext of the corresponding position according to the position information in the data, and then the position of the new data in DatumArray is stored in the storage location.

[0073] 5. Repeat steps 3 and 4 until all data is stored in HashBucket and HashNext (such as Figure 3 shown).

[0074] In this embodiment, the data is integer data, and the method for calculating the hash value is to use the value of the data itself, that is, the hash value H is the data value itself.

[0075] (2) Optimizing Hash Table Access

[0076] In the method of the present invention, the most significant 2 bits of the 32 bits in the int type are used as Hint prompts, and the remaining 30 bits are used to represent location information (such as Figure 4 As shown in the figure, the amount of data that can be represented by 30 bits is 1,048,576, which is sufficient for most location information. If the amount of location information is too large, the long type can be used. The capacity of the long type can reach 64 bits. This article only discusses the implementation steps of the int type.

[0077] If data is stored in the HashBucket array and there is a hash conflict between the data and other data, the data is detected to see if the space at the HashBucket array index + 1 is used. If it is used, a corresponding prompt message will be added to indicate that the conflicting data is in HashNext. If it is not used, the data that should have been put into the HashNext array will be stored in the space at the HashBucket array index + 1 (e.g. Figure 5As shown in the figure), and add corresponding prompt information. Since continuous memory space is used, the memory access performance is more friendly.

[0078] The amount of data that can be represented by 2 bits is 4, namely 00, 01, 10, and 11. The meanings of these 4 types of data are as follows:

[0079] 0 (or 00): indicates that the lower 30 bits of the int type of the current subscript space in the hash bucket array are not used;

[0080] 01: Indicates that there is no hash conflict in the lower 30 bits of the int type data in the current index space in the hash bucket array;

[0081] 10: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the space of the current index + 1 in the hash bucket array.

[0082] 11: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the hash conflict array.

[0083] 1. Traverse the hash bucket array starting from the index 0 of the hash bucket array to detect whether a hash conflict occurs in the data stored at that position.

[0084] 2. Based on the detection results of the previous step, use the free space of the original data position information stored in the hash bucket array and the hash collision array in the original data array to fill in the prompt information;

[0085] The detection results in the previous step may have the following four situations:

[0086] (1) If there is no hash conflict between the data, the original position information is filled in the lower 30 bits of the int type, and the prompt information 01 is filled in the highest 2 bits;

[0087] (2) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is used, the original position information is filled in the lower 30 bits of the int type and the prompt information 11 is filled in the upper 2 bits;

[0088] (3) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is not used, the original position information is filled in the lower 30 bits of the int type, and the prompt information 10 is filled in the upper 2 bits. The position information that should be stored in the hash conflict array is stored in the position of the current index + 1 in the hash bucket array, and the position information is filled in the lower 30 bits of the int type, and the prompt information 10 is filled in the upper 2 bits. Repeat this step until all hash conflict data are processed or the space at the current index + 1 is used, and then stop and process according to step (2).

[0089] (4) The space of the current hash bucket array index is not used and no processing is performed (because after the highest 2 bits are filled with 00, the value of this data is also 0, so no additional processing is required).

[0090] 3. Repeat steps 1 and 2 until the hash bucket array is completely detected.

[0091] A hash table access system in OpenGauss hash join (such as Figure 6 As shown), this system includes:

[0092] Hash table construction module: used to build the hash table and complete the filling of the hash bucket array and hash collision array;

[0093] Hash collision detection module: used to detect whether hash collision occurs in the data stored in the hash bucket array;

[0094] Prompt information filling module: used to fill prompt information into the empty space of the position information stored in the hash bucket array and the hash conflict array;

[0095] Information recognition module: used to identify prompt information and make corresponding processing decisions.

[0096] Each module operates according to the above-mentioned hash table access method.

[0097] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, substitutions, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for accessing a hash table in an OpenGauss hash connection, characterized in that: The method comprises: S1. Build a hash table and fill the hash bucket array and hash collision array. S2. Traverse the hash bucket array starting at index 0 and detect whether a hash collision occurs in the data stored at that index. S3 according to the detection results of step S2, the use of hash bucket array and hash collision array stored in the original data in the original data array position information free space, fill prompt information; When the position information of the original data stored in the hash bucket array and the hash collision array in the original data array is int type integer data, the step S3 fills the prompt information with the free space of the position information of the original data stored in the hash bucket array and the hash collision array in the original data array according to the detection result of step S2, including: (1) If there is no hash conflict between the data, the original position information is filled in the lower 30 bits of the int type, and the prompt information 01 is filled in the highest 2 bits; (2) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is used, the original position information is filled in the lower 30 bits of the int type, the prompt information 11 is filled in the highest 2 bits, and the position information is stored in the hash conflict array; (3) If a hash conflict occurs between data and the space at the current index + 1 in the hash bucket array is not used, the original position information is filled in the lower 30 bits of the int type, the prompt information 10 is filled in the highest 2 bits, and the position information that should be stored in the hash conflict array is stored in the current index + 1 position of the hash bucket array; (4) The space of the current hash bucket array index is not used and no processing is done; S4. Repeat steps S2 and S3 until the hash bucket array is completely probed.

2. The method for accessing a hash table in an OpenGauss hash connection according to claim 1, wherein: In step S1, the hash bucket array and the hash collision array store the position information of the original data in the original data array. The position information is represented by an integer data of type int or long.

3. The method for accessing a hash table in an OpenGauss hash connection according to claim 1, wherein: The step S1 of constructing a hash table includes: S11. Obtain M data from the data source and place them into the original data array; S12. Calculate the value of N based on the value of M, where N is the next power of 2 starting from M. S13. Traverse the original data array and calculate the hash value H of each data; S14. Modulo N based on the hash value H of each data item, calculate its subscript in the hash bucket array based on the modulo result, and store the position of the data item in the original data array in the hash bucket array; S15. Repeat steps S13 and S14 until all data are stored in the hash bucket array and the hash collision array.

4. The method for accessing a hash table in an OpenGauss hash connection according to claim 3, wherein: The hash value H of each data is calculated in S13. When the data is integer data, the hash value H is calculated by using the value of the data itself.

5. The method for accessing a hash table in an OpenGauss hash connection according to claim 3, wherein: The step S14 storing the position of the data in the original data array in the hash bucket array includes: (1) When the storage location space corresponding to the index in the hash bucket array is not used, the position of the new data in the original data array is directly stored in the storage location; (2) When the storage location space corresponding to the index in the hash bucket array has been used, the data in it is first taken out and stored in the hash collision array at the corresponding position according to the position information in the data. Then, the position of the new data in the original data array is stored in the storage location.

6. The method for accessing a hash table in an OpenGauss hash connection according to claim 1, wherein: The meaning of the prompt information is as follows: 0: Indicates that the lower 30 bits of the int type of the current subscript space in the hash bucket array are not used; 01: Indicates that there is no hash conflict in the lower 30 bits of the int type data in the current index space in the hash bucket array; 10: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the space of the current index + 1 in the hash bucket array. 11: Indicates that a hash conflict has occurred in the lower 30 bits of the int type data in the current index space in the hash bucket array. The conflicting data is stored in the hash conflict array.

7. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the steps of the method for accessing a hash table in an OpenGauss hash connection according to any one of claims 1 to 6.

Images