Hash join processing method, system and device, and storage medium and program product

Abstract

Provided in the embodiments of the present disclosure are a hash join processing method, system and device, and a storage medium and a program product. The method comprises: scanning a plurality of rows of data in a first data table within a target scanning range corresponding to a target worker thread, and on the basis of a first hash join mode, writing the plurality of rows of data into a first hash table in a memory; if it is determined that an available memory capacity corresponding to a hash join task is lower than a set threshold, sending the scanned plurality of rows of data to a data partitioning management thread, and switching to use a second hash join mode; and receiving first target row data distributed by the data partitioning management thread, and on the basis of the second hash join mode, writing the first target row data into a second hash table corresponding to the target worker thread in the memory, so as to perform persistence processing of the data in the second hash table. The present disclosure can flexibly switch between different hash join modes on the basis of the memory occupancy of a data table for hash table creation.

Description

Hash join processing method, system, device, storage medium and program product

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411021868.X, filed on July 26, 2024 with the Chinese Patent Office, entitled "Hash join processing method, system, device, storage medium and program product", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of database, and particularly relates to a hash join processing method, system, device, storage medium and program product. BACKGROUND

[0003] In a database system, hash join of two different tables according to target column data is a common operation. For example, table 1 is a table for storing the mathematics scores of each student, including student name, class, student ID, and mathematics score column data, and table 2 stores the student ID, mathematics score, and school column data of these students. For table 1 and table 2, the target column data set as the hash join key is assumed to be the mathematics score, so that the data in the two tables can be connected based on the mathematics score column data to obtain a new table representing the connection result.

[0004] A commonly used hash join method at present is a pure memory type hash join method, including but not limited to a parallel hash join (Parallel Hash Join) method and the like. The pure memory type hash join method refers to that the establishment and storage of the hash table are only performed in the memory. Due to the high speed performance of the memory, the execution speed is very fast, but it is limited by the capacity of the computer memory, and can only be used in the scene of hash join of tables with small data volume. SUMMARY

[0005] The present disclosure provides a hash join processing method, system, device, storage medium and program product, which can flexibly switch different hash join methods according to the memory occupation of the data table for hash table establishment.

[0006] In a first aspect, the present disclosure provides a hash join processing method applied to any target worker thread in a plurality of worker threads performing a hash join task in parallel, the hash join task being used to connect a first data table and a second data table with target column data in the first data table and the second data table as a connection key, and the method comprises:

[0007] scan multiple rows of data in a target scanning range corresponding to the target worker thread in the first data table, and write the multiple rows of data into a first hash table in memory based on a first hash join manner, the first hash join manner being a pure memory type hash join manner;

[0008] if it is determined that available memory capacity corresponding to the hash join task is lower than a set threshold, send the scanned multiple rows of data to a data partition management thread, and switch to using a second hash join manner, the second hash join manner being a hash join manner supporting data disk landing;

[0009] receive first target row data distributed by the data partition management thread, and write the first target row data into a second hash table corresponding to the target worker thread in memory based on the second hash join manner, and perform disk landing processing of data in the second hash table, wherein the first target row data includes each row of data determined by the data partition management thread after partitioning processing of the multiple rows of data to need to be distributed to the target worker thread.

[0010] In a second aspect, the embodiments of the present disclosure provide a hash join processing apparatus, applied to any target worker thread in a plurality of worker threads performing a hash join task in parallel, the hash join task being used to connect a first data table and a second data table with target column data in the first data table and the second data table as a connection key, and the apparatus comprising:

[0011] a writing module, configured to scan multiple rows of data in a target scanning range corresponding to the target worker thread in the first data table, and write the multiple rows of data into a first hash table in memory based on a first hash join manner, the first hash join manner being a pure memory type hash join manner;

[0012] a sending module, configured to, if it is determined that available memory capacity corresponding to the hash join task is lower than a set threshold, send the scanned multiple rows of data to a data partition management thread, and switch to using a second hash join manner, the second hash join manner being a hash join manner supporting data disk landing;

[0013] a receiving module, configured to receive first target row data distributed by the data partition management thread, and write the first target row data into a second hash table corresponding to the target worker thread in memory based on the second hash join manner, and perform disk landing processing of data in the second hash table, wherein the first target row data includes each row of data determined by the data partition management thread after partitioning processing of the multiple rows of data to need to be distributed to the target worker thread.

[0014] In a third aspect, the embodiments of the present disclosure further provide a hash join processing method applied to a data partition management thread, the method comprising:

[0015] receiving a plurality of rows of data sent by a target worker thread, the plurality of rows of data being sent by the target worker thread when it is determined that an available memory capacity corresponding to a hash join task is lower than a set threshold, the target worker thread being any one of a plurality of worker threads that execute the hash join task in parallel, the hash join task being used to join a first data table and a second data table with target column data in the first data table and the second data table as a join key, wherein the plurality of rows of data are scanned by the target worker thread within a target scan range in the first data table corresponding to the target worker thread, and the plurality of rows of data are written into a first hash table in memory based on a first hash join mode, the first hash join mode being a pure memory hash join mode;

[0016] performing partition processing on the plurality of rows of data to determine target rows of data that need to be distributed to the target worker thread;

[0017] sending the target rows of data to the target worker thread, so that the target worker thread writes the target rows of data into a second hash table in memory corresponding to the target worker thread based on a second hash join mode, and performs disk landing processing on data in the second hash table, the second hash join mode being a hash join mode that supports data disk landing.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a hash join processing apparatus applied to a data partition management thread, the apparatus comprising:

[0019] a receiving module configured to receive a plurality of rows of data sent by a target worker thread, the plurality of rows of data being sent by the target worker thread when it is determined that an available memory capacity corresponding to a hash join task is lower than a set threshold, the target worker thread being any one of a plurality of worker threads that execute the hash join task in parallel, the hash join task being used to join a first data table and a second data table with target column data in the first data table and the second data table as a join key, wherein the plurality of rows of data are scanned by the target worker thread within a target scan range in the first data table corresponding to the target worker thread, and the plurality of rows of data are written into a first hash table in memory based on a first hash join mode, the first hash join mode being a pure memory hash join mode;

[0020] a processing module configured to perform partition processing on the plurality of rows of data to determine target rows of data that need to be distributed to the target worker thread;

[0021] The sending module is configured to send the target row data to the target worker thread, so that the target worker thread writes the target row data into a second hash table corresponding to the target worker thread in the memory based on a second hash join mode, and performs disk writing of data in the second hash table, the second hash join mode being a hash join mode supporting data disk writing.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a database system, the system comprising:

[0023] a plurality of worker threads for performing a hash join task in parallel, and a data partition management thread, wherein the hash join task is used to join a first data table and a second data table based on target column data in the first data table and the second data table as a join key;

[0024] Any target worker thread of the plurality of worker threads is configured to scan a plurality of rows of data within a target scanning range corresponding to the target worker thread in the first data table, and write the plurality of rows of data into a first hash table in the memory based on a first hash join mode, and if it is determined that an available memory capacity corresponding to the hash join task is lower than a set threshold, send the scanned plurality of rows of data to the data partition management thread, and switch to a second hash join mode, receive target row data distributed by the data partition management thread, and write the target row data into a second hash table corresponding to the target worker thread in the memory based on the second hash join mode, and perform disk writing of data in the second hash table, wherein the first hash join mode is a pure memory hash join mode, and the second hash join mode is a hash join mode supporting data disk writing;

[0025] The data partition management thread is configured to perform partition processing on the plurality of rows of data to determine target row data that needs to be distributed to the target worker thread, and send the target row data to the target worker thread.

[0026] In a sixth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a communication interface, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the hash join processing method according to the first aspect or the third aspect.

[0027] In a seventh aspect, an embodiment of the present disclosure provides a non-transitory machine readable storage medium, the non-transitory machine readable storage medium storing executable code, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the hash join processing method according to the first aspect or the third aspect.

[0028] In an eighth aspect, the embodiments of the present disclosure provide a computer program product, which contains a computer program. When the computer program is executed by a processor of an electronic device, the processor can implement the hash join processing method of the first aspect or the third aspect.

[0029] In the hash join processing method provided by the embodiments of the present disclosure, the data in the first data table is extracted by scanning multiple rows of data in the target scanning range corresponding to the target worker thread in the first data table. Then, because the pure memory type hash join method has a fast task execution speed, when the available memory capacity is sufficient, the multiple rows of data scanned in the first data table are written into the first hash table in the memory based on the pure memory type first hash join method, so as to quickly execute the hash join task. In this process, if it is determined that the available memory capacity corresponding to the hash join task is lower than a set threshold, it indicates that the available memory capacity is insufficient. At this time, the scanned multiple rows of data are sent to the data partition management thread, and the hash join method is switched from the pure memory type hash join method to the second hash join method supporting data disk landing. Then, after receiving the first target row data distributed by the data partition management thread, the first target row data is written into the second hash table corresponding to the target worker thread in the memory based on the second hash join method. Then, the data in the second hash table is subjected to disk landing processing, so that part of the data is stored in the memory and the other part of the data is stored in the disk. Through the above scheme, when the memory is sufficient, the pure memory type hash join method is used to create the hash table, which can ensure the high-speed execution of the hash join task. When the memory is insufficient, the hash join method supporting data disk landing is switched to, which ensures that the creation of the hash table will not cause the memory overflow problem, so that the hash join task can be smoothly performed. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0031] FIG. 1 is a structural schematic diagram of a database system provided by the embodiments of the present disclosure;

[0032] FIG. 2 is a flowchart of a hash join processing method provided by the embodiments of the present disclosure;

[0033] FIG. 3 is a schematic diagram of a hash table provided by the embodiments of the present disclosure;

[0034] FIG. 4 is a flowchart of a hash join processing method provided by the embodiments of the present disclosure;

[0035] FIG. 5 is a flowchart of another hash connection processing method according to an embodiment of the present disclosure;

[0036] FIG. 6 is an application diagram of a hash connection processing method according to an embodiment of the present disclosure;

[0037] FIG. 7 is a structural diagram of a hash connection processing device according to an embodiment of the present disclosure;

[0038] FIG. 8 is a structural diagram of another hash connection processing device according to an embodiment of the present disclosure;

[0039] FIG. 9 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0041] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0042] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the case of no conflict between the embodiments, the embodiments described below and the features in the embodiments can be combined with each other. In addition, the sequence of steps in each method embodiment described below is only an example, not a strict limitation.

[0043] First, the terms or concepts involved in the embodiments of the present disclosure are explained and described:

[0044] Memory: a storage device in a computer system, used to store data and instructions being processed by the central processing unit (CPU). Compared with a disk, the memory has a faster access speed but a smaller storage capacity. In a database system, the memory is often used to cache data, indexes, and query results to reduce the number of accesses to the disk and improve performance.

[0045] Disk: A non-volatile storage device in a computer system, used for long-term storage of data and programs. Compared with memory, disk has slower access speed but larger storage capacity. In a database system, disk is usually used to store database files, including tables, indexes, views, stored procedures, and triggers, etc.

[0046] Parallelism: In a database system, in order to fully utilize the multi-threaded characteristics of CPU, a task will be divided into N sub-tasks, and these N sub-tasks correspond to N worker threads that execute the task in parallel. N is called the parallelism of the task.

[0047] Parallel Hash Join: A pure in-memory hash join method in a database system, which allows multiple threads to execute different parts of the hash join simultaneously, i.e. scanning data in different scan ranges of the data table used to build the hash table. Moreover, by executing the hash join algorithm in parallel, Parallel Hash Join can utilize multiple processor cores to process data simultaneously, resulting in fast execution speed.

[0048] Hybrid Hash Join: A hash join method in a database system that supports data disk overflow, which can overflow part of the data to disk when memory is insufficient to avoid memory overflow. However, when memory is sufficient, its execution speed is much worse than that of Parallel Hash Join.

[0049] Currently, when performing a hash join task, a pure in-memory hash join method is generally used. This pure in-memory hash join method means that the establishment and storage of the hash table are only performed in memory. Due to the natural high-speed performance of memory, the execution speed is very fast, but it is limited by the capacity of computer memory and can only be used in scenarios where small data tables are used for hash join.

[0050] In view of this, the embodiment of the present disclosure provides the following idea to solve the above-mentioned problem: when the memory is sufficient, first obtain the multiple rows of data corresponding to the target worker thread in the first data table used to establish the hash table, and write the multiple rows of data into the first hash table in the memory based on the pure memory type hash join mode, so as to ensure the task execution speed. In this process, if it is determined that the available memory capacity corresponding to the hash join task is lower than the set threshold, it means that the available capacity of the memory is insufficient, and the data cannot be written continuously. At this time, the multiple rows of data are sent to the data partition management thread, and the hash join mode is switched from the pure memory type hash join mode to the hash join mode supporting data landing. The data partition management thread performs partition processing on the multiple rows of data to determine the rows of data that need to be distributed to the target worker thread, and writes each row of data into the second hash table corresponding to the target worker thread in the memory in the hash join mode supporting data landing, and then performs landing processing on the data in the second hash table.

[0051] It can be seen that the embodiment of the present disclosure can flexibly switch different hash join modes according to the occupation amount of the first data table to the memory. Specifically, since the pure memory type hash join mode has fast task execution speed, when the occupation amount of the first data table to the memory is not large, that is, the available capacity of the memory is sufficient, the first hash table corresponding to multiple worker threads is created in the memory in priority, and data is written into the first hash table to ensure the execution speed. However, if it is found that the occupation amount of the first data table to the memory is too large, resulting in insufficient available capacity of the memory, the hash join mode is switched: from the pure memory type hash join mode to the hash join mode supporting data landing. Under the hash join mode supporting data landing, part of the data is stored in the memory, and another part of the data is transferred and stored in the disk, so as to avoid memory overflow.

[0052] The hash join processing scheme provided by the embodiment of the present disclosure will be introduced and described below.

[0053] FIG. 1 is a structural schematic diagram of a database system provided by an embodiment of the present disclosure. As shown in FIG. 1, the system includes multiple worker threads (the multiple worker threads include worker thread 1, worker thread 2, worker thread 3,..., and worker thread n in FIG. 1) performing hash join tasks in parallel, and a data partition management thread. The hash join task is used to connect a first data table and a second data table by taking target column data in the first data table and the second data table as a connection key.

[0054] It should be noted that the database system can be deployed in a cloud server or a cloud server cluster, and multiple CPU cores can be allocated in the cloud server or the cloud server cluster for processing tasks related to the database system. The above-mentioned multiple worker threads and the data partition management thread are threads running under these CPU cores.

[0055] The hash join task is mainly divided into two phases: a build phase and a probe phase. The solution provided by the embodiments of the present disclosure mainly focuses on the build phase, i.e., the phase of establishing a hash table. In the case of hash join of the first data table and the second data table, it is assumed in the embodiments of the present disclosure that the size of the first data table is smaller than that of the second data table, and thus the establishment of the hash table is based on the first data table. In addition, the plurality of work threads performing in parallel correspond to different scanning ranges of the first data table in the hash table establishment phase, for example, the scanning range of the work thread 1 is the first 500 rows of data of the first data table, the scanning range of the work thread 2 is the 501st-1000th row of data of the first data table, and so on.

[0056] For ease of understanding, the working process of the plurality of work threads is described by taking any target work thread in the plurality of work threads as an example (assuming that the target work thread is the work thread 1 in FIG. 1). The target work thread is configured to scan a plurality of rows of data in a target scanning range corresponding to the target work thread in the first data table, and write the plurality of rows of data into the first hash table in the memory based on the first hash join mode. In addition, if it is determined that the available memory capacity corresponding to the hash join task is lower than a set threshold, the plurality of rows of scanned data are sent to the data partition management thread, and the target work thread is switched to the second hash join mode to receive target row data distributed by the data partition management thread, and write the target row data into the second hash table corresponding to the target work thread in the memory based on the second hash join mode, and then perform the landing processing of the data in the second hash table. The first hash join mode is a pure memory hash join mode, and the second hash join mode is a hash join mode supporting data landing.

[0057] The data partition management thread is configured to perform partition processing on the plurality of rows of data to determine target row data to be distributed to the target work thread, and send the target row data to the target work thread.

[0058] It should be understood that only the working process of any target work thread in the plurality of work threads and the data partition management thread is simply described here, and the specific working process of any target work thread in the plurality of work threads and the data partition management thread can be referred to the related description in the subsequent embodiments.

[0059] FIG. 2 is a flowchart of a hash join processing method provided by the embodiments of the present disclosure. The method is applied to any target work thread in the plurality of work threads performing the hash join task in parallel, as shown in FIG. 2, and the method includes the following steps:

[0060] 201. Scan multiple rows of data within the target scan range corresponding to the target worker thread in the first data table, and write the multiple rows of data into the first hash table in memory based on the first hash connection method. The first hash connection method is a pure memory-based hash connection method.

[0061] 202. If it is determined that the available memory capacity corresponding to the hash connection task is lower than the set threshold, the scanned multiple rows of data are sent to the data partition management thread, and the second hash connection method is switched to be used. The second hash connection method is a hash connection method that supports data persistence to disk.

[0062] 203. Receive the first target row data distributed by the data partition management thread, and write the first target row data into the second hash table corresponding to the target worker thread in memory based on the second hash connection method, and perform disk write processing of the data in the second hash table. The first target row data includes each row of data that needs to be distributed to the target worker thread after the data partition management thread partitions multiple rows of data.

[0063] As described above, in this embodiment of the disclosure, it is assumed that the size of the first data table is smaller than that of the second data table, and thus a hash table is built based on the first data table.

[0064] In practical applications, a database system can receive a query statement triggered by a user. This query statement can include query keywords (such as select), hash join keywords (such as hash join), etc. The query keyword indicates which columns of data need to be read from the first data table, while the hash join keyword indicates the target column data that will serve as the join key. For example, if the first data table contains N columns of data, the query keyword indicates that only M columns of data need to be read, and these M columns of data contain the target column data that will serve as the join key, where 1 ≤ M ≤ N.

[0065] In response to the input of the above query statement, the database system requests memory allocation information corresponding to the query statement from the memory manager in the corresponding cloud server, such as allocating 2GB of memory space. Additionally, the database system starts multiple worker threads to execute the query statement. Since this embodiment only focuses on the execution of the hash join task in the query statement, it is described as follows: multiple worker threads are started to complete the hash join task in parallel, wherein different worker threads are used to scan different scan ranges of the first data table.

[0066] In practical applications, to prioritize execution speed, database systems can be configured by default to use the first hash join method, which is a pure in-memory hash join. When available memory is insufficient, the system can then switch to the second hash join method, which supports data persistence to disk.

[0067] Specifically, first, the plurality of worker threads scan the respective rows of data in the first data table in their respective assigned scan ranges, and write the scanned rows of data into the first hash table created in memory. In the first hash join mode, the plurality of worker threads can optionally share the first hash table, that is, the plurality of worker threads can all access the first hash table, including writing data into and reading data from the first hash table. It can be understood that, for the above query statement, when the memory manager allocates the memory space with an upper limit on the capacity for it, the address of the memory space will be indicated, and thus the first hash table is within the address range of the memory space.

[0068] The composition of the hash table will be described below in conjunction with FIG. 3. As shown in FIG. 3, the first hash table is actually an array for storing different hash values, such as hashl, hash2, …, hashN shown in FIG. 3. Specifically, the input information for calculating the hash value is the target column data contained in the row of data currently scanned by any worker thread. Each hash value can be associated with a linked list for storing multiple rows of data mapped to the hash value. It can be understood that, assuming there are two rows of data containing the same target column data, the same hash value will be obtained based on the target column data for hash value calculation, and thus the linked list associated with the hash value will record the two rows of data, and the two rows of data can be scanned by the same worker thread or different worker threads.

[0069] For example, in FIG. 3, it is assumed that the linked list under hashl contains the 1st row of data and the 10th row of data scanned from the first data table, the linked list under hash2 contains the 2nd row of data, the 15th row of data, and the 20th row of data scanned from the first data table, and the linked list under hashN contains the 25th row of data, the 30th row of data, the 40th row of data, and the 100th row of data scanned from the first data table.

[0070] In addition, in the pure memory type hash join manner, the first hash table created in the memory is shared by multiple worker threads, which can improve the utilization of limited available memory space. It should be understood that in the pure memory type hash join manner, if multiple worker threads each independently share a hash table, and each hash table needs to occupy a continuous memory space, when a certain worker thread scans a plurality of rows of data, and the values of the target column data in each row of data are the same, the length of the corresponding hash value under the linked list will be very long, and the length of the linked list associated with the corresponding hash value will be very short due to the different values of the target column data in some other rows of data scanned by the worker thread. If the data scanned by each worker thread presents such uneven data distribution, the utilization of memory space will be low. In view of this, the first hash table is shared by multiple worker threads in the embodiments of the present disclosure to fully utilize the limited available memory space.

[0071] In the process of using the first hash join manner by multiple worker threads to scan each row of data in the corresponding scanning range in parallel, performing hash calculation on the target column data in each row of data to obtain a hash value, and writing each row of data into the first hash table according to the obtained hash value, if it is determined that the available memory capacity corresponding to the hash join task is lower than a set threshold, the switching process of the hash join manner is started. The available memory capacity corresponding to the hash join task is the available memory capacity corresponding to the query statement. The threshold is, for example, 80% of the upper limit of the memory capacity (for example, 2 GB as an example in the foregoing). Alternatively, when the worker thread is started, the memory allocation information can be configured for it, including the upper limit of the memory capacity and the corresponding memory address information. The worker thread can query the occupation of the corresponding memory space in real time to determine whether the remaining available memory capacity is less than the set threshold.

[0072] In the switching process, first, the multiple worker threads send the scanned rows of data to the data partition management thread, and the data partition management thread performs data partition processing on the rows of data scanned from the first data table, that is, re-determines the corresponding worker thread of each row of data, and distributes each row of data to the corresponding worker thread according to the data partition processing result, and then each worker thread creates a corresponding second hash table for the distributed rows of data in the second hash join manner. Alternatively, in the embodiments of the present disclosure, each worker thread can independently share its own second hash table in the second hash join manner.

[0073] If the same target column data (i.e., the same value under the target column data) is contained in multiple rows of data, the data partition management thread determines that the multiple rows of data are distributed to the same worker thread. The data partition management thread is introduced because, in the second hash join mode, each worker thread creates a second hash table that is exclusively used by the worker thread, and the "target column data" of different worker threads should not overlap, i.e., the target column data is repartitioned between different worker threads.

[0074] For any worker thread, when it receives a row of data from the data partition management thread, on the one hand, the row of data is added to the second hash table created by the worker thread, and on the other hand, the data added to the second hash table is written to the disk, such as being deleted from the shared first hash table. The process of adding the row of data to the corresponding second hash table is as follows: the target column data contained in the row of data is hashed to obtain the corresponding hash value, and the hash value is searched in the second hash table: if the hash value does not exist, the hash value is inserted, and the row of data is written in the linked list associated with the hash value; if the hash value exists, the row of data is directly written in the linked list associated with the hash value.

[0075] As can be seen, in the above switching process, the occupation of the memory space by the multiple worker threads does not increase significantly, because each time a record is added to the second hash table exclusively used by each worker thread, a corresponding record is deleted from the first hash table.

[0076] After the data partition management thread completes the data partition processing of the multiple rows of data scanned by the multiple worker threads and redistributes the multiple rows of data to the corresponding worker threads, the data partition management thread can trigger a memory release instruction. Each worker thread, upon sensing the memory release instruction, can transfer at least part of the data in the second hash table exclusively used by each worker thread to the disk for storage, which can reduce the occupation of the memory by the multiple second hash tables, thereby accommodating the writing of new data scanned from the first data table to the second hash table.

[0077] In summary, after the target worker thread receives the first target row of data distributed by the data partition management thread, the data in the second hash table is written to the disk, which can include: deleting the first target row of data from the first hash table, and in response to the memory release instruction triggered by the data partition management thread after completing the partition processing of the above scanned multiple rows of data, transferring at least part of the data in the second hash table to the disk for storage.

[0078] In order to facilitate understanding of the above processing process, the above steps are described below in conjunction with a specific example:

[0079] Suppose the first data table is a table for storing the mathematics scores of 10000 students in a mathematics competition, which specifically includes the student's name, examination identification number, mathematics score and other columns of data, and the second data table is a table for storing the examination identification number, mathematics score, school and grade of the 100 students, and in the first data table and the second data table, each row of data corresponds to multiple columns of data of a student. For the first data table and the second data table, suppose the target column data set as the connection key of the hash connection is the mathematics score, so in the specific implementation, the data in the two tables can be connected based on the column data of the mathematics score to obtain a new table representing the connection result, thereby completing the hash connection task.

[0080] In this process, taking any target worker thread i in the multiple worker threads performing the hash connection task in parallel as an example, suppose the target scanning range corresponding to the target worker thread i is the first 500 rows of data, so in actual application, the data in the first data table can be scanned line by line first, and each row of data that has been scanned can be written into the first hash table in the memory based on the pure memory type hash connection mode.

[0081] Suppose the upper limit of the memory capacity corresponding to the hash connection task is 2GB, so suppose the worker thread i scans the first 300 rows and finds that the remaining available memory capacity is lower than the set threshold value, so at this time the worker thread i can send the first 300 rows of data that have been scanned to the data partition management thread to perform partition processing on the first 300 rows of data through the data partition management thread. In addition, the worker thread i switches the hash connection mode from the pure memory type hash connection mode to the hash connection mode supporting data falling to disk, for example, the worker thread i maintains an identification bit of the hash connection mode, and different values are set to represent different hash connection modes. In addition, the worker thread i can also pause the scanning of the remaining multiple rows of data at this time to avoid continuously writing data into the first hash table in the case of insufficient available memory capacity, resulting in too much data being written into the first hash table, thereby causing the memory occupancy to be too high and affecting the normal operation of the database system.

[0082] Of course, since other worker threads also send multiple rows of data that have been scanned to the data partition management thread, the data partition management thread performs data partition processing on all the data sent by the multiple worker threads.

[0083] Table 1: Multiple rows of data sent by the worker thread i

[0084] In order to facilitate understanding, the partition processing process and result will be illustrated below in combination with Table 1, wherein Table 1 is part of the multiple rows of data scanned by the worker thread i.

[0085] The data partition management thread processes the data partition of the eight rows of data in Table 1 as follows: based on a random number, the target column data in any of the scanned rows of data is hashed to obtain a hash value corresponding to the any row of data, and then the hash value is used to calculate the modulus of the parallelism to obtain the work thread label corresponding to the any row of data, so that the any row of data is distributed to the work thread with the corresponding label, wherein the parallelism is the total number of work threads. The target column data is the column that is the hash join key.

[0086] In a specific implementation, taking the first row of data in Table 1 as an example, it is assumed that the parallelism is 2, that is, there are two work threads: work thread i and work thread j. The target column data (math score = 70) in the first row of data is calculated by a set hash algorithm to obtain an initial hash value, and then the initial hash value is used to calculate the modulus of a random number to obtain the target hash value corresponding to the first row of data, wherein the hash algorithm includes but is not limited to a hash algorithm. It should be noted that the hash algorithm used here can be the same as or different from the hash algorithm used to create the first hash table. After obtaining the target hash value corresponding to the first row of data, the target hash value is used to calculate the modulus of the parallelism to obtain the work thread label corresponding to the first row of data. Based on the same method, the work thread labels corresponding to the remaining seven rows of data in the above eight rows of data can be determined.

[0087] It is assumed that the determined work thread labels corresponding to the above eight rows of data are as follows: the work thread labels corresponding to the first row of data, the second row of data and the third row of data in Table 1 with a math score of "70" are i, and the work thread labels corresponding to the fourth row of data to the eighth row of data with a math score of "65" and "80" are j.

[0088] As can be seen, through the data partition processing process of the data partition management thread, the work thread i sends multiple rows of data, and finally only part of the data may be redistributed to the work thread i, such as the first row of data, the second row of data and the third row of data in Table 1. The other part of the data may be sent to the other work thread j, such as the fourth row of data to the eighth row of data in Table 1.

[0089] Still taking the work thread i as an example, after receiving the target row data (such as the first three rows of data in Table 1) sent by the data partition management thread, the work thread i will write these rows of data into the second hash table shared by the work thread i in the memory based on the hash join mode supporting data disk landing, and delete these rows of data from the first hash table.

[0090] After the data partition management thread completes the partition processing of the scanned row data sent by all the worker threads, the data partition management thread triggers a memory release instruction, and the worker thread i transfers at least part of the data in the second hash table corresponding to the worker thread i to the disk for storage in response to the memory release instruction. For example, assuming that there are M hash values in the second hash table corresponding to the worker thread i, the M hash values can be sorted in descending order of the amount of data in the linked list associated with each hash value. Assuming that the target data amount that each worker thread can transfer each time is set, the data associated with the hash values at the top of the sorting result is transferred to the disk for storage. Of course, the specific transfer strategy is not limited thereto.

[0091] After the worker thread i completes the data transfer and storage processing, it can continue to scan the remaining rows of data in the target scanning range and write the scanned rows of data into the corresponding second hash table based on the second hash join method. When the available memory capacity is found to be lower than the set threshold again, the data transfer and storage process is executed again to ensure that there is no memory overflow problem.

[0092] In summary, in the case of multiple worker threads completing the hash join task in parallel, the first memory-based hash join method is used preferentially when the available memory capacity is sufficient, and the second hash join method supporting data landing on the disk is switched to when the available memory capacity is insufficient, while the execution speed and reliable execution of the task are taken into account.

[0093] FIG. 4 is a flowchart of a hash join processing method provided by an embodiment of the present disclosure, which can be executed by any target worker thread described above. As shown in FIG. 4, the method can include the following steps:

[0094] 401. Scan multiple rows of data in the target scanning range corresponding to the target worker thread in the first data table, and write the multiple rows of data into the first hash table in the memory based on the first hash join method, which is a pure memory-based hash join method.

[0095] 402. If it is detected that the memory manager sets the first flag bit to the first preset value, it is determined that the available memory capacity of the hash join task is lower than the set threshold, the scanning of the remaining rows of data in the target scanning range is paused, the scanned multiple rows of data are sent to the data partition management thread, and the second hash join method is switched to, which is a hash join method supporting data landing on the disk.

[0096] 403. Receive the first target row data distributed by the data partition management thread, and write the first target row data into the second hash table corresponding to the target worker thread in memory based on the second hash connection method, and delete the first target row data from the first hash table. The first target row data includes the rows of data that need to be distributed to the target worker thread after the data partition management thread partitions the multiple rows of data.

[0097] 404. In response to a memory release instruction triggered after the data partition management thread completes the partitioning of multiple rows of data, at least a portion of the data in the second hash table is transferred to the disk for storage.

[0098] 405. After transferring at least a portion of the data in the second hash table to the disk, continue scanning the remaining rows of data within the target scan range, determine the second target row data that needs to be written into the second hash table from the remaining rows of data scanned, and write the second target row data into the second hash table based on the second hash connection method.

[0099] Optionally, in practical applications, during the process of multiple worker threads scanning rows of data within their respective scanning ranges in parallel using the first hash join method, performing hash calculations on the target column data in each row to obtain a hash value, and writing each row of data into the first hash table based on the obtained hash value, any target worker thread among the multiple worker threads can monitor the first flag bit maintained by the memory manager in real time. If the memory manager sets the first flag bit to a first preset value (e.g., sets the first flag bit to true), it is determined that the available memory capacity corresponding to the hash join task is lower than a set threshold, and the hash join method switching process begins. During the switching process, the target worker thread pauses scanning the remaining rows of data within the target scanning range and sends the scanned rows of data to the data partition management thread. The memory manager sets the first flag bit corresponding to the hash join task to the first preset value when it detects that the available memory capacity corresponding to the hash join task is lower than the set threshold, indicating that the available memory capacity of the hash join task is insufficient. It is understood that the memory manager can maintain a first flag bit for different hash join tasks, setting it to the corresponding first flag bit based on the available memory capacity of different hash join tasks.

[0100] The specific execution process of steps 403 and 404 can be found in the above embodiments, and will not be repeated here.

[0101] In step 405, after transferring at least part of the data in the second hash table to the disk, the target worker thread continues to scan the remaining rows of data in the target scan range. Specifically, after transferring at least part of the data in the second hash table to the disk, the target worker thread feeds back confirmation information to the data partition management thread, so that the data partition management thread sets the second flag bit to a second preset value based on the confirmation information, and continues to scan the remaining rows of data in the target scan range when it is detected that the second flag bit is set to the second preset value. The second preset value indicates that all the worker threads have completed the memory release instructions, that is, each worker thread can send confirmation information to the data partition management thread after transferring at least part of the data in the corresponding second hash table to the disk based on the memory release instruction triggered by the data partition management thread. After receiving the confirmation information sent by all the worker threads, the data partition management thread sets the second flag bit to the second preset value.

[0102] The second flag bit is similar to the first flag bit and can be maintained by the memory manager. Based on this, the data partition management thread sends a command to the memory manager to set the second flag bit to the second preset value after receiving the confirmation information triggered by the worker threads. The target worker thread listens to the second flag bit in real time during the work process. If it detects that the second flag bit is set to the second preset value (for example, the second flag bit is set to true), it means that the hash join mode switching process has been completed and the available memory capacity is sufficient. At this time, the target worker thread can continue to scan the remaining rows of data in the target scan range.

[0103] After continuing to scan the remaining rows of data in the target scan range, the second target row data that needs to be written into the second hash table can be determined from the scanned remaining rows of data, and the second target row data is written into the second hash table based on the second hash join mode. Specifically, the target worker thread performs data partitioning processing on the scanned remaining rows of data through the data partitioning operator corresponding to the second hash join mode to determine the second target row data that needs to be written into the target cache space corresponding to the target worker thread, writes the second target row data into the target cache space, and then reads the second target row data from the target cache space and writes the second target row data into the second hash table based on the second hash join mode. Each of the plurality of worker threads corresponds to a different cache space.

[0104] In actual application, the specific steps of performing data partition processing on the scanned remaining rows of data are as follows: based on the random number, performing hash processing on the target column data in any row of data in the scanned remaining rows of data to obtain a hash value corresponding to the any row of data, performing modulo calculation on the parallelism degree by using the hash value to obtain a work thread label corresponding to the any row of data, and if the work thread label corresponds to the target work thread, determining that the second target row of data corresponding to the target work thread contains the any row of data, where the parallelism degree is the total number of the plurality of work threads.

[0105] In specific implementation, for example, it is assumed that there are 500 rows of data in the target scanning range, and there are 8 rows of data remaining after the last scanning, that is, the 8 rows of data in Table 1. Based on this, in actual application, the 8 rows of data in Table 1 can be partitioned by using a data partition operator corresponding to the second hash connection mode, and the partitioning process is the same as the partitioning process in the above embodiment, which will not be described here.

[0106] It is assumed that the work thread labels corresponding to the above-mentioned 8 rows of data finally determined are as follows: the work thread labels corresponding to the first row of data, the second row of data and the third row of data in Table 1 with the mathematics score of 70 are i, and the work thread labels corresponding to the fourth row of data to the eighth row of data with the mathematics score of 65 and 80 are j.

[0107] At this time, the data of each row can be written into the target cache space corresponding to the work thread, for example, the first row of data, the second row of data and the third row of data are written into the target cache space corresponding to the work thread i, and the fourth row of data to the eighth row of data are written into the target cache space corresponding to the work thread j. Then, each work thread reads the written data from the target cache space corresponding thereto at a regular time, and writes the read data into the second hash table corresponding thereto based on the second hash connection mode.

[0108] In the embodiments of the present disclosure, by causing the target work thread to listen to the first flag bit and the second flag bit in real time during the work process, the target work thread can be controlled to perform corresponding operations in a timely manner when the states of the first flag bit and the second flag bit change, for example, when the first flag bit is set to the first preset value, the target work thread needs to pause the scanning of the remaining rows of data in the target scanning range, send the scanned rows of data to the data partition management thread, and switch to the second hash connection mode, and when the second flag bit is set to the second preset value, the target work thread can continue the scanning of the remaining rows of data in the target scanning range, and the two are mutually matched to flexibly realize the switching of data scanning or paused scanning.

[0109] FIG. 5 is a flowchart of another hash join processing method provided by the embodiment of the present disclosure, which is applied to a data partition management thread. As shown in FIG. 5, the method comprises the following steps:

[0110] 501. Receiving multi-line data sent by a target worker thread, the multi-line data being sent by the target worker thread when it is determined that the available memory capacity corresponding to a hash join task is lower than a set threshold, the target worker thread being any one of a plurality of worker threads that execute the hash join task in parallel, the hash join task being used to connect a first data table and a second data table by taking target column data in the first data table and the second data table as a connection key, wherein the multi-line data is scanned by the target worker thread in a target scan range corresponding to the target worker thread in the first data table, and the multi-line data is written into a first hash table in the memory based on a first hash join method, the first hash join method being a pure memory type hash join method.

[0111] 502. Partitioning the multi-line data to determine target line data that needs to be distributed to the target worker thread.

[0112] 503. Sending the target line data to the target worker thread, so that the target worker thread writes the target line data into a second hash table corresponding to the target worker thread in the memory based on a second hash join method, and performs disk writing of data in the second hash table, the second hash join method being a hash join method that supports data disk writing.

[0113] The execution process of the hash join processing in the embodiment will not be described in detail, and reference can be made to the description in the foregoing other embodiments.

[0114] In order to enable a more intuitive understanding of the hash join processing method provided by the embodiment of the present disclosure, an example is illustrated in combination with FIG. 6. As shown in FIG. 6, in the embodiment, the hash join processing method is divided into three stages: a first stage, a second stage and a third stage, wherein the first stage indicates that the memory is sufficient, in this stage, the hash join task is executed by using a pure memory type hash join method, the execution speed is fast, and the speed advantage can be fully utilized. The second stage indicates that the memory is insufficient, in this stage, state switching needs to be performed, from the pure memory type hash join method to the hash join method that supports data disk writing. The third stage indicates that the state switching is completed, in this stage, the hash join task is continued to be executed by using the hash join method that supports data disk writing.

[0115] In actual application, it is assumed that the first data table is a table for storing the mathematics scores of 10000 students in a mathematics competition, and specifically includes the name, examination identification number, and mathematics score of each student, and the second data table is a table for storing the examination identification number, mathematics score, school, and grade of 100 students, and in the first data table and the second data table, each row of data corresponds to the multi-column data of a student. For the first data table and the second data table, it is assumed that the target column data set as the connection key of the hash connection is the mathematics score, and then in specific implementation, the data in the two tables can be connected based on the column data of the mathematics score to obtain a new table representing the connection result, thereby completing the hash connection task. In addition, it needs to be noted that in FIG. 6, the data amount of the work thread is 2 (including the work thread 1 and the work thread 2), that is, the parallelism is 2. It is assumed that the target scanning range corresponding to the work thread 1 is the first 5000 rows of data, and the target scanning range corresponding to the work thread 2 is the last 5000 rows of data.

[0116] As shown in FIG. 6, in the process of executing the hash connection task, each work thread uses many operators or components to perform different processing on the scanned data.

[0117] The above three stages are described in detail as follows.

[0118] The first stage:

[0119] Assume that at this time, the hash join mode is the first hash join mode, and the hash join operator actually used is the parallel hash join operator shown in FIG. 6. The work thread 1 performs scanning in the first data table through the table scanning operator 1. In order to improve the scanning efficiency, the scanning of the data rows can be performed in units of data blocks (also referred to as chunks). Each data block is assumed to contain 1000 data rows. Then, the table scanning operator 1 corresponding to the work thread 1 can input the data block scanned by the table scanning operator 1 to the data distribution operator 1 corresponding to the work thread 1 after scanning the data block. Optionally, as shown in FIG. 6, the data scheduler 1 can be included between the table scanning operator 1 and the data distribution operator 1. The data scheduler 1 can be used to send the data block scanned by the table scanning operator 1 to the data distribution operator 1 in real time or periodically. Assume that the distribution strategy adopted by the data distribution operator 1 is random distribution. Then, the data distribution operator 1 can randomly send the received data block to the buffer 1 corresponding to the work thread 1 and the buffer 2 corresponding to the work thread 2. For example, the data distribution operator 1 sends the first received data block to the buffer 1 and the second received data block to the buffer 2. Then, the parallel hash join operator 1 corresponding to the work thread 1 reads and parses each data row included in each data block from the buffer 1, calculates the hash value according to the target column data, and writes each data row into the first hash table based on the hash value calculation result. Optionally, as shown in FIG. 6, the data scheduler 2 can be included between the buffer 1 and the parallel hash join operator 1. The data scheduler 2 can be used to read the data block written in the buffer 1 and send the data block to the parallel hash join operator 1 in real time or periodically.

[0120] Similarly, the work thread 2 performs the similar data scanning, distribution, and writing into the first hash table through the table scanning operator 2, the data scheduler 3, the data distribution operator 2, the buffer 2, the data scheduler 4, and the parallel hash join operator 2. Details are not described herein.

[0121] Second stage:

[0122] Assume that at this time, the hash join operator adopted in the second hash join mode is the hybrid hash join operator shown in FIG. 6. The worker thread 1 and the worker thread 2 listen to the first flag (assume named as: isStartMemoryRevoke) maintained by the memory manager in real time. If the memory manager sets the flag isStartMemoryRevoke to true, it is determined that the available memory capacity of the hash join task is lower than the set threshold, and the switching of the hash join mode is started. At this time, on the one hand, the worker thread 1 and the worker thread 2 can modify the data distribution strategy adopted in the data distribution operator 1 and the data distribution operator 2, such as modifying from the random distribution strategy to the partition distribution strategy shown in FIG. 6, and on the other hand, the respective sampled hash join operators are modified to the hybrid hash join operator 1 and the hybrid hash join operator 2. In addition, at this time, the table scan operator 1 and the table scan operator 2 will not continue to scan data in the first data table, and the data distribution operator 1 and the data distribution operator 2 will not send data to the buffer 1 and the buffer 2.

[0123] Specifically, the data scheduler 1 and the data scheduler 3 corresponding to the worker thread 1 and the worker thread 2 can listen to the flag isStartMemoryRevoke. When it is found that the flag isStartMemoryRevoke is set to true, the table scan operator 1 and the table scan operator 2 are controlled to stop scanning data, so that the data distribution operator 1 and the data distribution operator 2 will not receive data any more, and will not send data to the buffer 1 and the buffer 2 any more. The data scheduler 1 and the data scheduler 3 can also control the respective data distribution operator 1 and the data distribution operator 2 to modify the adopted data distribution strategy.

[0124] In addition, the data scheduler 2 and the data scheduler 4 corresponding to the worker thread 1 and the worker thread 2 can listen to the flag isStartMemoryRevoke. When it is found that the flag isStartMemoryRevoke is set to true, the respective worker thread is controlled to send the scanned rows of data to the data partition management thread. The scanned rows of data include the rows of data in the buffer 1 and the buffer 2 which have not been written into the first hash table and the rows of data which have been written into the first hash table. The data scheduler 2 and the data scheduler 4 can also control the respective hash join operator to switch to the hybrid hash join operator 1 and the hybrid hash join operator 2.

[0125] Afterwards, the data partition management thread receives each row of data, and performs partition processing on the data, and based on the determination result of the work thread label corresponding to each row of data, sends each row of data to the hash join operator corresponding to the work thread, i.e., the hybrid hash operator 1 and the hybrid hash operator 2, which will respectively construct the second hash table corresponding thereto.

[0126] Afterwards, the data partition management thread completes the partition processing on all received data and distributes the data to the corresponding hash join operator, triggers a memory release instruction. Specifically, the data partition management thread can call the setting interface function (for example, named startMemoryRevoke) of each hash join operator (hybrid hash join operator 1 and hybrid hash join operator 2) to trigger the memory release instruction. Based on the memory release instruction, the hybrid hash join operator 1 and the hybrid hash join operator 2 transfer at least part of the data in the second hash table corresponding thereto to the disk, and after transferring at least part of the data to the disk, feed back confirmation information to the data partition management thread, so that the data partition management thread sets the second flag (assuming named isFinishMemoryRevoke) to true based on the confirmation information, indicating the end of the switching process.

[0127] The third stage:

[0128] When the data scheduler 1 and the data scheduler 3 corresponding to the work thread 1 and the work thread 2 detect that the second flag is set to true through the set monitoring code, the corresponding table scan operator 1 and the table scan operator 2 continue to scan the remaining rows of data within the target scan range of the corresponding work thread.

[0129] Taking the table scan operator 1 as an example, after the data partition processing of each row of data newly scanned by the table scan operator 1 through the data distribution operator 1, the data is sent to the buffer 1 and the buffer 2, and the hybrid hash operator 1 reads each row of data newly scanned from the buffer 1 and writes it into the second hash table corresponding thereto.

[0130] In addition, it should be noted that in actual application, the number of data partition management threads can be one or multiple in parallel to improve the data partition processing efficiency of the scanned rows of data in the first hash join mode.

[0131] In summary, the embodiments of the present disclosure can flexibly switch different hash join modes according to the memory occupation of the first data table. Specifically, since the pure memory hash join mode has fast task execution speed, when the memory occupation of the first data table is not large, that is, the available memory capacity is sufficient, the data is preferentially written into the first hash table in the memory to ensure the execution speed. In this process, if it is found that the memory occupation of the first data table is too large, resulting in insufficient available memory capacity, the hash join mode is switched (from the pure memory hash join mode to the hash join mode supporting data disk landing), so that part of the data is stored in the memory and the other part of the data is stored in the disk, ensuring the smooth creation of the second hash table and the smooth progress of the hash join task.

[0132] The hash join processing apparatus of one or more embodiments of the present disclosure will be described in detail below. Those skilled in the art can understand that these apparatuses can be configured using commercially available hardware components through the steps taught by the present solution.

[0133] FIG. 7 is a structural schematic diagram of a hash join processing apparatus provided by an embodiment of the present disclosure. The apparatus is applied to any target worker thread in a plurality of worker threads performing a hash join task in parallel, and the hash join task is used to connect a first data table and a second data table by taking target column data in the first data table and the second data table as a connection key. As shown in FIG. 7, the apparatus includes a writing module 71, a sending module 72, and a receiving module 73.

[0134] The writing module 71 is configured to scan a plurality of rows of data in a target scanning range corresponding to the target worker thread in the first data table, and write the plurality of rows of data into a first hash table in the memory based on a first hash join mode, the first hash join mode being a pure memory hash join mode.

[0135] The sending module 72 is configured to send the scanned plurality of rows of data to a data partition management thread if it is determined that the available memory capacity corresponding to the hash join task is lower than a set threshold, and switch to a second hash join mode, the second hash join mode being a hash join mode supporting data disk landing.

[0136] The receiving module 73 is configured to receive first target row data distributed by the data partition management thread, and write the first target row data into a second hash table corresponding to the target worker thread in the memory based on the second hash join mode, and perform disk landing processing of the data in the second hash table, wherein the first target row data includes each row of data determined by the data partition management thread after partitioning processing of the plurality of rows of data and needing to be distributed to the target worker thread.

[0137] Optionally, the apparatus further comprises a scanning module configured to, if it is determined that the available memory capacity corresponding to the hash join task is below a set threshold, pause scanning of the remaining rows of data within the target scan range.

[0138] Optionally, the apparatus further comprises a listening module configured to, if it is listened that a memory manager sets a first flag bit to a first preset value, determine that the available memory capacity corresponding to the hash join task is below a set threshold, the memory manager being configured to set the first flag bit to the first preset value when it is detected that the available memory capacity corresponding to the hash join task is below the set threshold.

[0139] Optionally, the scanning module is further configured to, after transferring at least part of the data in the second hash table to the disk, continue scanning of the remaining rows of data within the target scan range. The writing module 71 is further configured to determine, from the scanned remaining rows of data, second target rows of data that need to be written into the second hash table, and write the second target rows of data into the second hash table based on the second hash join manner.

[0140] Optionally, the scanning module is specifically configured to, after transferring at least part of the data in the second hash table to the disk, feed back confirmation information to the data partition management thread, so that the data partition management thread sets a second flag bit to a second preset value based on the confirmation information, the second preset value indicating that the plurality of worker threads have completed respective memory release instructions; and if it is listened that the second flag bit is set to the second preset value, continue scanning of the remaining rows of data within the target scan range.

[0141] Optionally, the writing module 71 is specifically configured to: perform data partition processing on the scanned remaining rows of data respectively by a data partition operator corresponding to the second hash join manner, to determine the second target rows of data that need to be written into a target cache space corresponding to the target worker thread, the plurality of worker threads each corresponding to a different cache space; write the second target rows of data into the target cache space; read the second target rows of data from the target cache space, and write the second target rows of data into the second hash table based on the second hash join manner.

[0142] Optionally, the writing module 71 is specifically further configured to: perform hash processing on any row of data in the scanned remaining rows of data based on a random number, to obtain a hash value corresponding to the any row of data; perform modulo calculation on a parallelism degree by using the hash value, to obtain a worker thread label corresponding to the any row of data, the parallelism degree being a total number of the plurality of worker threads.

[0143] The device shown in FIG. 7 can perform the steps of the hash join processing method executed in the foregoing embodiments. For details of the execution process and technical effects, refer to the descriptions in the foregoing embodiments, which will not be repeated here.

[0144] FIG. 8 is a structural schematic diagram of another hash join processing device provided by an embodiment of the present disclosure. The device is applied to a data partition management thread. As shown in FIG. 8, the device includes a receiving module 81, a processing module 82, and a sending module 83.

[0145] The receiving module 81 is configured to receive multiple rows of data sent by a target worker thread. The multiple rows of data are sent by the target worker thread when it is determined that the available memory capacity corresponding to a hash join task is lower than a set threshold. The target worker thread is any one of multiple worker threads that execute the hash join task in parallel. The hash join task is used to connect a first data table and a second data table by taking target column data in the first data table and the second data table as a connection key. The multiple rows of data are scanned by the target worker thread in a target scanning range corresponding to the target worker thread in the first data table. The multiple rows of data are written into a first hash table in the memory based on a first hash join manner. The first hash join manner is a pure memory type hash join manner.

[0146] The processing module 82 is configured to perform partition processing on the multiple rows of data to determine target rows of data that need to be distributed to the target worker thread.

[0147] The sending module 83 is configured to send the target rows of data to the target worker thread, so that the target worker thread writes the target rows of data into a second hash table corresponding to the target worker thread in the memory based on a second hash join manner and performs disk writing processing on data in the second hash table. The second hash join manner is a hash join manner that supports data disk writing.

[0148] The device shown in FIG. 8 can perform the steps of the hash join processing method executed in the foregoing embodiments. For details of the execution process and technical effects, refer to the descriptions in the foregoing embodiments, which will not be repeated here.

[0149] FIG. 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG. 9, the electronic device can include a processor 91, a memory 92, and a communication interface 93. The memory 92 stores executable code. When the executable code is executed by the processor 91, the processor 91 performs the hash join processing method in the foregoing embodiments.

[0150] In addition, an embodiment of the present disclosure provides a non-transitory machine readable storage medium. The non-transitory machine readable storage medium stores executable code. When the executable code is executed by a processor of an electronic device, the processor performs the hash join processing method in the foregoing embodiments.

[0151] The embodiment of the present disclosure provides a computer program product, which comprises a computer program, when the computer program is executed by a processor of an electronic device, the processor executes the hash connection processing method in the foregoing embodiment.

[0152] The apparatus embodiments described above are merely illustrative, wherein the network elements described as separate components can or can not be physically separated. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0153] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of an appropriate general hardware platform, and of course can also be implemented by means of a combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of a computer program product, and the present disclosure can be embodied in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0154] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A hash join processing method, wherein, A target worker thread of a plurality of worker threads applied to perform a hash join task, the hash join task being used to join a first data table and a second data table with target column data in the first data table and the second data table as a join key, the method comprising: scanning a plurality of rows of data in a target scan range corresponding to the target worker thread in the first data table, and writing the plurality of rows of data into a first hash table in memory based on a first hash join manner, the first hash join manner being a pure memory hash join manner; if it is determined that available memory capacity corresponding to the hash join task is lower than a set threshold, sending the scanned plurality of rows of data to a data partition management thread, and switching to a second hash join manner, the second hash join manner being a hash join manner supporting data disk landing; receiving first target rows of data distributed by the data partition management thread, and writing the first target rows of data into a second hash table corresponding to the target worker thread in memory based on the second hash join manner, and performing disk landing processing of data in the second hash table, wherein the first target rows of data include each row of data determined by the data partition management thread after partition processing of the plurality of rows of data to need to be distributed to the target worker thread.

2. The method of claim 1, wherein, The first hash table is shared by the plurality of worker threads, and the second hash table is exclusively used by the target worker thread.

3. The method of claim 1 or 2, wherein, The performing disk landing processing of data in the second hash table comprises: deleting the first target rows of data from the first hash table; in response to a memory release instruction triggered by the data partition management thread after completing the partition processing of the plurality of rows of data, transferring at least part of data in the second hash table to a disk for storage.

4. The method according to any one of claims 1 to 3, wherein, The method further comprises: if it is determined that available memory capacity corresponding to the hash join task is lower than the set threshold, pausing scanning of remaining rows of data in the target scan range.

5. The method according to any one of claims 1 to 4, wherein, The method further comprises: if a first flag bit is set to a first preset value by a memory manager, it is determined that available memory capacity corresponding to the hash join task is lower than a set threshold, the memory manager being used to set the first flag bit to the first preset value when detecting that available memory capacity corresponding to the hash join task is lower than the set threshold.

6. The method according to any one of claims 3-5, wherein, The method further comprises: after transferring at least part of data in the second hash table to a disk, continuing scanning of remaining rows of data in the target scan range; determining second target rows of data that need to be written into the second hash table from the scanned remaining rows of data, and writing the second target rows of data into the second hash table based on the second hash join manner.

7. The method of claim 6, wherein, The continuing scanning of remaining rows of data in the target scan range after transferring at least part of data in the second hash table to a disk comprises: After transferring at least part of data in the second hash table to the disk, feedback confirmation information is fed back to the data partition management thread, so that the data partition management thread sets a second flag bit to a second preset value based on the confirmation information, and the second preset value indicates that the plurality of worker threads have completed their respective memory release instructions; If the second flag bit is set to the second preset value, scanning of the remaining rows of data in the target scanning range is continued.

8. The method of claim 6 or 7, wherein, The second target row data determined from the scanned remaining rows of data that need to be written into the second hash table is written into the second hash table based on the second hash join manner, including: The plurality of worker threads each correspond to a different cache space; The second target row data is written into the target cache space corresponding to the target worker thread; The second target row data is read from the target cache space and written into the second hash table based on the second hash join manner.

9. The method of claim 8, wherein, The second target row data determined by separately performing data partition processing on the scanned remaining rows of data, includes: Based on a random number, the target column data in any row of data in the scanned remaining rows of data is hashed to obtain a hash value corresponding to the any row of data; The hash value is used to perform modulo calculation on the parallelism to obtain a worker thread label corresponding to the any row of data, and the parallelism is the total number of the plurality of worker threads; If the worker thread label corresponds to the target worker thread, it is determined that the second target row data includes the any row of data.

10. A hash join processing method, wherein, The method applied to a data partition management thread, includes: Receiving a plurality of rows of data sent by a target worker thread, the plurality of rows of data being sent by the target worker thread when determining that the available memory capacity corresponding to a hash join task is lower than a set threshold, the target worker thread being any one of a plurality of worker threads that execute a hash join task in parallel, the hash join task being used to connect a first data table and a second data table with target column data in the first data table and the second data table as a connection key, wherein the plurality of rows of data are scanned by the target worker thread in a target scanning range corresponding to the target worker thread in the first data table, and the plurality of rows of data are written into a first hash table in memory based on a first hash join manner, the first hash join manner being a pure memory type hash join manner; Partitioning the plurality of rows of data to determine target row data that needs to be distributed to the target worker thread; The target row data is sent to the target worker thread, so that the target worker thread writes the target row data into a second hash table corresponding to the target worker thread in the memory based on a second hash join mode, and performs disk writing of data in the second hash table, the second hash join mode being a hash join mode supporting data disk writing.

11. A database system, wherein, The method comprises: A plurality of worker threads and a data partition management thread are used to perform a hash join task in parallel, wherein the hash join task is used to join a first data table and a second data table based on target column data in the first data table and the second data table as a connection key; Any target worker thread in the plurality of worker threads is used to scan a plurality of rows of data in a target scanning range corresponding to the target worker thread in the first data table, and write the plurality of rows of data into a first hash table in the memory based on a first hash join mode, and if it is determined that an available memory capacity corresponding to the hash join task is lower than a set threshold, the plurality of rows of scanned data are sent to the data partition management thread, and a second hash join mode is switched to, target row data distributed by the data partition management thread is received, and the target row data is written into a second hash table corresponding to the target worker thread in the memory based on the second hash join mode, and disk writing of data in the second hash table is performed; wherein the first hash join mode is a pure memory hash join mode, and the second hash join mode is a hash join mode supporting data disk writing; The data partition management thread is used to partition the plurality of rows of data to determine the target row data that needs to be distributed to the target worker thread, and send the target row data to the target worker thread.

12. An electronic device, comprising: The method comprises: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the hash join processing method according to any one of claims 1 to 10.

13. A non-transitory machine-readable storage medium, wherein, The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor executes the hash join processing method according to any one of claims 1 to 10.

14. A computer program product, wherein, The method comprises: A computer program, when executed by the processor of the electronic device, makes the processor execute the hash join processing method according to any one of claims 1 to 10.

Images