Hash connection processing method and device, equipment, medium and product

Abstract

The invention provides a Hash connection processing method and device, equipment, a medium and a product, in the process of establishing Hash connection, a plurality of Hash tables corresponding to a plurality of batches of a construction table are generated in a memory, and a Hash value corresponding to each piece of data in the construction table is calculated; and in the process of storing the hash value corresponding to each piece of data into the hash table corresponding to each piece of data in the memory, storing at least one hash table in the plurality of hash tables from the memory into the disk according to the remaining memory, and finally scanning each piece of data in the detection table to obtain the hash connection. Due to the fact that the established hash tables of different batches are stored in the memory as much as possible, the number of times of interaction with the disk is reduced in the process of establishing the hash connection, the data interaction amount between the disk and the disk is reduced, and then the performance of the database is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, device, equipment, medium and product for processing hash connections. Background Art

[0002] Hash join is a connection algorithm between tables in a database, which obtains a hash result set through hash operation. This connection method has high efficiency when processing large amounts of data, so it has been widely used. When establishing a hash join, the hash join establishment device first uses one table as a build table and one table as a detection table, and establishes a hash table based on the build table, and then scans the data in the detection table based on the hash table to obtain a hash join.

[0003] With the prior art, when the hash table to be processed is too large, the hash connection establishment device will build a table to build the hash table in batches, wherein, during the process of building the hash table, the first batch of hash tables are stored in the memory, and the other batches of hash tables are stored in the disk. When all batches of hash tables are built, the detection table is scanned, and during the scanning process, the data in the detection table is connected to the corresponding hash tables in multiple hash tables, thereby obtaining a hash connection.

[0004] With the existing technology, during the process of establishing a hash connection, the hash connection establishing device will interact with the disk multiple times to obtain different batches of hash tables, thereby increasing the amount of data interaction with the disk and further reducing the performance of the database. Summary of the invention

[0005] The present application provides a hash connection processing method, device, equipment, medium and product to reduce the amount of data interaction with the disk during the process of establishing a hash connection.

[0006] The first aspect of the present application provides a method for processing hash connections, including: obtaining a construction table and a detection table of the hash connection; when the memory required for the hash table of the construction table is greater than a first preset value, generating multiple hash tables corresponding to multiple batches of the construction table in the memory; calculating the hash value of each data in the construction table, and in the process of storing the hash value of each data in the hash table corresponding to each data in the memory, storing at least one hash table of the multiple hash tables from the memory to the disk according to the remaining memory.

[0007] A second aspect of the present application provides a hash connection processing device, including: an acquisition module, used to acquire a construction table and a detection table of a hash connection; a construction module, used to generate multiple hash tables corresponding to multiple batches of the construction table in the memory when the memory required by the hash table of the construction table is greater than a first preset value, calculate the hash value of each data in the construction table, and in the process of storing the hash value of each data in the hash table corresponding to each data in the memory, store at least one hash table of the multiple hash tables from the memory to the disk according to the remaining memory; a detection module, used to scan each data in the detection table, and during the scanning process, connect the corresponding hash tables in the multiple hash tables according to the hash value of each data to obtain the hash connection.

[0008] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method described in the first aspect of the present application.

[0009] A fourth aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method described in the first aspect of the present application.

[0010] A fifth aspect of the present application provides a computer program product, including a computer program, which implements the method described in the first aspect of the present application when executed.

[0011] The hash connection processing method, device, equipment, medium and product provided by the present application, in the process of establishing a hash connection, the database generates multiple hash tables corresponding to multiple batches of the construction table in the memory, calculates the hash value corresponding to each data in the construction table, and in the process of storing the hash value corresponding to each data in the hash table corresponding to each data in the memory, according to the remaining memory, at least one hash table of the multiple hash tables is stored from the memory to the disk, and finally the detection table is scanned to obtain the hash connection. Since the hash tables of different batches established by the database are stored in the memory as much as possible, the number of interactions with the disk in the process of establishing the hash connection is reduced, thereby reducing the amount of data interaction between the database and the disk, thereby improving the performance of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 A schematic diagram of a hash connection scenario used in this application;

[0014] Figure 2 A schematic diagram of a method for constructing a hash connection in the prior art;

[0015] Figure 3 A flowchart of an embodiment of a method for processing a hash connection provided by the present application;

[0016] Figure 4 A schematic diagram of a storage state of a hash table provided in this application;

[0017] Figure 5 A schematic diagram of another storage state of the hash table provided for this application;

[0018] Figure 6 A schematic diagram of the database expansion provided for this application;

[0019] Figure 7 A flowchart of another embodiment of the method for processing hash connections provided by the present application;

[0020] Figure 8 A flowchart of another embodiment of the method for processing hash connections provided by the present application;

[0021] Fig. 9 A schematic diagram of the structure of a hash connection processing device provided by the present application;

[0022] Fig.10 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Figure 1 This is a schematic diagram of the hash connection scenario used in this application, such as Figure 1 As shown in the figure, in the database, join is a very important operation, which is used to combine multiple tables according to specified conditions to generate a new result set. Join algorithm is an important part of query optimization, which directly affects the performance of database query, among which hash join is an important join algorithm. Hash join is a join algorithm between tables in the database, and a hash result set is obtained through hash operation. This join method has high efficiency when processing large amounts of data, so it has been widely used. The database can establish hash joins between different tables as needed, and the specific execution subject can be the database management system, or the device or system used to establish hash joins in the database.

[0026] When the hash connection establishment device is establishing a hash connection between different tables, first in the first stage corresponding to sequence number ①, one of the two tables to be hashed is used as Figure 1 The build table on the left and the other table as Figure 1 The detection table on the right. The build table refers to the data table selected to build the hash table in the build phase of the hash join. It is usually the smaller of the two tables to reduce the time and space required to build the hash table. The detection table refers to the data table used in the detection phase of the hash join. For each row, the hash value of the join column is calculated and the matching row is searched in the hash table. It is usually the larger of the two tables.

[0027] After determining the build table and the detection table, the hash connection establishment device establishes a hash table based on the build table, wherein a hash table is a data structure that maps data to an array (or bucket) of a fixed size through a hash function, thereby achieving fast search, insertion, and deletion operations. In a hash connection, all rows of the build table will calculate hash values ​​based on the values ​​of the connection condition columns and store them in the hash table.

[0028] Subsequently, in the second stage corresponding to sequence number ②, the hash connection establishment device scans the data in the detection table according to the hash table generated in the first stage, and finally obtains the hash connection between the construction table and the detection table according to the correspondence between the scanned data in the construction table and the detection table.

[0029] It can be seen that in Figure 1 In the process of establishing the hash table shown, hash join is a process of mapping the construction table into the hash table using a hash function, and then scanning the detection table to match data according to the hash value.

[0030] However, when the hash connection establishment device is establishing a hash connection, the hash table it builds needs to occupy the memory space of the database. If the hash table to be processed is too large, the generated hash table may not be completely stored in the database memory. At this time, the hash connection establishment device will build the table in batches, thereby building smaller hash tables in batches for separate processing.

[0031] Figure 2 The schematic diagram of a hash connection construction method in the prior art is shown, wherein the hash connection establishment device first divides the construction table into different batches in the first stage corresponding to the sequence number ①, recorded as batch 0, batch 2, ... batch n-1. Subsequently, a hash table corresponding to the data of the construction table of each batch is generated, and the hash value corresponding to each data in the construction table is calculated, and each data and its corresponding hash value are stored together in the hash table corresponding to the data. Finally, the hash connection establishment device scans the hash table in the memory, so that in the second stage corresponding to the sequence number ②, only one batch in the memory is processed at a time, thereby reducing the use of memory.

[0032] For example, in Figure 2 In the example shown, the hash connection establishment device sequentially obtains the data of the build table, obtains the batch number after calculating the hash value, stores the value of the build table and the hash value together in the corresponding hash table in the memory, and stores the group of other batch numbers in the build table in the temporary file established for each batch on the disk. Subsequently, the database scans the detection table, calculates the hash value and obtains the batch number. When the batch number is equal to the batch number currently in the memory, it is directly scanned in the memory for connection. When the batch number is not equal to the batch number currently in the memory, the data of the detection table is stored in the temporary file established for each batch on the hard disk. Subsequently, the hash connection establishment device reads the temporary file of the next batch from the disk into the memory for scanning and processing to generate a hash table, and reads the temporary file of the corresponding batch in the detection table into the memory for scanning to determine the corresponding relationship between the build table and the detection table. The batches of the build table and the detection table are repeatedly read in the same way as above until the entire hash connection is successfully established.

[0033] However, in Figure 2 In the prior art shown, during the process of establishing a hash connection, the hash connection establishment device will first store the established hash tables in the hard disk, and then obtain the hash tables from the hard disk, and then scan the detection table according to the hash tables to establish the hash connection. As a result, the hash connection establishment device needs to interact with the disk multiple times to obtain different batches of hash tables during the process of establishing the hash connection, thereby increasing the amount of I / O data interaction between the hash connection establishment device and the disk, thereby reducing the performance of the database.

[0034] In addition, when the hash connection establishing device is in the process of establishing a hash connection, if the database memory is expanded, the data read into the memory will be written again to the expanded batch file, thereby further increasing the amount of data interaction between the hash connection establishing device and the disk, further reducing database performance.

[0035] As database hardware develops, the computing power of processors is getting higher and higher, while the development of hard disks is slightly inferior and gradually cannot match the speed of processors. Therefore, reducing the number of disk reads and writes is becoming more and more important. How to reduce the amount of data interaction between the database and the disk during the process of establishing a hash connection has become a technical problem that needs to be solved in this field.

[0036] Regarding the above Figure 2 In order to solve the technical problems existing in the prior art shown in , the present application provides a hash connection processing method to reduce the amount of data interaction between the database and the disk in the process of establishing a hash connection. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0037] Figure 3 A flowchart of an embodiment of a method for processing hash connections provided by the present application is shown in FIG. Figure 3 The method shown can be applied to Figure 1 In the scenario of establishing a hash connection shown in FIG. 1 , the hash connection is established by the establishment device in the database for establishing a hash connection. Specifically, Figure 3 The hash join processing methods shown include:

[0038] S101: The database obtains a construction table and a detection table of a hash connection.

[0039] Specifically, please refer to Figure 1 In the hash connection scenario shown, when establishing a hash connection between two tables, the hash connection establishing device uses one table as a building table and the other table as a detection table for subsequent calculations.

[0040] S102: When the memory required by the entire hash table of the construction table is greater than a first preset value, the hash connection establishment device generates a plurality of hash tables corresponding to the plurality of batches of the construction table in the memory.

[0041] Specifically, in this embodiment, when the database constructs a hash table, since the memory required for constructing the hash table corresponding to the table is large, the entire hash table may not be stored in the memory, so the hash table is processed in batches, but the multiple hash tables obtained after batch processing are still preferentially stored in the memory, so the database's memory simultaneously stores multiple hash tables corresponding to multiple batches. In the embodiment of the present application, whether the hash table can be stored in the memory is determined based on whether the memory is greater than the first preset value. If the memory is greater than the first preset value, it means that the hash table cannot be stored in the memory and the hash table needs to be processed in batches. If the memory is not greater than the first preset value, it means that the hash table can be stored in the memory and the hash table does not need to be processed in batches. The embodiment of the present application does not limit the specific setting of the first preset value, which can be preset or configured by the database according to different scenarios.

[0042] For example, Figure 4 A schematic diagram of a storage state of a hash table provided for the present application, wherein it is assumed that the hash connection establishment device divides the construction table into n batches, recorded as batch 0, batch 1... batch n-1, during the process of building the hash connection, and the hash table corresponding to each batch is recorded as hash table-0, hash table-1... hash table-n-1. In an embodiment of the present application, the hash connection establishment device preferentially stores the hash table-0, hash table-1... hash table-n-1, a total of n hash tables, calculated in S102, in the memory, and does not store them directly on the disk.

[0043] S103: The hash connection establishment device calculates the hash value corresponding to each data in the construction table, and stores the hash value corresponding to each data in the hash table corresponding to each data. At the same time, in the process of storing the data in the hash table, at least one hash table among the multiple hash tables is stored from the memory to the disk according to the remaining memory.

[0044] The embodiment of the present application does not limit the batching method of data. The hash connection establishment device can determine each corresponding hash table in the process of building the hash table according to the numerical calculation and modulus operation of each data in the construction table. For example, assuming that the construction table is divided into 8 batches, the database performs a modulus 8 operation on the number of rows of the data in the first row of the construction table to obtain 1, indicating that the data corresponds to batch 1, and calculates the hash value of the data, and finally stores the data of the first row together with the hash value in the hash table -1 corresponding to batch 1. Perform a modulus 8 operation on the number of rows of the data in the first row to obtain 2, indicating that the data corresponds to batch 2, and calculates the hash value of the data, and finally stores the data of the first row together with the hash value in the hash table -2 corresponding to batch 2, and so on.

[0045] S103: The hash connection establishment device scans each data in the detection table, and during the scanning process, connects each hash table in the multiple hash tables determined in S102 according to the hash value of each data to obtain a hash connection.

[0046] Reference Figure 1 and Figure 4 When the memory of the database is sufficient, the n hash tables established by the hash connection establishment device in the first stage can be stored in the memory. Then in the second stage, the database executes step S103 to scan each data in the detection table, which can be specifically calculated by calculating the hash value of each data and comparing it with multiple hash tables, thereby connecting the corresponding hash tables, and finally completing the scanning of all data in the detection table to obtain the hash connection between the construction table and the detection table.

[0047] In summary, in the hash connection processing method provided in this embodiment, the hash connection establishment device generates multiple hash tables corresponding to multiple batches of the construction table in the memory during the process of establishing the hash connection, calculates the hash value corresponding to each data in the construction table, and in the process of storing the hash value corresponding to each data in the hash table corresponding to each data in the memory, according to the remaining memory, stores at least one hash table from the multiple hash tables from the memory to the disk, and finally scans the detection table to obtain the hash connection. Compared with the prior art in which the hash connection establishment device only stores one batch of established hash tables in the memory, the hash tables of different batches established by the hash connection establishment device are all stored in the memory, so that the hash connection establishment device reduces the number of interactions with the disk during the process of establishing the hash connection, thereby reducing the amount of data interaction between the hash connection establishment device and the disk, thereby improving the performance of the database.

[0048] In the above embodiment, the hash connection establishing device preferentially stores multiple hash tables of different batches in the construction table in the memory. At this time, the memory of the database can be used to store multiple hash tables.

[0049] In some cases, if the database memory is not enough to store multiple hash tables of different batches, the hash connection establishment device still keeps the hash table in memory as much as possible, but stores some hash tables that must be moved out of memory on disk.

[0050] Therefore, in one embodiment of the present application, specifically in the above-mentioned S103 process, in the process of calculating the hash value of the data in the build table and storing it in the corresponding hash table, if the hash value of the current data in the build table currently calculated cannot be stored in the memory, it means that the remaining memory in the memory is insufficient and more hash values ​​cannot be stored, then at least one hash table among the multiple hash tables is stored from the memory to the disk.

[0051] For example, Figure 5 A schematic diagram of another storage state of a hash table provided for the present application, wherein hash table-0, hash table-1...hash table-n-1 generated by a hash connection establishing device in the process of building a hash connection are all stored in memory. Subsequently, when the hash value of the current data in the currently calculated building table cannot be stored in the memory, the hash connection establishing device can determine to store a total of m hash tables, hash table-m1...hash table-mm, from the n hash tables from the memory to the disk, and delete the m hash tables stored in the memory to release the memory corresponding to the m hash tables, wherein m is an integer greater than 0 and less than n.

[0052] In this embodiment, the hash connection establishment device stores multiple hash tables corresponding to different batches of the construction table in the memory as much as possible during the process of establishing the hash connection. Only when the hash value of the current data cannot be stored in the memory, some hash tables are stored in the disk, but not all hash tables are stored in the disk, so that the hash connection establishment device can also reduce the number of interactions with the disk to a certain extent during the process of establishing the hash connection, thereby reducing the amount of data interaction between the hash connection establishment device and the disk, thereby improving the performance of the database. In this embodiment, when the database can be used to store multiple hash tables, the larger the memory, the smaller the amount of data interaction between the hash connection establishment device and the disk, and the shorter the time required for the hash connection establishment device to establish the hash connection, so that the hash connection establishment device has a stronger performance in establishing the hash connection. At the same time, this embodiment can also make more memory in the database used to establish hash connections, and also improve the utilization of memory.

[0053] In one embodiment, the present application also provides a method for determining at least one hash table, wherein, when the calculated hash value of the current data in the construction table cannot be stored in the memory, the hash connection establishing device can determine at least one hash table from multiple hash tables based on the remaining memory and the size of each hash table in the multiple hash tables, so as to store the determined at least one hash table in the disk.

[0054] Specifically, the hash connection establishment device obtains the current scan progress P of the build table according to the ratio of the number of rows with calculated hash values ​​to the total number of rows in the build table, and determines the memory Q currently actually occupied by multiple hash tables, thereby obtaining the current memory value F to be released through F=Q*(1-P). Then, according to the memory value F to be released, the hash connection establishment device stores at least one hash table among the multiple hash tables stored in the memory from the memory to the disk.

[0055] In one embodiment, the hash connection establishing device specifically sorts the hash tables according to the actual occupied memory size, and stores at least one hash table among the multiple hash tables whose occupied memory is smaller than the to-be-released memory value F in order from large to small to the disk.

[0056] If at least one hash table with memory less than the memory value F to be released is written to the disk, and the released memory is still less than F, then at least one hash table with memory greater than the memory value F to be released is stored in the disk in ascending order until the released memory is less than the memory value F to be released.

[0057] Further, refer to Figure 5 As shown in the storage state, since the hash connection establishing device in this embodiment stores part of the hash table in the memory and the other part of the hash table in the hard disk, in the process of establishing the hash connection, the hash connection establishing device calculates the hash value corresponding to each data in the construction table in the first stage, and then stores the calculated hash value of the data in the corresponding hash table in the memory or hard disk according to whether the hash table corresponding to the data currently exists in the memory or hard disk.

[0058] In the second stage, when the hash connection is obtained, the hash connection is first scanned according to the data of the multiple hash tables and the corresponding batch of detection tables stored in the memory, referring to Figure 5 The order in the example is hash table-0, hash table-1, ... hash table-n-1. Then the hash table stored in the memory is deleted, and the data of hash table-m1 ... hash table-mm stored in the disk and the corresponding batch of detection tables are stored in the memory. The hash connection establishment device scans hash table-m1 ... hash table-mm stored in the memory again, and finally obtains the complete hash connection between the construction table and the detection table.

[0059] In the above embodiment, each hash table can be stored in the memory. In this case, the hash connection establishment device does not need to expand the capacity of the hash table in the memory. In some cases, the hash connection establishment device needs to expand the memory to store the hash table.

[0060] Specifically, Figure 6 The schematic diagram of the database expansion provided for this application is as follows: Figure 6 As shown, assuming that hash table-0, hash table-1, and hash table-2 are stored in the memory, and hash table-3 is stored in the disk. The hash connection establishment device can expand the multiple hash tables in the memory and create new memories to store more hash values. For example, new hash tables-4, hash tables-5, and hash tables-6 are created in the memory, corresponding to hash table-0, hash table-1, and hash table-2, respectively, to achieve expansion.

[0061] At the same time, when the database is expanded, the hash table corresponding to the batch stored in the memory is directly expanded, while the hash table corresponding to the batch stored on the disk is not processed, and the batch number on the disk and the total batch number during the expansion are recorded and written to the disk. Figure 6 In the example shown, the database does not process hash table-3 on disk, and directly writes the record of hash table-7 to disk.

[0062] Subsequently, based on Figure 6 As shown in the storage state after expansion, in the detection phase, the hash connection establishment device traverses the detection table, calculates the hash value and the batch number, and directly connects if the hash table corresponding to the batch number is stored in the memory; otherwise, it is written to the temporary file used to record the corresponding batch number of the detection table. When the detection table traversal is completed, the hash connection establishment device releases the hash table in the memory, reads the hash table batch in the file and the associated expanded batch number into the memory, and does not read the associated batch number first if the memory cannot hold a group. Read the exploration table data corresponding to the batch number in the memory and connect. After the connection is completed, read the next group of hash table batches until the connection is completed. For example, after hash table-3 is read, try to read hash table-7. If the memory can hold two batches, read the detection table of hash table-3 and hash table-7 for connection, otherwise only read hash table-3, and write the data in hash table-3 that needs to be expanded to hash table-7 into hash table-7, and read hash table-7 after hash table-3 is connected.

[0063] Figure 7 A flowchart of another embodiment of the processing method for hash connection provided by the present application shows a specific implementation method of establishing multiple hash tables in the present application, wherein, when the hash connection establishment device starts to build a hash connection, the construction table is scanned, and each row of data in the construction table is calculated respectively, the hash value of the row of data is calculated, and the batch number corresponding to the row is determined. Then it is determined whether the batch to which the row belongs is in the memory. If so, the hash value of the row of data is stored in the corresponding hash table in the memory. If not, the hash value of the row of data is stored in the corresponding hash table in the disk. Then it is determined whether the hash table is built. If so, it ends. If not, it is determined whether the memory is sufficient. If sufficient, the next row of data of the construction table is read. If not, when there is not only one batch of hash tables left in the memory, the progress P of building the hash table is calculated to calculate the memory size F to be released, and the hash table in the memory is released to the disk according to the memory size F to be released, and the next row of data is read after all batches less than F are released. If the memory is still insufficient after all batches less than F are released, the hash table of the smallest batch greater than F is continued to be released.

[0064] Figure 8A flow chart of another embodiment of the processing method of hash connection provided by the present application shows a specific implementation method of the present application in the case of expansion when establishing a hash table, wherein, when the hash connection establishment device establishes multiple hash tables, if expansion occurs, the hash table in the disk is written to the disk and the written batch number and the total number of matches are recorded. When expanding, the batch number that has been written and the total batch number at the time of expansion are recorded and written to the disk. Then, in the detection phase, the hash connection establishment device determines whether all the detection tables have been traversed. If so, the hash table batch and the associated batch are read from the disk, and the expansion batch that cannot be placed in the memory is written to the corresponding disk, and then the detection file corresponding to the batch number is read and connected. If not, the hash value and the batch number are calculated to determine whether the batch number is in the memory. If so, it is connected, otherwise it is written to the temporary file of the corresponding batch number.

[0065] In the aforementioned embodiments of the present application, the processing method of hash connection provided by the embodiments of the present application is introduced. In order to realize the functions in the processing method of hash connection provided by the embodiments of the present application, the establishment device of hash connection as the execution subject can be realized by hardware structure and / or software module, for example, in the form of hardware structure, software module, or hardware structure plus software module to realize the above functions. Whether one of the above functions is executed in the form of hardware structure, software module, or hardware structure plus software module depends on the specific application and design constraints of the technical solution.

[0066] For example, Fig. 9 A schematic diagram of a hash connection processing device provided in this application is shown in FIG. Fig. 9 The device shown can be used to execute the hash connection processing method provided in any embodiment of the present application. In one embodiment, Fig. 9 The hash connection processing device 1000 shown includes: an acquisition module 1001, a construction module 1002 and a detection module 1003. The acquisition module 1001 is used to acquire a construction table and a detection table for hash connections; the construction module 1002 is used to generate multiple hash tables corresponding to multiple batches of the construction table in the memory when the memory required by the hash table of the construction table is greater than a first preset value, calculate the hash value of each data in the construction table, and in the process of storing the hash value of each data in the hash table corresponding to each data in the memory, store at least one hash table of the multiple hash tables from the memory to the disk according to the remaining memory; the detection module 1004 is used to scan each data in the detection table, and in the scanning process, connect the corresponding hash tables in the multiple hash tables according to the hash value of each data to obtain the hash connection.

[0067] The specific implementation method and principle of the above-mentioned hash connection processing device refer to the description of the above-mentioned hash connection processing method, which will not be repeated here.

[0068] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0069] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0070] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0071] For example, Fig.10 A schematic diagram of the structure of an electronic device provided in this application, such as Fig.10 The device shown can be used to execute the hash connection processing method provided in any embodiment of the present application. In one embodiment, Fig.10 The electronic device 2000 shown includes one or more processors 2001 and a memory 2002; wherein the memory 2002 is used to store computer executable instructions, and the processor 2001 can execute the computer executable instructions stored in the memory 2002. When the computer executable instructions are executed by the processor 2001, the processor 2001 implements any hash connection processing method in the aforementioned embodiments of the present application. In one embodiment, Figure 8 The electronic device 2000 shown further includes a communication interface 2003 , wherein the processor 2001 can communicate with other devices via the communication interface 2003 .

[0072] The present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, they can be used to implement any hash connection processing method in the aforementioned embodiments of the present application.

[0073] An embodiment of the present application also provides a chip for executing instructions, wherein the chip is used to execute any of the aforementioned hash connection processing methods of the present application.

[0074] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any of the aforementioned hash connection processing methods of the present application.

[0075] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing hash connections, characterized in that: include: Get the build table and probe table of hash connection; When the memory required for the hash table of the construction table is greater than a first preset value, generating multiple hash tables corresponding to multiple batches of the construction table in the memory; Calculating a hash value of each data in the construction table, and in the process of storing the hash value of each data in a hash table corresponding to each data in the memory, storing at least one hash table of the plurality of hash tables from the memory to the disk according to the remaining memory; Scan each data in the detection table, and connect the corresponding hash tables in the multiple hash tables according to the hash value of each data during the scanning process to obtain the hash connection.

2. The method according to claim 1, characterized in that According to the remaining memory, storing at least one hash table among the plurality of hash tables from the memory to the disk includes: When the calculated hash value of the current data in the construction table cannot be stored in the memory, at least one hash table among the multiple hash tables is stored from the memory to the disk.

3. The method according to claim 2, characterized in that The step of storing at least one of the plurality of hash tables from the memory to the disk comprises: Determine the memory value to be released according to the ratio of the number of rows scanned to the total number of rows in the build table and the memory occupied by the multiple hash tables; According to the memory value to be released, at least one hash table among the multiple hash tables is stored from the memory to the disk.

4. The method according to claim 3, characterized in that According to the memory value to be released, storing at least one hash table of the multiple hash tables from the memory to the disk includes: Store at least one hash table among the multiple hash tables that is smaller than the memory value to be released from the memory to the disk in descending order; Alternatively, after storing at least one hash table among the multiple hash tables that is smaller than the memory value to be released from the memory to the disk in order from large to small, if the released memory is less than the memory value to be released, store at least one hash table among the multiple hash tables that is larger than the memory value to be released from the memory to the disk in order from small to large until the released memory is less than the memory value to be released.

5. The method according to claim 4, characterized in that The step of scanning each data in the detection table and connecting corresponding hash tables in the plurality of hash tables according to the hash value of each data in the scanning process to obtain the hash connection includes: After scanning the data in each hash table stored in the memory in turn, the hash table stored in the memory is deleted, and the data of the hash table stored in the disk and the corresponding batch of detection tables are stored in the memory, and the data in each hash table stored in the memory is scanned again. During the scanning process, the corresponding hash tables in the multiple hash tables are connected according to the hash value of each data to obtain the hash connection.

6. The method according to any one of claims 1 to 5, characterized in that: New memories are created for each of the multiple hash tables stored in the memory to achieve capacity expansion.

7. A hash connection processing device, characterized in that: include: An acquisition module is used to obtain a construction table and a detection table of a hash connection; A construction module, configured to generate a plurality of hash tables corresponding to a plurality of batches of the construction table in the memory when the memory required for the hash table of the construction table is greater than a first preset value, calculate a hash value of each data in the construction table, and in a process of storing the hash value of each data in the hash table corresponding to each data in the memory, store at least one hash table of the plurality of hash tables from the memory to a disk according to the remaining memory; The detection module is used to scan each data in the detection table, and connect the corresponding hash tables in the multiple hash tables according to the hash value of each data during the scanning process to obtain the hash connection.

8. An electronic device, characterized in that: include: Memory and processor; The memory stores computer executable instructions; The processor executes the computer executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when the computer executable instructions are executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed.

Images