Hash table creation apparatus, method, and program

Abstract

This invention provides a hash table creation device that can reduce the overhead when creating a hash table composed of data that meets certain conditions using parallel processing. [Solution] The mask condition verification unit obtains a row index, which is identification information for rows that match the mask condition (a condition for identifying data in the table), from the rows of the assigned table for each thread to which rows separated from the table are assigned. The hash table creation unit obtains row values ​​for each thread that match the key conditions set for each thread, for the table keys corresponding to all obtained row indexes, and inserts the row data corresponding to the table keys into the hash table.

Description

Technical Field

[0001] The present disclosure relates to a hash table creation device, a hash table creation method, and a hash table creation program for creating a hash table.

Background Art

[0002] A hash table is a data structure that stores a plurality of key-value pairs and is used to reference the value corresponding to a key. The hash table is also used when implementing table data joining processing (Join) and grouping processing (Groupby).

[0003] For example, Patent Document 1 describes a method for creating a hash table. In the method described in Patent Document 1, divided data obtained by dividing a data group into a plurality of intervals of a predetermined number of bits is generated, and when registering the data group, a hash table in which a hash value of a predetermined number of bits and NULL as a pointer are associated with each other is created, and a tree having each hash table as a node is constructed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above-described joining process, it is often the case that the processing target is narrowed down to data that satisfies the conditions. For example, in terms of the appearance of the program, before and after the joining process, a process of filtering data according to conditions is performed. This process can be realized by creating a conditional hash table.

[0006] One way to quickly create a hash table composed of data that meets certain conditions is through parallel processing using multiple threads. However, while parallel processing can be fast, it also introduces overhead problems.

[0007] Therefore, the purpose of this disclosure is to provide a hash table creation device, a hash table creation method, and a hash table creation program that can reduce the overhead when creating a hash table composed of data that satisfies certain conditions using parallel processing. [Means for solving the problem]

[0008] The hash table creation device according to this disclosure is characterized by comprising: a mask condition verification unit that, for each thread to which rows divided from a table are assigned, obtains a row index, which is identification information of a row that matches a mask condition, which is a condition for identifying data in the table, from among the rows of the assigned table; and a hash table creation unit that, for each thread, obtains the row value that matches the key condition set for each thread for the table key corresponding to all the obtained row indexes, and inserts the row data corresponding to the table key into the hash table.

[0009] The hash table creation method according to this disclosure is characterized in that, for each thread to which rows separated from a table are assigned, a row index, which is identification information of a row that matches a mask condition, which is a condition for identifying data in the table, is obtained from the rows of the assigned table, and for each thread, the row value that matches the key condition set for that thread is obtained for the table key corresponding to all the obtained row indexes, and the row data corresponding to the table key is inserted into the hash table.

[0010] The hash table creation program according to this disclosure is characterized in that it causes a computer to perform a mask condition verification process for each thread to which rows divided from a table are assigned, in which it obtains row indexes, which are identification information for rows that match the mask conditions, which are conditions for identifying data in the table, from among the rows of the assigned table, and a hash table creation process for each thread to obtain row values ​​that match the key conditions set for each thread for the table keys corresponding to all the obtained row indexes, and insert the row data corresponding to the table keys into the hash table. [Effects of the Invention]

[0011] According to this disclosure, the overhead of creating a hash table composed of data that meets certain conditions using parallel processing can be reduced. [Brief explanation of the drawing]

[0012] [Figure 1] This block diagram shows an example configuration of a hash table creation device according to the present disclosure. [Figure 2] This is an explanatory diagram illustrating an example of how to create a conditional hash table. [Figure 3] This is an explanatory diagram illustrating an example of how to create a hash table. [Figure 4] This is an explanatory diagram illustrating an example of how to create a hash table. [Figure 5] This is an explanatory diagram illustrating an example of how to create a hash table using parallel processing. [Figure 6] This is an explanatory diagram illustrating an example of how to create a conditional hash table using parallel processing. [Figure 7] This is an explanatory diagram illustrating an example of parallel processing. [Figure 8] This is an explanatory diagram illustrating an example of the process of dividing data among different threads. [Figure 9] This is an explanatory diagram illustrating an example of the process of creating a hash table. [Figure 10]It is an explanatory diagram showing an example of a process for creating a hash table. [Figure 11] It is a flowchart showing an operation example of a hash table creation device. [Figure 12] It is a block diagram showing an overview of the hash table creation device of the present disclosure. [Figure 13] It is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] FIG. 1 is a block diagram showing a configuration example of a hash table creation device of the present disclosure. The hash table creation device 100 of the present embodiment includes a storage unit 10, an input unit 20, and a parallel task execution unit 30.

[0015] First, a method for creating a hash table (hereinafter, may also be referred to as a conditional hash table) composed of data that satisfies conditions will be described. A hash table is a data structure that stores a plurality of pairs of keys and values and is used to refer to the value corresponding to a key (key). FIG. 2 is an explanatory diagram showing an example of a method for creating a conditional hash table.

[0016] For the table T11 as illustrated in FIG. 2, first, a mask condition confirmation step is executed. In the present disclosure, a mask condition means a condition for specifying data in a specified table. As an example of a mask condition, a condition specified by a where clause of SQL (Structured Query Language) can be mentioned.

[0017] In the mask condition verification step, all data in table T11 is processed from top to bottom. As a result, the data in table T11 that satisfies the mask condition is identified as a mask value T (True), and the mask values ​​are obtained as shown in table T12. Next, a key retrieval step is performed to obtain the key for which the mask value is T. Then, a hash table insertion step is performed to insert a value into hash table T13 for each retrieved key.

[0018] Next, we will explain how to create a hash table using parallel processing (parallel programming). When creating a single hash table using parallel processing, it is common practice to first partition the data contained in the target table, then have each thread create a separate hash table in parallel for the partitioned data it targets, and finally join the created hash tables. Methods for partitioning data include partitioning the data evenly or partitioning the data according to its value.

[0019] Figure 3 is an explanatory diagram illustrating an example of how to create a hash table by equally dividing the data. In the example shown in Figure 3, first, the 12 data entries in table T11 are divided into two sets of 6 data entries each (tables T21 and T22). Then, two hash tables T23 and T24 are created independently from each set of data, and these two hash tables are joined to create the final hash table T25.

[0020] Figure 4 is an explanatory diagram illustrating an example of how to create a hash table by partitioning data according to its values. Here, we define the remainder when the value (val) for a key is divided by a predetermined divisor. In the example shown in Figure 4, we assume that A mod 2 = 0, B mod 2 = 1, C mod 2 = 0, and D mod 2 = 1.

[0021] Subsequently, a key condition verification step is performed to confirm the key conditions (hereinafter simply referred to as key conditions), and hash table T31 is created with only data where "mod 2 is 0" added, and hash table T32 is created with only data where "mod 2 is 1" added. Note that in the example shown in Figure 4, a hash table is created for each key according to its value, so a join process is unnecessary.

[0022] In the example shown in Figure 4, we illustrate a method of splitting data according to the remainder (mod). For example, if the range of keys is known, it is also possible to split the data according to the range of keys.

[0023] Figure 5 is an explanatory diagram illustrating an example of how to create a hash table using parallel processing. The method illustrated in Figure 5 is a combination of the methods illustrated in Figures 3 and 4. In other words, different key conditions are set for each thread in order to execute parallel tasks. Each thread processes all data from top to bottom according to the set key conditions (specifically, the key retrieval step, the key condition verification step, and the hash table insertion step).

[0024] Next, we will explain how to create a conditional hash table using parallel processing. Figure 6 is an explanatory diagram illustrating an example of how to create a conditional hash table using parallel processing. Compared to the process illustrated in Figure 5, the process of checking the mask conditions (mask condition check step) is performed for all data in the table.

[0025] Here, the inventors noticed that when executing parallel tasks, the same mask condition verification process is generally performed by all threads. That is, in a naive method of creating a conditional hash table in parallel, all threads perform the same mask condition verification step for all data. The inventors realized that having all threads perform the same mask condition verification step results in a lot of processing overhead.

[0026] In other words, when creating conditional hash tables in parallel, a naive approach would require each thread to check the mask condition and the key condition for each thread for all data. This means that all threads perform the mask condition check for all data, resulting in unnecessary processing. Therefore, this disclosure describes a method to reduce overhead by streamlining the processing.

[0027] The storage unit 10 stores various information used by the hash table creation device 100 of this embodiment for processing. The storage unit 10 also stores the target table when creating a hash table, and the created hash table. The storage unit 10 is implemented, for example, by a magnetic disk.

[0028] The input unit 20 accepts input for instructions to create a hash table. Specifically, the input unit 20 accepts input for specifying the table for which the hash table is to be created, and for specifying the conditions (i.e., mask conditions) for identifying the data within the specified table. The input unit 20 may also accept input for the table for which the hash table is to be created.

[0029] The input unit 20 may, for example, receive input of a hash table creation instruction from another device (not shown) that operates in cooperation with the hash table creation device 100 of this disclosure, or it may receive input of a creation instruction from a user.

[0030] The parallel task execution unit 30 executes the process of creating a single hash table in parallel using multiple threads. The number of threads executing the parallel processing is predetermined.

[0031] Figure 7 is an explanatory diagram showing an example of parallel processing performed by the parallel task execution unit 30. The parallel processing performed by the parallel task execution unit 30 to create a hash table can be broadly divided into two steps: the "data parallel mask condition confirmation step" and the "key condition parallel hash table creation step".

[0032] In the data parallel mask condition verification step, processing is performed for each thread to which rows separated from the table are assigned. On the other hand, in the key condition parallel hash table creation step, processing is performed for each thread on all row data obtained as a result of the data parallel mask condition verification step. The processing by the parallel task execution unit 30 will be described in detail below.

[0033] First, the parallel task execution unit 30 obtains, for each thread to which a row divided from the table is assigned, identification information (hereinafter referred to as a row index) of a row that matches the conditions for identifying data in the table (i.e., a mask condition) from the rows of the assigned table. The row index is, for example, a row number or a pointer to a row.

[0034] The parallel task execution unit 30 may also divide the rows contained in the table and allocate them to each thread. The parallel task execution unit 30 may then obtain row indexes that match the mask condition from among the rows of the table allocated to the threads.

[0035] In this case, the method by which the parallel task execution unit 30 divides the data is arbitrary. It is preferable that the parallel task execution unit 30 divides the data uniformly so that processing is not unevenly distributed among threads. For example, as shown in Figures 3 and 4 above, the parallel task execution unit 30 may divide the data evenly, or it may divide the data according to a value or a range of values.

[0036] By doing this, the number of processes that check the mask condition can be reduced per thread compared to the process illustrated in Figure 6, thereby reducing the overhead when creating a hash table in parallel.

[0037] Figure 8 is an explanatory diagram illustrating an example of the process of dividing data for each thread. The parallel task execution unit 30 divides the data from table T11 and table T12, which shows the mask values, equally (six rows each) from the beginning. Then, for each thread, the parallel task execution unit 30 obtains row index IX1 and row index IX2 from the rows of the assigned table that match the mask condition. As a result, row index IX, which is the sum of the two row indexes, is obtained.

[0038] Next, the parallel task execution unit 30 retrieves rows corresponding to the retrieved row indexes from the target table for each thread. For each thread, the parallel task execution unit 30 retrieves row values ​​that match the key conditions set for each thread for the target table keys corresponding to all retrieved row indexes, and inserts the data (hereinafter sometimes referred to as row data) corresponding to the table keys into the hash table. Through these processes, a hash table is created.

[0039] The key conditions set for each thread are defined by multiple key conditions that allow for the exhaustive and exclusive retrieval of rows contained in the table. An example of a key condition is the condition shown as modulo (mod) in Figure 4 above.

[0040] Here, two methods are considered for retrieving rows from the target table that correspond to the retrieved row indexes. The first method is to target all retrieved row indexes and retrieve the corresponding rows from the table in each thread. The second method is to first target only the row indexes retrieved by each thread, retrieve the corresponding rows from the table in parallel for each thread, and then integrate the results.

[0041] The following will explain each method in detail. Figure 9 is an explanatory diagram showing an example of the process of creating a hash table using the first method. In the example shown in Figure 9, the modulo (mod) of the key is used as the key condition and is set for each thread.

[0042] First, the parallel task execution unit 30 retrieves rows from table T11 that correspond to the retrieved row index IX. Here, the values ​​of each column of index IX are denoted as idx. Next, the parallel task execution unit 30 retrieves rows whose keys match the key conditions set for the thread.

[0043] For example, as illustrated in Figure 9, if the thread has "Key mod 2==0" set as the key condition, the parallel task execution unit 30 retrieves the row identified by idx=0 that matches the key condition. The same applies to other rows. The parallel task execution unit 30 then inserts the row data, which associates the table key (key=A) with the retrieved row value (10), into the hash table T41. The same process is performed for threads with "Key mod 2==1" set, and the hash table T42 is generated.

[0044] Next, the second method described above will be explained. Figure 10 is an explanatory diagram showing an example of the process of creating a hash table using the second method. In the example shown in Figure 10, the modulo (mod) of the key is used as the key condition and is set for each thread.

[0045] First, the parallel task execution unit 30 targets row indexes IX1 and IX2 obtained by each thread and retrieves rows corresponding to these row indexes from table T11 in parallel for each thread. As a result, sets of rows T51 and T52 are obtained. The parallel task execution unit 30 then integrates these retrieval results to obtain set of rows T53.

[0046] Subsequently, similar to the method illustrated in Figure 9, the parallel task execution unit 30 retrieves rows whose keys match the key conditions set for the thread, and inserts the row data, which associates the table keys with the values ​​of the retrieved rows, into hash tables T41 and T42.

[0047] Here, we will explain the processing load (cost) of the first and second methods. In the first method, each thread references the entire index and table, so the number of memory reads (Read) for the index and table (specifically, Key) is equal to the number of threads, and the number of value (Val) reads is 1.

[0048] On the other hand, the second method first retrieves a set of rows from the index. Therefore, the number of memory reads and writes when retrieving the set of rows is one read for the index, and one read / write each for the table (specifically, the Key) and the value (Val). Furthermore, since the index is not read when creating the hash table, the number of memory reads for the table (specifically, the Key) is equal to the number of threads, and the number of value (Val) reads is one.

[0049] As shown above, the amount of read and write processing required differs depending on the method. This processing amount changes depending on the characteristics of the data, or in other words, the amount of data. Specifically, if there is a large amount of data to be processed in the table, the first method is effective. On the other hand, if there is a small amount of data to be processed in the table, the second method is effective. The number of data items to be processed changes depending on the masking conditions.

[0050] Therefore, the parallel task execution unit 30 may calculate the ratio of the number of rows that match the mask condition to the total number of rows in the table. If the ratio is greater than a predetermined threshold, the parallel task execution unit 30 may select the first method, and if the ratio is less than or equal to the predetermined threshold, it may select the second method. This threshold is predetermined by an administrator or the like based on past statistics, etc.

[0051] The input unit 20 and the parallel task execution unit 30 are implemented by a computer processor (e.g., a CPU (Central Processing Unit)) that operates according to a program (a hash table creation program). For example, the program may be stored in the storage unit 10 of the hash table creation device 100, and the processor may read the program and operate as the input unit 20 and the parallel task execution unit 30 according to the program.

[0052] Furthermore, each function of the hash table creation device 100 may be provided in SaaS (Software as a Service) format. Also, the input unit 20 and the parallel task execution unit 30 may each be implemented with dedicated hardware.

[0053] Furthermore, some or all of the components of each device may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be comprised of a single chip or multiple chips connected via a bus. Some or all of the components of each device may be implemented by a combination of the aforementioned circuits, etc., and programs.

[0054] Furthermore, if some or all of the components of the hash table creation device 100 are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system.

[0055] Next, an example of the operation of the hash table creation device 100 of this embodiment will be described. Figure 11 is a flowchart showing an example of the operation of the hash table creation device 100 of this embodiment.

[0056] The parallel task execution unit 30 retrieves, for each thread, the row index of the row that matches the mask condition from the rows of the table that have been assigned (step S11). The parallel task execution unit 30 retrieves, for each thread, the row value that matches the key condition set for each thread, for the table key corresponding to all the retrieved row indexes (step S12). Then, the parallel task execution unit 30 inserts the row data, which is the obtained value and the table key, into the hash table (step S13).

[0057] As described above, according to this embodiment, the parallel task execution unit 30 obtains, for each thread, row indexes of rows that match the mask condition from the rows of the table that have been assigned, and for each thread, obtains row values ​​that match the key conditions set for each thread for the table keys corresponding to all the obtained row indexes. Then, the parallel task execution unit 30 inserts the row data, which is the obtained value and the table key, into the hash table. Therefore, the overhead of creating a hash table consisting of data that satisfies the conditions in parallel processing can be reduced.

[0058] In other words, instead of all threads having to perform the mask condition check process for all data, the mask condition check is performed in parallel by data partitioning, thus reducing inefficiencies in parallel computation.

[0059] Next, an overview of the present disclosure will be described. Figure 12 is a block diagram showing an overview of the hash table creation device of the present disclosure. The hash table creation device 80 (for example, the hash table creation device 100) according to the present disclosure includes a mask condition verification unit 81 (for example, a parallel task execution unit 30) that, for each thread to which rows separated from the table are assigned, obtains a row index, which is identification information of a row that matches a mask condition, which is a condition for identifying data in the table, from among the rows of the assigned table; and a hash table creation unit 82 (for example, a parallel task execution unit 30) that, for each thread, obtains the row value (for example, val) that matches the key condition set for each thread for the table key corresponding to all the obtained row indexes, and inserts the row data (for example, Key+Val) corresponding to the table key into the hash table.

[0060] Such a configuration reduces the overhead involved in creating a hash table composed of data that satisfies certain conditions using parallel processing.

[0061] Alternatively, the hash table creation unit 82 may, for each thread, retrieve a row from the table corresponding to the retrieved row index, retrieve a row whose key matches the key condition set for that thread, and insert row data into the hash table that associates the table key with the value of the retrieved row.

[0062] Specifically, the hash table creation unit 82 may, for each thread, retrieve rows from the table corresponding to all retrieved row indexes (for example, using the first method described above), retrieve rows whose keys match the key conditions set for that thread, and insert row data into the hash table that associates the table key with the value of the retrieved row.

[0063] Alternatively, the hash table creation unit 82 may target the row index obtained by each thread, obtain rows corresponding to that row index from the table in parallel for each thread, integrate the results (for example, the second method described above), obtain rows whose keys match the key conditions set for the thread, and insert row data corresponding to the table key and the value of the obtained row into the hash table.

[0064] Furthermore, the hash table creation unit 82 may calculate the ratio of the number of rows that match the mask condition to the total number of rows in the table. If the ratio is greater than a predetermined threshold, it may retrieve the row corresponding to the row index from the table for each thread, targeting all retrieved row indexes. If the ratio is less than or equal to the predetermined threshold, it may retrieve the row corresponding to the row index obtained by each thread in parallel from the table for each thread, and then integrate the results.

[0065] Furthermore, the key conditions set for each thread (for example, conditions defined by the remainder) may be defined by multiple key conditions so that all rows contained in the table can be retrieved exhaustively and exclusively.

[0066] Alternatively, the mask condition verification unit 81 may divide the rows contained in the table and assign them to each thread, and for each thread, obtain the row index that matches the mask condition from among the rows of the table assigned to that thread.

[0067] The mask condition confirmation unit 81 may also assign rows from a table divided according to equality, value, or range of values ​​to each thread.

[0068] Figure 13 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. Computer 1000 comprises a processor 1001, main memory 1002, auxiliary memory 1003, and interface 1004. Computer 1000 may also be connected to a computer for running a mathematical programming solver, an annealing machine, a simulator, etc.

[0069] The hash table creation device 80 described above is implemented in the computer 1000. The operation of each processing unit described above is stored in the auxiliary storage device 1003 in the form of a program (hash table creation program). The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main memory 1002, and executes the above processing according to the program.

[0070] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-temporary tangible medium. Other examples of non-temporary tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), and semiconductor memory connected via the interface 1004. Furthermore, if this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may expand it into the main memory 1002 and execute the above processing.

[0071] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in the auxiliary storage device 1003.

[0072] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention. [Explanation of Symbols]

[0073] 10 Storage section 20 Input section 30 Parallel Task Execution Unit 100 Hash Table Generators

Claims

1. For each thread to which rows split from the table are assigned, a mask condition verification unit obtains a row index, which is identification information for rows that match the mask condition, which is a condition for identifying data within the table, from among the assigned rows of the table. The system includes a hash table creation unit that, for each thread, retrieves row values ​​that match the key conditions set for each thread for the table keys corresponding to all the retrieved row indexes, and inserts the row data corresponding to the table keys into a hash table. A hash table creation device characterized by the following features.

2. The hash table creation unit retrieves rows from the table corresponding to the retrieved row index for each thread, retrieves rows whose keys match the key conditions set for that thread, and inserts row data that associates the table key with the value of the retrieved row into the hash table. The hash table creation device according to claim 1.

3. The hash table creation unit, for each thread, retrieves rows from the table corresponding to all retrieved row indexes, selects rows whose keys match the key conditions set for that thread, and inserts the row data, which associates the table key with the value of the retrieved row, into the hash table. The hash table creation device according to claim 2.

4. The hash table creation unit targets the row index obtained by each thread, retrieves rows corresponding to that row index from the table in parallel for each thread, integrates the retrieval results, selects rows whose keys match the key conditions set for the thread, and inserts the row data, which associates the table key with the value of the retrieved row, into the hash table. The hash table creation device according to claim 2.

5. The hash table creation unit calculates the ratio of the number of rows that match the mask condition to the total number of rows in the table. If the ratio is greater than a predetermined threshold, it retrieves the row corresponding to the row index from the table for each thread, targeting all retrieved row indices. If the ratio is less than or equal to the predetermined threshold, it retrieves the row corresponding to the row index obtained by each thread in parallel from the table for each thread, and then integrates the results. The hash table creation device according to claim 2.

6. The key conditions set for each thread are defined by multiple key conditions to ensure that all rows in the table are retrieved comprehensively and exclusively. A hash table creation device according to any one of claims 1 to 5.

7. The mask condition verification unit divides the rows contained in the table and assigns them to each thread, and for each thread, it retrieves the row index that matches the mask condition from among the rows of the table assigned to that thread. A hash table creation device according to any one of claims 1 to 5.

8. The mask condition verification unit assigns rows from a table, divided according to equality, value, or value range, to each thread. A hash table creation device according to any one of claims 1 to 5.

9. For each thread to which rows separated from the table are assigned, a row index, which is the identification information of a row that matches a mask condition (a condition for identifying data within the table), is obtained from the assigned rows of the table. For each thread, the values ​​of rows that match the key conditions set for each thread are retrieved for the table keys corresponding to all the retrieved row indexes, and the row data corresponding to the table keys is inserted into the hash table. A method for creating a hash table characterized by the following features.

10. On the computer, For each thread to which rows split from the table are assigned, a mask condition verification process is performed to obtain a row index, which is the identification information of a row that matches the mask condition, which is the condition for identifying data within the table, from among the rows of the assigned table. Hash table creation process: For each thread, for the keys of the table corresponding to all the retrieved row indexes, retrieve the row values ​​that match the key conditions set for each thread, and insert the row data corresponding to the table keys into the hash table. A hash table creation program to execute the following.

Images