Index Management Device

Abstract

To provide an index management device that can apply a Bloom filter to a method requiring data sorting to perform update processing rapidly even when there is a large amount of existing data, and is also expected to improve search performance.SOLUTION: Provided is an index management device 3 for a database that stores multidimensional data including a quantity with which order and distance between two values can be properly defined, and unordered discrete values in which neither the order nor the distance between the two values can be properly defined. The index management device comprises index generation means 31 of generating an index with a hierarchical structure consisting of a plurality of nodes including a root node and leaf nodes; entry processing means 32 of performing processing to store pointer and key information in the node; and update processing means 33 of calculating a minimum positive integer PS under a predetermined condition for a plurality of multidimensional data, and updating the nodes by repeating division based on the positive integer PS.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an index management device for a database that stores multidimensional data including quantities for which the order and distance between two values ​​can be properly defined, and unordered discrete values ​​for which neither the order nor the distance between two values ​​can be properly defined. [Background technology]

[0002] A technique for creating search indexes is known as a technology for speeding up data searches. In short, when searching for data corresponding to a specific key, it takes an enormous amount of time to check all records in a database one by one from the beginning, so it is known that an index tree is generated in advance to speed up searches for a specific key (see Patent Documents 1 and 2, etc.).

[0003] Here, a set of data recorded in a database is called a record, and the data used for searching among them is called a key. To search for records stored in a database by key, it is desirable that the records be sorted in key order, but the process of sorting and recording the records in key order takes time.

[0004] Therefore, it is common to record records in the order in which they arrive, and then sort and separately record pointers to the records that correspond to the keys in a tree structure, which is called an index tree. A typical index implementation method is one that uses a data structure called a Balanced Tree (B-tree), which is often used in commercial databases and file systems.

[0005] A B-tree can be thought of as data sorted by a specific column and divided into a nested structure. This allows range searches on a single column (extracting records with a value greater than or equal to a specified lower limit and less than or equal to a specified upper limit) to be performed quickly using only a simple magnitude comparison of the keys. Range searches on multiple columns can also be handled in a similar way by comparing the second column only if the first column has the same value, and if the second column also has the same value, then comparing the third column, etc. However, in practice, only the first few columns are often meaningful.

[0006] On the other hand, the Rectangle Tree (R-tree) disclosed in Patent Documents 1 and 2 can be considered as data recursively divided into rectangles (in the two-dimensional case, generally a hyper-rectangular prism). Instead of a size comparison like a B-tree, the range search is performed by treating the search range as a rectangle and examining the containment relationships between the rectangles. However, since the area of ​​the rectangle is used as a criterion during internal processing, the R-tree index can only be set to quantities for which the order and distance between two values ​​can be properly defined, and it cannot handle unordered discrete values ​​for which neither the order nor the distance between two values ​​can be properly defined.

[0007] Therefore, the C-ND tree, disclosed in Non-Patent Document 1, has been proposed as an extension of the R-tree so that it can also handle unordered discrete values. For sequences of unordered discrete values, the C-ND tree can be regarded as an R-tree in which the "length of one side of a rectangle" is replaced with the number of distinct values. Because the C-ND tree can handle unordered discrete values ​​while inheriting the good properties of the R-tree, it has the potential to overcome the weaknesses of existing indexes with a single index.

[0008] On the other hand, C-ND trees use simple bit arrays to represent "ranges" of unordered discrete values, so they require all possible distinct values ​​to be known in advance, and require enormous storage capacity if the number of distinct values ​​is very large.

[0009] Therefore, as disclosed in Patent Document 3, the applicant has applied a Bloom filter to the setting of a specified value search for unordered discrete values, thereby enabling high-speed searches of multidimensional data containing unordered discrete values, for which neither the order nor the distance between two values ​​can be properly defined, while keeping storage capacity low. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-330943 [Patent Document 2] Patent No. 3708809 [Patent Document 3] Patent Publication No. 2021-114037 [Non-patent literature]

[0011] [Non-Patent Document 1] Changqing Chen et al.,The C-ND Tree:A Multidimensional Index for Hybrid Continuous and Non-ordered Discrete Data Spaces,EDBT '09 Proceedings of the 12th International Conference on Extending Database Technology,2009.3,p.462-471 [Non-patent document 2] Scott Leutenegger et al.,STR: A Simple and Efficient Algorithm for R-Tree Packing, Proceedings 13th International Conference on Data Engineering,1997.4,p.497-506 Summary of the Invention [Problem to be solved by the invention]

[0012] However, in an update method such as that disclosed in Patent Document 3, in which an index is constructed for any number of records by appropriately repeating the update procedure from a state of 0 records when one new record of data is added, the update process takes a huge amount of time when there is a large amount of existing data.

[0013] On the other hand, as disclosed in Non-Patent Document 2, a bulk loading method has been developed for R-trees. In the method described in Non-Patent Document 2, the data order replacement However, it is not known what the appropriate order of Bloom filters is.

[0014] Therefore, the present invention is based on the data sequence replacement The objective of this invention is to provide an index management device that enables Bloom filters to be applied to methods that require updating, thereby enabling rapid update processing even when there is a large amount of existing data, and is also expected to improve search performance. [Means for solving the problem]

[0015] In order to achieve the above object, the index management device of the present invention is an index management device for a database that stores multidimensional data including quantities for which the order and distance between two values ​​can be properly defined and unordered discrete values ​​for which neither the order nor the distance between two values ​​can be properly defined, and includes index generation means that generates an index of a hierarchical structure consisting of a plurality of nodes, entry processing means that, when performing a process of storing pointers and key information in the nodes, sets the key information to a range search if the key information is the quantity, and sets the key information to a specified value search to which a Bloom filter is applied if the key information is the unordered discrete value, and a positive integer PS that is smallest under predetermined conditions for a plurality of the multidimensional data, and performs a sorting process for the key information. replacementand an update processing means for updating the nodes by repeating the process of dividing the nodes based on the positive integer PS and the positive integer PS, and when rearranging the nodes in the update processing means, if the key information is the unordered discrete value, the presence or absence of a bit that is 1 is checked for each predetermined group, and a magnitude relationship is determined based on a 2-bit bit string in which 1 indicates presence and 0 indicates absence. [Effects of the Invention]

[0016] The index management device of the present invention configured as described above has an entry processing means that stores pointers and key information in nodes, and when the key information is a quantity, it is set to a range search, and when the key information is an unordered discrete value, it is set to a specified value search. A Bloom filter is applied to the specified value search setting.

[0017] In addition, for multiple multidimensional data, the smallest positive integer PS is calculated under predetermined conditions, and the key information is sorted. replacement The node updating unit 100 includes an update processing unit that updates the nodes by repeating the process of dividing the nodes based on the positive integer PS.

[0018] When rearranging the data in the update processing means, if the key information is an unordered discrete value, the presence or absence of a bit that is 1 is checked for each predetermined group, and the magnitude relationship is determined based on a two-bit string in which presence is represented by 1 and absence is represented by 0.

[0019] In short, the sequence of data replacement By making it possible to apply Bloom filters to methods that require updating, it will be possible to quickly update large amounts of existing data, and it will also be possible to build indexes that are expected to improve search performance. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram illustrating an outline of the overall configuration of a multidimensional data search system incorporating an index management device according to an embodiment of the present invention. [Figure 2] FIG. 2 is an explanatory diagram illustrating the basic structure of an index. [Figure 3] FIG. 10 is an explanatory diagram of an example of a search using an index of multidimensional data that includes numerical values ​​and partition values. [Figure 4] FIG. 10 is an explanatory diagram illustrating the results of calculating the false positive probability of a Bloom filter. [Figure 5] FIG. 10 is an explanatory diagram illustrating assumed data when considering parameters of a Bloom filter. [Figure 6] FIG. 10 is an explanatory diagram assuming an average number of partition values ​​included in the descendants of each node in each level of the index. [Figure 7] In considering the number K of hash functions, (a) is a table showing the calculation results of the false positive probability of the partition value A, and (b) is a table showing the calculation results of the false positive probability of the partition value B. [Figure 8] FIG. 10 is an explanatory diagram showing an outline of how an index is updated when an additional record is found. [Figure 9] This is an explanatory diagram showing an image of target data being distributed in a space with the first key and second key as its axes. [Figure 10] FIG. 10 is an explanatory diagram showing an image of leaf nodes distributed in a space with the first key and second key as axes. [Figure 11] FIG. 10 is an explanatory diagram showing an image of intermediate nodes distributed in a space whose axes are the first key and the second key. [Figure 12] FIG. 1 is an explanatory diagram schematically illustrating indexes used in the verification of Example 1. [Figure 13] FIG. 10 is an explanatory diagram showing a list of the average number of nodes and standard deviation of indexes that use the "simple comparison method." [Figure 14] FIG. 10 is an explanatory diagram showing a list of the average number of nodes and standard deviation of indexes that employ the "present comparison method." DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The index management device of this embodiment is incorporated into a multidimensional data search system 1 that includes a database in which multidimensional data is stored. FIG.

[0022] The multidimensional data search system 1 comprises an input means 2, an index management device 3, and a database 4. The input means 2 includes means for directly inputting commands and data using a keyboard, mouse, touch panel, etc., means for importing from a storage medium such as a flash memory or hard disk, and means for importing via a communication line such as the Internet or a LAN.

[0023] The database 4 also stores index information 41 such as the structure and data of index Z, and records 42 which are multidimensional data to be searched. In this embodiment, the records 42 are a combination of quantitative data and unordered discrete value data.

[0024] Here, "quantity" refers to data for which the order and distance between two values ​​can be properly defined, such as length, weight, time, and other numerical values ​​for which magnitudes can be compared. On the other hand, "unordered discrete values" refer to data for which neither the order nor the distance between two values ​​can be properly defined, such as color types, place names, and characters.

[0025] Figure 2 is a diagram that shows a schematic example of a record set W, which is a database in which multiple records (data rows identified by row IDs such as "R00" and "R01") are recorded, and an index Z that is created to extract a desired record from the record set W in a short period of time.

[0026] The basic structure of this index Z is a tree structure in which nodes are arranged hierarchically over three or more levels. The topmost node is called the root node ZA, and the bottommost node is called the leaf node ZC. The nodes at one or more levels between the root node ZA and the leaf node ZC are called intermediate nodes (internal nodes) ZB.

[0027] Each node (ZA, ZB, ZC) consists of a node ID for identifying the node and an entry for storing data in the node. The entry for node (ZA, ZB, ZC) stores a pointer and key information.

[0028] In nodes (ZA, ZB) other than leaf node ZC, a "pointer" refers to data that points to a lower-level node (child node) corresponding to the key information recorded in the entry. On the other hand, in leaf node ZC, a pointer is data that points to the record that is the original data (for example, a data set of one row identified by row ID "R00").

[0029] "Key information" refers to data contained in a record that is used for searching. In this embodiment, both quantities and unordered discrete values ​​can be key information. The key information stored in a leaf node ZC is the value of the key column of the original data (for example, one row of record). Nodes (ZA, ZB) other than the leaf node ZC record a summary of the key information of all leaf nodes ZC that are descendants (child nodes and child nodes connected to them, etc.). For example, in the case of a numeric string, a pair of minimum and maximum values ​​is recorded as key information, and in the case of a segmented value string, a Bloom filter, described below, is recorded as key information.

[0030] For example, referring to Figure 2, the top-level root node ZA has a node with node ID "N10," and the 10 entries (entry 0 to entry 9) stored in "N10" store a pointer to the corresponding node in the intermediate node ZB (for example, "entry 0" stores "N20"), a numeric value "0 to 600" that serves as key information, and a segment value "11010101" that serves as key information. Here, the numeric value is set in the range "0 to 600," and the segment value is set to the specified value "11010101."

[0031] Similarly, intermediate node ZB also has a node with multiple entries stored in it, and if it corresponds to "entry 0" of node ID "N20," for example, it will move to the node with lower node ID "N30." When it reaches node ID "N90" of leaf node ZC at the bottom level, it will extract the record with row ID "R00" from record set W pointed to by "entry 0."

[0032] To explain the above-mentioned data search in another way, if the node X that is the target of the search is a leaf node ZC, and the key information of node X matches the search criteria, one row of the original data pointed to by the pointer of node X (such as "R00", "R01", etc.) will be added to the search result set S. On the other hand, if the node X that is the target of the search is not a leaf node ZC, a search for lower nodes will be performed for each entry of node X that records key information whose descendants may match the search criteria.

[0033] 1, the index management device 3 of this embodiment for performing such searches at high speed includes index generation means 31 for generating a tree-structured index Z, entry processing means 32 for processing data stored in the entry of each node, and update processing means 33 that operates when it becomes necessary to update the index Z due to the addition of a new record, etc. The entry processing means 32 also includes a Bloom filter setting unit 321 for setting parameters of a Bloom filter, which will be described later.

[0034] Next, referring to Figure 3, we will explain an example of a search using index Z for multidimensional data containing numerical values ​​and segment values. First, a search condition is set for a search. In this example, the search condition is "50 ≦ numerical value column ≦ 70 and segment value column = A." The "A" in the segment value column is replaced with the Bloom filter "00010001."

[0035] When the root node ZA is referenced based on the search conditions, "Entry 0" of node ID "N10" meets the search conditions in both the numeric range search and the specified value search of the classification value, so the process moves to node ID "N20" of the lower intermediate node ZB.

[0036] In "N20" of intermediate node ZB, the numeric column of "Entry 0" does not meet the search criteria, but both the numeric column and the segment value column of "Entry 1" meet the search criteria, so the process follows pointer "N31" of "Entry 1" to "N31" of leaf node ZC. Then, following pointer "R05" of "Entry 1" that meets the search criteria, one record with row ID "R05" is extracted from record set W. In addition to the numeric and segment value, which are key information, this record also contains the data "へ" in a non-key column that is not considered key information. Note that in this example, "Entry 1" of node ID "N10" of root node ZA does not meet the search criteria, so further searching of the lower nodes pointed to by "Entry 1" is unnecessary.

[0037] Here, in the C-ND tree (Non-Patent Document 1), a simple bit array is used to represent the segment values, so it is necessary to grasp all possible distinct values ​​in advance and reserve the number of bits according to that number. Therefore, as the number (type) of segment values ​​increases, the storage capacity increases, and it may not be applicable if the range of segment values ​​cannot be predicted.

[0038] Therefore, in the index management device 3 of this embodiment, partition values ​​are expressed by applying a Bloom filter. A Bloom filter is a data structure that can probabilistically determine whether or not a set contains an element, and is said to be a space-efficient structure. By applying a Bloom filter, it becomes possible to express a range of unordered discrete values ​​with a fixed number of bits, regardless of the number of distinct values.

[0039] To apply a Bloom filter, a suitable positive integer N and one or more (let's say K) different hash functions (F_1, F_2,...,F_K) are determined in advance. However, each hash function must have an integer range of 1 to N. This can be done by taking the result of any hash function and calculating the remainder +1 modulo N.

[0040] For example, a Bloom filter B(X) for data X can be expressed as an N-bit string with the F_i(X)th bit (i = 1, 2, , K) set to "1" and the rest set to "0." Then, a Bloom filter B(S) for a data set S = {X_1, X_2,} is the result of the bitwise OR operation of B(S) = B(X_1), B(X_2), .

[0041] For example, if the number of bits N is "8", and F_1("red") = 1 and F_2("red") = 4 for the data "red", the Bloom filter B("red") will be "00001001". Also, if F_1("blue") = 4 and F_2("blue") = 8 for the data "blue", the Bloom filter B("blue") will be "10001000". The result of the bitwise OR operation of these is B({"red", "blue"}) = 10001001.

[0042] For any set S, if B(X) AND B(S) ≠ B(X), then X does not belong to S. Here, "AND" indicates a bitwise AND operation. On the other hand, if B(X) AND B(S) = B(X), then X ∈ S does not hold. Therefore, if you record a Bloom filter as a summary of a set, you can easily determine whether certain data is included in the set without referring to the original set. However, "does not belong (No)" is a definite result, while "does belong (Yes)" is a probabilistic result.

[0043] For example, if B(S) = 10001001, and B("yellow") = 00100001, then B("yellow") AND B(S) = 00000001 ≠ B("yellow"), and it can be said with certainty that "yellow" does not belong to set S. On the other hand, if B("purple") = 10000001, then B("purple") AND B(S) = 10000001 = B("purple"), and it can be said that there is a possibility that "purple" belongs to set S ("purple"∈S).

[0044] Furthermore, when summarizing a set S of numerical values ​​x into its minimum value (m) and maximum value (M), "if x < m or M < x, then x does not belong to S" is true, but "if m ≦ x ≦ M, then x ∈ S" is not true, so Bloom filters can be seen as a non-numeric version of such summarization.

[0045] Next, we will explain the size of each data. When determining the width of a data column, if it is a numeric value, the "maximum value - minimum value" can be used as the width of the entire data. On the other hand, if it is a partitioned value, the number of "1"s in the Bloom filter can be used as the width of the entire data. For example, if it is "10010001", the width will be "3". Also, let's assume that "normalized width = width / width of entire data".

[0046] If the size of each data item is taken as the sum of the squares of the relative widths (normalized widths) for each column, then for data that consists of only numerical values, the size can be considered to be the square of the length of the diagonal of a hyper-rectangle defined from the minimum to maximum values ​​of each column.

[0047] Therefore, we will continue to explain the data size using the index Z shown in Figure 2 as an example. The width of the entire data is "900" for numeric columns, since it becomes "900 - 0" when looking at the root node ZA, and "8" for segment value columns, since the maximum is "11111111" (only segment values ​​(A, B) are shown in the record group W in Figure 2).

[0048] Furthermore, the width of "entry 0" of "N20" of intermediate node ZB is "90" for the numeric value column and "4" for the partition value column. Therefore, the normalized width of "entry 0" of "N20" is "0.1" for the numeric value column and "0.5" for the partition value column, and the size of "entry 0" of "N20" is 0.26.

[0049] By the way, Bloom filters do not have false negatives, but they can produce false positives. Also, elements can be added to the set, but cannot be removed. As more elements are added to the set, the possibility of false positives increases.

[0050] Therefore, it is necessary to appropriately set and apply the number of bits N of the Bloom filter and the number of hash functions K. Below, we will explain the results of examining each parameter when applying it, with reference to Figures 4 to 7.

[0051] First, we will explain the false positive probability of a Bloom filter with N bits. The probability P1 that a bit is 0 in a Bloom filter representation of one value using one hash function is P1 = (1 - 1 / N). From this, the probability P2 that a bit is 0 in a Bloom filter representation of one value using K hash functions is P2 = P1^K = (1 - 1 / N)^K.

[0052] Furthermore, the probability P3 that a bit is 0 in a Bloom filter representation of M values ​​using K hash functions is P3 = P2^M = (1 - 1 / N)^(K×M). And the probability P4 that a bit is 1 in a Bloom filter representation of M values ​​using K hash functions is P4 = 1 - P3 = 1 - (1 - 1 / N)^(K×M).

[0053] As a result, the probability (false positive probability) of determining that a Bloom filter containing M values ​​using K hash functions may (possibly) contain a value that it does not actually contain is the probability that all bits in the Bloom filter corresponding to the results of the K hash functions are 1, so P4^K = (1 - (1 - 1 / N)^(K×M))^K.

[0054] Figure 4 shows an example of the results of calculating the false positive probability of a Bloom filter by varying the values ​​of M and K mentioned above when the number of bits is N = 128. In other words, from the formula for false positive probability, it is possible to calculate the value of K that minimizes the false positive probability for given N and M.

[0055] Next, we will explain the parameters that must be determined when applying a Bloom filter. First, the number of bits N of the Bloom filter is a parameter. Although it is possible to set a different number of bits for each data string, we will assume that the number of bits N is a constant value here.

[0056] Another parameter is the number K of hash functions used when calculating the Bloom filter. Furthermore, the upper limit (E_max) and lower limit (E_min) of the number of entries per node are also parameters that must be determined. If data is added that exceeds the upper limit (E_max) of the number of entries per node, node splitting will occur. Furthermore, the lower limit (E_min) of the number of entries per node is (E_max / 2), and if it is exceeded, a violation will occur when the node is split.

[0057] When considering the optimal parameter values, the assumed data conditions were a record set W consisting of approximately 1 billion (≒ 2^30) records, containing two numeric columns and two segment value columns. Figure 5 shows an example of part of the assumed data. Here, the number of possible values ​​for segment value column A is "16" and for segment value column B is "256".

[0058] First, let's consider the number of bits N. In many cases, numbers are expressed in 8 bytes or less. Therefore, 16 bytes are required for the key information of a numeric string to store the minimum and maximum values. It is desirable that the number of bytes for the segment value string also fit within a similar range, so the number of bits N is set to 128 bits (= 16 bytes).

[0059] Next, we will consider the upper limit (E_max) and lower limit (E_min) of the number of entries. First, we will consider the upper limit (E_max) of the number of entries. Database speed is greatly affected by the amount of disk I / O (data exchange with storage devices such as hard disks), and the database block size (the amount of data per disk I / O) is usually around 8KiB (= 8192 bytes). Therefore, it is desirable to keep the amount of data held by one index node within 8KiB. Under the assumed data conditions, the amount of data per entry is (16 bytes for the numeric column x 2 + 16 bytes for the partition value column x 2) = 64 bytes, so the upper limit (E_max) of the number of entries is set to E_max = 8192 / 64 = 128.

[0060] Regarding the lower limit of the number of entries (E_min), setting it small may increase the number of options for node splitting, which may result in improved search performance, but this is difficult to evaluate.On the other hand, from the perspective of disk I / O, a larger value is better as it means fewer empty entries per node, so we set the upper limit, E_max / 2 = 64, as the lower limit of the number of entries (E_min).

[0061] Finally, let us consider the number of hash functions, K. In the case of the assumed number of data items (1 billion items) mentioned above, under the conditions of the upper limit (E_max) and lower limit (E_min) of the number of entries mentioned above, the height of the tree of index Z will be 5 levels.

[0062] Therefore, in the data addition procedure and node splitting procedure, the number of values ​​included in the Bloom filter is selected to be small so as to keep the size small. Therefore, we assume that the average number of partitioned values ​​included in the descendants of each node at each level of index Z is the value shown in the table in Figure 6.

[0063] Based on the average number of partitioned values ​​shown in Figure 6 and the false positive probability of the Bloom filter shown in Figure 4, we can calculate the expected number of nodes to access when searching for a partitioned value that is not included in the original table. The number of accessed nodes is roughly the same as the number of required disk I / Os, so it can be said to be a reasonable index for evaluating the impact of false positives on search performance.

[0064] Therefore, we calculate the impact on false positives for each partition value (A, B). For partition value A, the number of nodes accessed when K=1 is "1" if the first level is the root node only, "128 x 0.0608" since the second level is the false positive entries of the root node, and "128 x 0.0608 x 128 x 0.0309" since the third level is the false positive entries of the nodes accessed in the second level. In other words, the number of nodes accessed up to the fifth level is 1 + 128×0.0608 + 128×0.0608×128×0.0309 + 128×0.0608×128×0.0309×128×0.0156 + 128×0.0608×128×0.0309×128×0.0156×128×0.0078 ≒ 162.

[0065] Similarly, if we calculate for K=2, 3, 4, 5, we get the following. K=2: 1 + 128×0.0139×(1 + 128×0.0037×( 1 + 128×0.0010×(1 + 128×0.0002))) ≒ 3.73 K=3: 1 + 128×0.0051×(1 + 128×0.0007×( 1 + 128×0.0001×(1 + 128×0.0000))) ≒ 1.71 K=4: 1 + 128×0.0024×(1 + 128×0.0002×( 1 + 128×0.0000×(1 + 128×0.0000))) ≒ 1.32 K=5: 1 + 128×0.0014×(1 + 128×0.0001×( 1 + 128×0.0000×(1 + 128×0.0000))) ≒ 1.18 FIG. 7(a) shows the calculation results of the false positive probability for the division value A in a table.

[0066] On the other hand, for the partition value B, the number of nodes accessed when K=1 is 1 + 128×0.3947 + 128×0.3947×128×0.1179 + 128×0.3947×128×0.1179×128×0.0309 + 128×0.3947×128×0.1179×128×0.0309×128×0.0078 ≒ 6840.

[0067] The calculation results for K=2, 3, 4, 5 are as follows: K=2: 1 + 128×0.4014×(1 + 128×0.0493×( 1 + 128×0.0037×(1 + 128×0.0002))) ≒ 534 K=3: 1 + 128×0.4712×(1 ​​+ 128×0.0309×( 1 + 128×0.0007×(1 + 128×0.0000))) ≒ 321 K=4: 1 + 128×0.5617×(1 + 128×0.0243×( 1 + 128×0.0002×(1 + 128×0.0000))) ≒ 302 K=5: 1 + 128×0.6545×(1 + 128×0.0220×( 1 + 128×0.0001×(1 + 128×0.0000))) ≒ 323 FIG. 7(b) shows the calculation results of the false positive probability for the cutoff value B in a table.

[0068] From the above results, it seems appropriate to set the number K of hash functions to "K = 4" or "K = 5" under the assumed data conditions mentioned above.

[0069] Next, the update process of the index management device 3 will be described. Fig. 8 is a diagram for explaining an outline of index updating when new multidimensional data is added as an additional record W1 to a record set W. Here, Fig. 2 is the index Z before the update, and Fig. 8 is the index NZ after the update.

[0070] That is, when an additional record W1 (row ID: R08) is added to record set W, index Z is updated to index NZ to enable searching for the additional record W1. Compared to index Z shown in Figure 2, index NZ shown in Figure 8 has updated entries for nodes at each level: root node ZA, intermediate node ZB, and leaf node ZC.

[0071] On the other hand, even if a bulk loading method is applied, in which initial data is input all at once, it is possible to build an index with updated nodes. The flow of update processing by the update processing means 33 to which the bulk loading method is applied will be described below.

[0072] Specifically, an improved bulk loading method for R-trees (see Non-Patent Document 2) is applied. That is, the bulk loading method described in Non-Patent Document 2 requires a comparison of data magnitudes, and therefore cannot be applied directly to unordered discrete values. Therefore, in the index management device 3 of this embodiment, which converts unordered discrete values ​​into Bloom filters for handling, an effective comparison procedure for Bloom filters is introduced.

[0073] Here, the number of target data items to be updated is NN, the number of key columns in the index is NK, and the maximum number of entries held by one node is NM. In the first step, if NN≦NM, an index node (root node) pointing to all target data is created, and the update process can be completed.

[0074] On the other hand, if NN > NM, the second step is to calculate the smallest positive integer PS that satisfies the predetermined condition NM × PS^NK ≥ NN. Then, in the third step, the NN pieces of target data are sorted according to the first key in the key information, and divided into PS groups of roughly the same number of items in that order.

[0075] In the fourth step, the data belonging to each of these groups is sorted in accordance with the second key, which is key information, and further subdivided into PS groups of roughly the same number of items in that order.

[0076] Figure 9 is an explanatory diagram showing an image when the number of target data items NN = 28, the number of index key columns NK = 2 (vertical and horizontal axes), and the maximum number of entries held by one node NM = 4. In this case, the smallest positive integer PS for which NM × PS^NK ≧ NN is 3. In short, it shows an image in which the data sorted for the first and second keys is divided into three parts (see dashed lines).

[0077] On the other hand, if the number of key columns NK is greater than 2, the same procedure is repeated up to the NKth column in the fifth step, and finally a set of groups (PS^NK) is obtained, each group consisting of NM or less data items.

[0078] In the sixth step, an index node (leaf node) is created for each group, pointing to the data within that group. In the seventh step, the target data is replaced with these nodes, and the number of target data items (NN) is replaced with the number of groups. Then, the process returns to the first step and repeats.

[0079] Figure 10 shows an image in which data with a number of groups NN = 9 is created by leaf nodes. Therefore, if we calculate the smallest positive integer PS such that NM × PS^NK ≧ NN again, we get 2. In short, Figure 10 shows an image in which the data by leaf nodes sorted by the first key and second key are each divided into two (see dashed lines).

[0080] Therefore, for each group, we create an index node (intermediate node) that points to the data within that group. Then, we replace the target data with these nodes, replace the number of target data items NN with the new number of groups NN = 4, and return to the first step described above to repeat the process.

[0081] Figure 11 shows an image in which data with a group count of NN = 4 is created by an intermediate node. Then, in the first step, when the number of data NN is compared with the maximum number of entries NM = 4 that one node can hold, NN (= 4) ≤ NM (= 4), so the update process can be completed.

[0082] By the way, if 2^NK > NM, an infinite loop will occur, so in that case, the fourth and fifth steps described above will be terminated appropriately midway to keep the final number of groups below NM. Also, the sorting method in steps 3 to 5 is based on a general sorting algorithm such as quicksort.

[0083] The method for comparing data in the sorting algorithm differs depending on the type of key. First, if the key is a quantity and it is the first loop (when creating a leaf node), the original data can simply be compared as a number.

[0084] On the other hand, from the second loop onwards, the key information will be a pair of minimum and maximum values, but the minimum values ​​will be compared first, and if the minimum values ​​are equal, the maximum values ​​will be compared. On the other hand, when the key is an unordered discrete value, that is, when the key information is a Bloom filter, the comparison method is different from when it is a quantity, and will be described in detail below.

[0085] The magnitude relationship (X1 > Y1 or X1 = Y1 or X1 < Y1) between the Bloom filters X1 and Y1 to be compared is determined using the following procedure. For simplicity, the explanation will be given here using 16 bits.

[0086] First, for X1, every 8 bits are checked to see if there is a bit that is 1, and converted into a 2-bit bit string X8, where presence is represented by 1 and absence is represented by 0. (Example) 0001101100011011 → 11, 0000000001000000 → 01

[0087] Next, an XOR operation (exclusive OR) is performed on X8 and a bit string obtained by shifting X8 one bit to the right, and the result is defined as X8'. (Example) 00 xor00 → 00, 01 xor 00 → 01, 10 xor 01 → 11, 11 xor 01 → 10

[0088] Similarly, calculate Y8', and compare X8' and Y8' as binary numbers. If X8' > Y8', determine that X1 > Y1 and the comparison is complete. If X8' < Y8', determine that X1 < Y1 and the comparison is complete.

[0089] Otherwise, X1 is checked for the presence or absence of a bit that is 1 for every four bits, and converted into a four-bit bit string X4 in which presence is represented by 1 and absence by 0. (Example) 0001101100011011 → 1111, 0000100001000010 → 0111

[0090] Then, X4 and a bit string obtained by shifting X4 one bit to the right are XORed, and the result is designated as X4'. (Example) 0000 xor0000 → 0000, 1010 xor 0101 → 1111, 1110 xor 0111 → 1001

[0091] Similarly, calculate Y4', and compare X4' and Y4' as binary numbers. If X4' > Y4', determine that X1 > Y1 and complete the comparison. If X4' < Y4', determine that X1 < Y1 and complete the comparison. This procedure is repeated two bits at a time, then one bit at a time, and if the magnitude relationship cannot be determined until the end, determine that X1 and Y1 match and set X1 = Y1.

[0092] The above explanation is for 16 bits, but it can be easily expanded to 32 bits or 64 bits. Therefore, the magnitude relationship between two Bloom filters (BX, BY) where the number of bits N is 2^n (n: an integer greater than or equal to 1) is explained below. Note that for bit numbers other than 2^n, application is possible by adding 0 as upper bits as appropriate to make it 2^n.

[0093] (Step 1) Initialize the variable KK to n-1. (Step 2) For each of BX and BY, check whether there is a bit that is 1 for every 2^KK bits for each specified group, and let BX' and BY' be 2-bit bit strings in which presence is represented by 1 and absence is represented by 0. (Example) 00011011 00011011 → 11 (=BX'), 00000000 01000000 → 01 (=BY')

[0094] (Step 3) BX'' and BY'' are the bit strings obtained by performing a 1-bit right shift operation on BX' and BY', respectively. (Example) 00 → 00, 01 → 00 (=BY), 10 → 01, 11 → 01 (=BX)

[0095] (Step 4) The result of the exclusive OR (XOR operation) of BX' and BX'' is BX''', and the result of the exclusive OR of BY' and BY'' is BY'''. (Example) 00 xor 00 = 00,01 xor 00 = 01(=BY'''),10 xor 01 = 11,11 xor 01 = 10(=BX''')

[0096] (Step 5) When viewed as binary numbers, if BX''' > BY''', then determine that BX > BY; if BX''' < BY''', then determine that BX < BY; if neither applies, then if KK=0, then determine that BX=BY; if KK≠0, then set KK-1, and repeat from (Step 2).

[0097] An example of repeating from (step 2) for KK-1 is as follows: (Example of step 2) 0001 1011 0001 1011 → 1111(=BX'), 0000 1000 0100 0010 → 0111(=BY') (Example of step 3) 1010 → 0101,1110 → 0111,0101 → 0010,0111 → 0011(=BY''),1111 → 0111(=BX'') (Example of step 4) 1111 xor 0111 = 1000(=BX'''),0111 xor 0011 = 0100(=BY'''), 1010 xor 0101 = 1111,1110 xor 0111 = 1001,0101 xor 0010 = 0111

[0098] Next, the operation of the index management device 3 of this embodiment will be described. The index management device 3 of this embodiment configured as above has an entry processing means 32 that stores pointers and key information in nodes (ZA, ZB, ZC), and sets it to range search when the key information is a quantity such as a numeric value, and sets it to specified value search when the key information is an unordered discrete value such as a partition value. A Bloom filter is applied to set the specified value search.

[0099] This enables high-speed searches of multidimensional data that includes unordered discrete values ​​(distinct values) where neither the order nor the distance between two values ​​can be properly defined. In other words, because index Z has an index structure that is suitable for range searches across multiple columns, it can achieve high-speed and stable search speeds.

[0100] Furthermore, for multiple multidimensional data input at once, the smallest positive integer PS is calculated under a predetermined condition (NM × PS^NK ≧ NN), and all multidimensional data to be updated are sorted in terms of key information. replacementIn addition, the node management unit 30 includes an update processing means 33 that updates the node including the stored entry by repeating the process of dividing the node based on the positive integer PS.

[0101] When rearranging in the update processing means 33, if the key information is an unordered discrete value, the presence or absence of a bit that is 1 is checked for each predetermined group (every 2^KK bits), and the magnitude relationship is determined based on a 2-bit bit string in which presence is represented by 1 and absence is represented by 0.

[0102] In short, the sequence of data replacement By making it possible to apply Bloom filters to methods that require updating, it becomes possible to quickly update large amounts of existing data, and it is also possible to build indexes that are expected to improve search performance.

[0103] Furthermore, it is possible to handle segmented values ​​(unordered discrete values) while keeping the memory capacity low. In short, by applying a Bloom filter with a probabilistic data structure, it is no longer necessary to reserve a huge amount of memory capacity in advance to accommodate the number of distinct values, making it possible to build a device with a small memory capacity. It is also possible to handle cases where the distinct values ​​cannot be known in advance.

[0104] Furthermore, if such a single index Z can handle range searches for quantities and specified value searches for unordered discrete values ​​in the same way, the design work for index Z can be simplified.

[0105] Furthermore, by using the Bloom filter setting unit 321 to set the number of bits N and the number of hash functions K that are suitable for the search strategy, it is possible to reduce the false positive probability and make the Bloom filter function effectively. [Example]

[0106] The effectiveness of the update processing means 33 of the index management device 3 described in the above embodiment will be described below with reference to Figures 12 to 14. Note that the same terms or the same reference numerals will be used to describe the same or equivalent parts as those described in the above embodiment.

[0107] In this Example 1, the effectiveness of a comparison method introduced to apply a Bloom filter to the bulk loading method described in the above embodiment was verified. The verification was performed in the following manner while changing the value of the variable C several times.

[0108] In verification procedure 1, C types of unordered discrete values ​​are prepared, and 100 x 100 x 100 (= SN) pieces of data that take any of these values ​​are generated. Next, in verification procedure 2, the maximum number of entries per node, NM, is set to 100, the number of bits in the Bloom filter, N, is set to 64, and the number of hash functions, K, used to create the Bloom filter is set to 4.

[0109] FIG. 12 shows the state in which a three-layer index is constructed using the above procedure, with each node holding 100 entries. Then, in verification procedure 3, two indexes are constructed using the "simple comparison method," which considers bit strings as binary numbers and determines which is larger, and the "present comparison method," which employs the Bloom filter comparison method described in the embodiment above, with the construction algorithm described in the embodiment above. In short, the "simple comparison method" was used as a comparison object to confirm the effectiveness of the "present comparison method."

[0110] In verification procedure 4, random value searches (see the above embodiment for the search procedure) were performed 10,000 times for each of the two constructed indexes, and the average and standard deviation of the number of nodes that needed to be accessed during the search were measured.

[0111] Figure 13 shows a table of the average number of nodes and standard deviation for indexes that use the "simple comparison method," while Figure 14 shows a table of the average number of nodes and standard deviation for indexes that use the "full comparison method."

[0112] Looking at these results, there was not much difference between the two comparison methods when the type C of unordered discrete values ​​was C = 100, 1000. On the other hand, when C = 10,000, 100,000, the average number of nodes for the "comparison method" was clearly smaller, and the variation indicated by the standard deviation was also smaller.

[0113] From the above verification results, it is considered that in many cases, using the "present comparison method" described in the above embodiment rather than using the "simple comparison method" will enable stable and fast searches, especially when the number of types C of unordered discrete values ​​is large.

[0114] The other configurations and effects are substantially the same as those of the above embodiment, and therefore the description thereof will be omitted.

[0115] The embodiments of the present invention have been described above in detail with reference to the drawings, but the specific configurations are not limited to these embodiments and examples, and design changes that do not deviate from the gist of the present invention are included in the present invention.

[0116] For example, in the above embodiment and Example 1, multidimensional data was described with a small number of dimensions for simplicity of explanation, but the present invention is not limited to this, and can also be applied to multidimensional data that includes a large number of quantities and key information of unordered discrete values. [Explanation of symbols]

[0117] 3: Index management device 31: Index generation method 32: Entry processing means 321: Bloom filter setting section 33: Update processing means Z: Index ZA : Root node ZB: Intermediate node ZC: Leaf node

Claims

1. An index management device for a database storing multidimensional data including quantities whose order and distance between two values ​​can be properly defined and unordered discrete values ​​whose order and distance between two values ​​cannot be properly defined, comprising: an index generating means for generating an index of a hierarchical structure consisting of a plurality of nodes; an entry processing means for performing a process of setting a range search when the key information is the quantity and setting a specified value search to which a Bloom filter is applied when the key information is the unordered discrete value, when performing a process of storing a pointer and key information in the node; an update processing means for updating the nodes by repeating a process of calculating a smallest positive integer P S for a plurality of the multidimensional data under a predetermined condition, rearranging the key information, and dividing the data based on the positive integer P S ; An index management device characterized in that, when rearranging in the update processing means, if the key information is the unordered discrete value, the presence or absence of a bit that is 1 is checked for each specified group, and the magnitude relationship is determined based on a 2-bit bit string in which presence is represented by 1 and absence is represented by 0.

2. 2. The index management device according to claim 1, wherein the magnitude relationship between two Bloom filters (BX, BY) whose number of bits N is 2^n (n: an integer equal to or greater than 1) is determined by the following procedure. (Step 1) Initialize the variable KK to n-1. (Step 2) For each of BX and BY, check whether there is a bit that is 1 for every 2^KK bits, and let BX' and BY' be 2-bit bit strings in which presence is represented by 1 and absence is represented by 0. (Step 3) BX'' and BY'' are the bit strings obtained by performing a 1-bit right shift operation on BX' and BY', respectively. (Step 4) The result of the exclusive OR (XOR operation) of BX' and BX'' is BX''', and the result of the exclusive OR of BY' and BY'' is BY'''. (Step 5) When viewed as binary numbers, if BX''' > BY''', determine that BX > BY, if BX''' < BY''', determine that BX < BY, and if neither applies, if KK=0, determine that BX=BY, and if KK≠0, set KK-1, and repeat from (Step 2).

Images