A distributed index construction method based on improved iSAX encoding

Abstract

The application provides a distributed index construction method based on improved iSAX coding, first, aiming at the digital feature selection problem, in order to increase the coding similarity of similar data, a similarity digital coding is designed, and through the transposition of the matrix, the problem of different weights of high bits and low bits of each digital coding in the whole digital coding is solved. Secondly, aiming at the problem of high time consumption in the traversal process caused by the too high index tree, a B+ index tree based on the similarity digital coding is designed, the number of child nodes is increased, the height of the tree is reduced, and the access speed of adjacent nodes is improved. And a leaf partition snake-shaped packing algorithm is designed, which guarantees the load balance, shortens the leaf node packing time, improves the index construction speed, and improves the overall index construction speed compared with the traditional distributed index construction algorithm, and provides a more efficient index framework for the similarity query process.

Description

A Distributed Index Construction Method Based on Improved iSAX Encoding Technical Field

[0001] This invention belongs to the field of computer time-series data management technology, specifically relating to a distributed index construction method based on improved iSAX encoding. Background Technology

[0002] In recent years, data mining technology has seen rapid development, and similarity querying, as a foundation of data mining, has attracted extensive research. Indexing technology, with its rapid location capabilities, has become a crucial guarantee for achieving similarity queries and improving query speed. Due to the massive volume of data currently available, single-machine indexing is no longer sufficient to meet current query demands; therefore, utilizing distributed indexes to implement similarity queries has become a popular approach.

[0003] To reduce the storage space occupied by the index frame, researchers generally convert data into iSAX encoded form and use the characteristics of binary number encoding to construct a binary index tree. The iSAX encoding method first divides the space along the y-axis into different regions and assigns multiple binary numbers as symbols for each region. Then, it divides the data sequence into segments of equal length, calculates the average value of all points in each segment, and obtains the Piecewise Aggregate Approximation (PAA). Finally, it aggregates the symbols of all PAA regions to obtain the iSAX code of the data sequence.

[0004] Using iSAX encoding to build index trees results in highly similar data being stored on different subtrees, increasing the time cost of similarity queries. Current partitioning algorithms based on encoding, while ensuring load balancing among slave nodes, suffer from long partition packaging times and the storage of similar data across different slave nodes, impacting query efficiency and user experience. Therefore, a distributed indexing method that supports efficient similarity queries for time-series data is becoming increasingly important. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a distributed index construction method based on improved iSAX encoding, comprising:

[0006] Step 1: Construct similarity numerical codes, the specific process is as follows:

[0007] Step 1.1: The master node in the distributed system divides the y-axis into multiple segments, each represented by a binary number, and sends the segmentation results of the y-axis to each slave node;

[0008] Step 1.2: The master node distributes the data in the Distributed File Storage System (HDFS) according to the number of slave nodes, assigning the original time-series data to different slave nodes;

[0009] Step 1.3: After receiving the y-axis segmentation result and the address of the original data sequence from the node, read the original data in HDFS and divide each piece of original data into segments on an average basis;

[0010] Step 1.4: After segmenting, calculate the average value within each segment. Record the corresponding binary number as the iSAX code of the segment according to the range of the average value on the y-axis. Store multiple iSAX codes in the slave node in the form of a two-dimensional array, and convert the two-dimensional array into a matrix, called the digital encoding matrix.

[0011] Step 1.5: After all the initial digital codes are calculated from the nodes, the digital code matrix within the nodes is transposed. The resulting matrix is ​​called the similarity code matrix. The similarity code matrix is ​​written out in binary form to obtain the similarity digital code.

[0012] Step 1.6: After all similarity numeric codes are calculated, each slave node converts the obtained similarity numeric codes into a tuple of <similar numeric code, original file address> and stores it in HDFS. After all slave nodes have stored the files, the master node merges the tuple files sent by each slave node.

[0013] Construct a distributed index structure consisting of a global index and a local index, and establish links between the local index and the global index.

[0014] Step 2: Construct a global index, specifically as follows:

[0015] Step 2.1: The master node first performs MapReduce operation on the data in the merged similarity number encoding file, counts the number of identical number codes, and forms a new tuple <similar number codes, number>.

[0016] Step 2.2: The master node constructs the global index framework using a B+ tree. The index tree framework is built based on the number of rows and columns of the similarity coding matrix. For an m x n similarity coding matrix, the global index tree has m+1 levels, and each node has 2 child nodes. n indivual.

[0017] Step 2.3: After the global index tree framework within the master node is constructed, the global index tree is loaded. The loading method is from top to bottom. According to the high digit to low digit of the similarity numerical code, the <similarity numerical code, number> tuple is loaded into the corresponding leaf node.

[0018] Step 2.4: Perform leaf partitioning and packaging on the leaf nodes of the global index tree, and then allocate the packaged nodes to different slave nodes. Assuming that all slave nodes have the same storage and processing capabilities, a serpentine algorithm is used for leaf partitioning and packaging, specifically as follows:

[0019] Step 2.4.1: Pack all the data of all child nodes of each smallest parent node into a single package. The similarity coding matrix is ​​an m x n column global index tree constructed from the time-series dataset. The packaged result is 2. (m-1)n One package;

[0020] Step 2.4.2: When allocating these packets, a serpentine algorithm is used to sort the packets in descending order of size. The t slave nodes are numbered. In each allocation, the packet with the largest amount of data among the currently unallocated packets is assigned to the slave node. In the first round of allocation, the allocation order starts from slave node 1 and proceeds in ascending order of number until slave node t. In the second round of allocation, the allocation order starts from slave node t and proceeds in descending order of number until slave node 1. The allocation order in the third round is the same as the first round, the allocation order in the fourth round is the same as the second round, and so on, until all packets are allocated. At the same time as allocation, the information of the pre-allocated slave nodes is recorded in the corresponding leaf nodes, and connections are established.

[0021] Step 2.4.3: After partitioning, distribute the packaged data to the slave nodes.

[0022] Step 3: After receiving data from the node, construct the local index and establish a link between the local index and the global index. The specific process is as follows:

[0023] Step 3.1: After the leaf partition is packaged, each slave node constructs a local index and loads the encoded data of the received data;

[0024] The local index is constructed by building a framework for the local index tree based on the similarity numeric codes in the received <similarity numeric codes, number> tuples. The local index tree is constructed in the form of a B+ tree; the specific process is as follows:

[0025] Step 3.1.1: Each slave node counts the first n bits of each received similarity numeric code, builds the first layer of nodes based on the count of the first n bits, then counts the first 2n bits of the similarity numeric code, locates the intermediate node corresponding to the first n bits based on the count of the first n bits, creates a child node under the intermediate node, and names the node as the (n+1)th to the 2nth bit of the numeric code, until all bits are counted, completing the framework construction; the loading process is the same as the global index data loading process;

[0026] Step 3.1.2: After the local index tree framework within the node is constructed, the local index tree is loaded. The loading method is from top to bottom. According to the high-order digit of the similarity numerical code, the <similarity numerical code, number> tuple is loaded into the corresponding leaf node.

[0027] Step 3.2: Materialize the leaf nodes in the local index; match the <similarity numeric code, number> in the leaf node of the local index with the <similarity numeric code, original file address> in HDFS, and store the <similarity numeric code, original file address> tuple in the leaf node to complete the materialization process of the leaf node;

[0028] Step 3.3: Access the leaf nodes of the global index tree, write the leaf node values ​​and corresponding local index tree addresses into the mapping table, and store them in the memory of the master node. At this point, the global index and local index are completed, and similarity queries of time-series data can be supported.

[0029] The beneficial effects of this invention are:

[0030] 1. The similarity numerical encoding representation method provided by this invention addresses the problem of encoding and representing similar data, and can achieve similar representation of similar data.

[0031] 2. The distributed index structure proposed in this invention consists of a global index and local indexes, and connects the local indexes with the global index. Compared with traditional index construction methods, the B+ index tree reduces the height of the index tree, shortening the traversal time of the index tree during similarity queries.

[0032] 3. This invention proposes a leaf partitioning serpentine packing algorithm for global indexes, which distributes similar data to the same slave node, improves the index building speed, increases the data similarity within the local index, thereby speeding up similarity queries and improving the accuracy of similarity queries. Attached Figure Description

[0033] Figure 1 is a schematic diagram of the present invention converting time-series data into similarity digital codes;

[0034] Figure 2 is a flowchart of the index construction process of the present invention;

[0035] Figure 3 is a flowchart of the similarity digital coding statistical stage of the present invention;

[0036] Figure 4 is a schematic diagram of the global index structure of the present invention;

[0037] Figure 5 is a schematic diagram of the partial index structure of the present invention. Detailed Implementation

[0038] The invention will be further explained below with reference to the accompanying drawings and specific implementation examples.

[0039] This invention addresses the problem of building distributed indexes supporting similarity queries by proposing a distributed index construction method based on improved iSAX encoding. First, regarding the issue of numeric feature selection, a similarity-based numeric encoding is designed to increase the encoding similarity of similar data. By transposing the matrix, the problem of different weights for the high and low bits of each numeric segment within the overall numeric encoding is resolved. Second, to address the issue of excessive time consumption during traversal due to excessively tall index trees, a B+ index tree based on similarity-based numeric encoding is designed. By increasing the number of child nodes, the tree height is reduced, improving the access speed to adjacent nodes. Furthermore, a leaf partitioning serpentine packing algorithm is designed to ensure load balancing while shortening the leaf node packing time, thus improving the index construction speed. Compared to traditional distributed index construction methods, this method significantly improves the overall index construction speed, providing a more efficient index framework for similarity queries.

[0040] This invention provides a distributed index construction method based on improved iSAX encoding, which constructs a distributed index for 26 time series data; the method includes the following steps:

[0041] Step 1: Construct similarity numerical codes, the specific process is as follows:

[0042] Step 1.1: The master node in the distributed system divides the y-axis into multiple segments, each represented by a binary number, and sends the segmentation results of the y-axis to each slave node;

[0043] Step 1.2: The master node distributes the data in the Distributed File Storage System (HDFS) according to the number of slave nodes, assigning the original time-series data to different slave nodes;

[0044] Step 1.3: After receiving the y-axis segmentation result and the address of the original data sequence from the node, read the original data in HDFS and divide each piece of original data into segments on an average basis;

[0045] Step 1.4: After segmenting, calculate the average value within each segment. Record the corresponding binary number as the iSAX code of the segment according to the range of the average value on the y-axis. Store multiple iSAX codes in the slave node in the form of a two-dimensional array, and convert the two-dimensional array into a matrix, called the digital encoding matrix.

[0046] Step 1.5: After all the initial digital codes are calculated from the nodes, the digital code matrix within the nodes is transposed. The resulting matrix is ​​called the similarity code matrix. The similarity code matrix is ​​written out in binary form to obtain the similarity digital code.

[0047] In this embodiment, the time series data shown in Figure 1 is converted by similarity digital encoding to obtain the similarity digital encoding of the time series as 0110 1001 0010.

[0048] Step 1.6: After all similarity numeric codes are calculated, each slave node converts the obtained similarity numeric codes into a tuple of <similar numeric code, original file address> and stores it in HDFS. After all slave nodes have stored the files, the master node merges the tuple files sent by each slave node.

[0049] A distributed index structure is constructed, consisting of a global index and a local index. The construction process of the global index and the local index is shown in Figure 2.

[0050] Step 2: First, construct the global index, specifically as follows:

[0051] Step 2.1: The master node first performs MapReduce operation on the data in the merged similarity number encoding file, counts the number of identical number codes, and forms a new tuple <similar number codes, number>.

[0052] In this embodiment, the calculation process and binary form of the similarity encoding are shown in Figure 3.

[0053] Step 2.2: The master node constructs the global index framework using a B+ tree. The index tree framework is built based on the number of rows and columns of the similarity coding matrix. For an m x n similarity coding matrix, the global index tree has m+1 levels, and each node has 2 child nodes. n indivual.

[0054] In this embodiment, the similarity coding matrix has 3 rows, and each row has 4 bits. Therefore, every node in the framework except the leaf nodes has 2 bits. 4 There are 4 child nodes, named from 0000 to 1111. This B+ global index tree has 4 levels.

[0055] Step 2.3: After the global index tree framework within the master node is constructed, the global index tree is loaded. The loading method is from top to bottom. According to the high digit to low digit of the similarity numerical code, the <similarity numerical code, number> tuple is loaded into the corresponding leaf node.

[0056] In this embodiment, after the master node reads the tuple to be loaded, it compares the high four bits of the similarity code with the intermediate nodes of the first level of the global index tree. After finding the corresponding node, it compares the middle four bits of the similarity code with the child nodes of that node to find the intermediate nodes of the second level. Finally, it compares the low four bits of the similarity code with the child nodes of the located intermediate node of the second level to find the corresponding leaf node and saves the tuple <similarity code, number> into the leaf node. The result of the loading process is shown in Figure 4. At this point, the global index tree is complete.

[0057] Step 2.4: Perform leaf partitioning and packaging on the leaf nodes of the global index tree, and then allocate the packaged nodes to different slave nodes. Assuming that all slave nodes have the same storage and processing capabilities, a serpentine algorithm is used for leaf partitioning and packaging, specifically as follows:

[0058] Step 2.4.1: Pack all the data of all child nodes of each smallest parent node into a single package. The similarity coding matrix is ​​an m x n column global index tree constructed from the time-series dataset. The packaged result is 2. (m-1)n One package;

[0059] Step 2.4.2: When allocating these packets, a serpentine algorithm is used to sort the packets in descending order of size. The t slave nodes are numbered. In each allocation, the packet with the largest amount of data among the currently unallocated packets is assigned to the slave node. In the first round of allocation, the allocation order starts from slave node 1 and proceeds in ascending order of number until slave node t. In the second round of allocation, the allocation order starts from slave node t and proceeds in descending order of number until slave node 1. The allocation order in the third round is the same as the first round, the allocation order in the fourth round is the same as the second round, and so on, until all packets are allocated. At the same time as allocation, the information of the pre-allocated slave nodes is recorded in the corresponding leaf nodes, and connections are established.

[0060] Step 2.4.3: After partitioning, distribute the packaged data to the slave nodes.

[0061] Step 3: After receiving data from the node, construct the local index and establish a link between the local index and the global index. The specific process is as follows:

[0062] Step 3.1: After the leaf partition is packaged, each slave node constructs a local index and loads the encoded data of the received data;

[0063] The local index is constructed by building a framework for the local index tree based on the similarity numeric codes in the received <similarity numeric codes, number> tuples. The local index tree is constructed in the form of a B+ tree; the specific process is as follows:

[0064] Step 3.1.1: Each slave node counts the first n bits of each received similarity numeric code, builds the first layer of nodes based on the count of the first n bits, then counts the first 2n bits of the similarity numeric code, locates the intermediate node corresponding to the first n bits based on the count of the first n bits, creates a child node under the intermediate node, and names the node as the (n+1)th to the 2nth bit of the numeric code, until all bits are counted, completing the framework construction; the loading process is the same as the global index data loading process;

[0065] Step 3.1.2: After the local index tree framework within the node is constructed, the local index tree is loaded. The loading method is from top to bottom. According to the high-order digit of the similarity numerical code, the <similarity numerical code, number> tuple is loaded into the corresponding leaf node.

[0066] In this embodiment, as shown in Figure 5, the similarity numerical codes 1111 1100 1001 and 1011 1101 1001 are assigned to the same slave node. First, the first four digits are counted, and two intermediate nodes, 1111 and 1101, are created under the root node. Then, the first eight digits of the codes are counted, resulting in 1111 1100 and 1011 1101. The intermediate node 1111 is located first, and then a child node 1100 is created under this node. Similarly, a child node 1101 is created under the node 1011. Finally, leaf nodes 1001 and 1001 are created under the 1100 and 1101 nodes in the second-level intermediate nodes, respectively, according to the path. At this point, the B+ local index framework is fully constructed, and data is loaded into the leaf nodes according to the corresponding similarity codes.

[0067] Step 3.2: Materialize the leaf nodes in the local index; match the <similarity numeric code, number> in the leaf node of the local index with the <similarity numeric code, original file address> in HDFS, and store the <similarity numeric code, original file address> tuple in the leaf node to complete the materialization process of the leaf node;

[0068] Step 3.3: Access the leaf nodes of the global index tree, write the leaf node values ​​and corresponding local index tree addresses into the mapping table, and store them in the memory of the master node. At this point, the global index and local index are completed, and similarity queries of time-series data can be supported.

Claims

1. A distributed index construction method based on improved iSAX encoding, characterized in that, include: Step 1: Construct similarity numerical codes; Step 2: Construct the global index in the distributed index structure; Step 3: After receiving data from the nodes, construct the local index in the distributed index structure and establish the link between the local index and the global index; Step 2 includes: Step 2.1: The master node first performs MapReduce operation on the data in the merged similarity numerical code file, counts the number of identical numerical codes, and forms a new tuple <similar numerical code, number>; Step 2.2: The master node constructs the global index framework, using a B+ tree to construct the global index tree; the framework construction process of the index tree is based on the number of rows and columns of the similarity coding matrix. For an m x n similarity coding matrix, the global index tree has m+1 levels, and the number of child nodes of each node is... Step 2.3: After the global index tree framework within the master node is constructed, the global index tree is loaded. The loading method is from top to bottom. According to the high-order value of the similarity numerical code from low to high, the <similarity numerical code, number> tuple is loaded into the corresponding leaf node. Step 2.4: Perform leaf partitioning and packaging of the leaf nodes of the global index tree, and then assign them to different slave nodes.

2. The distributed index construction method based on improved iSAX encoding according to claim 1, characterized in that, Step 1 includes: Step 1.1: The master node in the distributed system divides the y-axis into multiple segments, each represented by a binary number, and sends the y-axis segmentation results to each slave node; Step 1.2: The master node distributes the data in the HDFS distributed file storage system according to the number of slave nodes, allocating the original time-series data to different slave nodes; Step 1.3: After receiving the y-axis segmentation results and the original data sequence address, the slave node reads the original data in HDFS and divides each original data segment equally; Step 1.4: After segmentation, the average value within each segment is calculated, and the corresponding binary number within the y-axis range of the average value is recorded as the iSAX code of that segment of data, and multiple iSAX codes are... The X code is stored in the slave node as a two-dimensional array, and the two-dimensional array is converted into a matrix called the digital encoding matrix; Step 1.5: After the slave node has completed all the calculations of the initial digital code, the digital encoding matrix in the slave node is transposed, and the new matrix is ​​called the similarity encoding matrix. The similarity encoding matrix is ​​written in binary form to obtain the similarity digital code; Step 1.6: After all the similarity digital codes have been calculated, each slave node converts the obtained similarity digital code into a tuple of <similar digital code, original file address> and stores it in HDFS. After all the slave nodes have stored it, the master node merges the tuple files sent by each slave node.

3. The distributed index construction method based on improved iSAX encoding according to claim 1, characterized in that, In step 2.4, under the premise that the storage capacity and processing capacity of each slave node are the same, a snake algorithm is used to perform leaf partitioning and packaging.

4. The distributed index construction method based on improved iSAX encoding according to claim 3, characterized in that, Step 2.4 is specifically expressed as follows: Step 2.4.1: Pack all the data of all child nodes of each smallest parent node into a package. The similarity coding matrix is ​​a global index tree constructed from an m-row n-column time-series dataset. The packaged result is... One package; Step 2.4.2: When allocating these packets, a serpentine algorithm is used to sort the packets in descending order of size. The t slave nodes are numbered. In each allocation, the packet with the largest amount of data among the currently unallocated packets is allocated to the slave node. In the first round of allocation, the allocation order starts from slave node 1 and proceeds incrementally by number until slave node t. In the second round of allocation, the allocation order starts from slave node t and proceeds in descending order by number until slave node 1. The allocation order in the third round is the same as the first round, and the allocation order in the fourth round is the same as the second round, until all packets have been allocated. At the same time as allocation, the information of the pre-allocated slave nodes is recorded in the corresponding leaf nodes, and connections are established. Step 2.4.3: After partitioning, the packaged data is allocated to the slave nodes.

5. A distributed index construction method based on improved iSAX encoding according to claim 1, characterized in that, Step 3 includes: Step 3.1: After the leaf partition is packaged, each slave node constructs a local index and loads the encoded data for the received data; Step 3.2: Materialize the leaf nodes in the local index; match the <similarity numeric code, number> in the local index leaf node with the <similarity numeric code, original file address> in HDFS, and store the <similarity numeric code, original file address> tuple in the leaf node to complete the materialization process of the leaf node; Step 3.3: Access the leaf nodes of the global index tree, write the leaf node values ​​in the global index tree and the corresponding local index tree addresses into the mapping table, and store them in the memory of the master node. At this point, the global index and local index are completed, which can support similarity queries of time-series data.

6. A distributed index construction method based on improved iSAX encoding according to claim 5, characterized in that, In step 3.1, the construction of the local index involves building a framework for the local index tree based on the similarity numeric codes in the received <similarity numeric codes, number> tuples. The local index tree is constructed in the form of a B+ tree.

7. A distributed index construction method based on improved iSAX encoding according to claim 6, characterized in that, The specific process of step 3.1 is as follows: Step 3.1.1: Each slave node counts the first n bits of each received similarity numeric code, builds the first layer of nodes according to the count of the first n bits, and then counts the first 2n bits of the similarity numeric code. Based on the count of the first n bits, it locates the intermediate node corresponding to the first n bits, and creates a child node under the intermediate node. The node name is the (n+1)th to the 2nth bit of the numeric code. This process continues until all bits are counted, completing the framework construction. Step 3.1.2: After the local index tree framework within the slave node is completed, the local index tree is loaded. The loading method is from top to bottom. According to the value of the similarity numeric code from the high bit to the low bit, the <similar numeric code, number> tuple is loaded into the corresponding leaf node.

Images