Distributed similarity join method for data streams based on emd distance

Abstract

Distributed parallel computing is an effective means to improve the efficiency of data flow similarity connection processing. EMD distance can more accurately quantify the similarity between histogram tuples, but it has a computational complexity of up to cubic, which hinders its similarity in data flow. For this reason, the present invention discloses a data flow distributed similarity connection method based on EMD distance, based on the open source data flow distributed parallel computing framework Apache Storm, and adopts data locality-based data in data distribution The flow partition strategy maintains the data locality on the connected computing nodes, and based on the data locality, the connection algorithm’s filtering performance for the EMD calculation between dissimilar histogram tuple pairs is enhanced, and the execution efficiency of each connected computing node is improved; at the same time Based on the cost model of connecting computing nodes, the feedback-based load balancing strategy is adopted to effectively improve the overall processing performance and system scalability of the system for data streams, which improves the processing throughput by up to 50% compared with the existing technology.

Description

technical field [0001] The invention belongs to the technical field of data flow similarity connection, in particular to a data flow distributed similarity connection method based on EMD distance. Background technique [0002] With the continuous improvement of data acquisition equipment and the rapid development of data acquisition technology, how to conduct high-quality data analysis on the continuous and rapidly generated data stream has become a common concern in the industry and academia. Data stream similarity join returns similar tuple pairs on two data streams, which is an important operation for analyzing and mining data streams, and is widely used in practical applications such as event detection and data deduplication. [0003] The similarity connection between data streams composed of data tuples represented by the histogram data model (referred to as histogram tuples for short), each histogram tuple can be formally expressed as P={p 1 ,...,p n}, where p i ∈[0...

AI Technical Summary

Problems solved by technology

Due to the high computational complexity of EMD distance, the number of EMD distance calculations that actually occur when nodes perform similarity connections will be the focus of the connection cost model. Therefore, the above research work is not suitable for solving distributed similarity based on EMD distance on data streams. sexual connection problems

Although the EMD-MPJ algorithm and the Top-k DLPJ algorithm design the data similarity connection scheme in the distributed parallel computing environment based on the characteristics of the EMD distance, they all deal with static data sets, and their load balancing strategies are not suitable for dynamic changes. data flow environment

Images