A Distributed Equivalent Data Stream Connection Method

Abstract

The invention discloses a distributed equivalent data stream connection method. The method comprises the following steps: cost model calculation used for connecting a partition set with the smallest cost; data distribution used for transmitting data streams which need to be connected to a same task, and distributing the task to a node for calculation, so that the data streams which need to be connected are calculated at the same node; and data connection used for constituting a relational data model by the received data, connecting and load shedding the data through a relational tree model. The distributed equivalent data stream connection method provided by the invention is beneficial to reducing the maintenance of an intermediate result and the waste of computer resources, reducing time overhead during data connection, reducing memory overhead, guaranteeing the stable operation of a system and increasing the possibility of successful data stream connection.

Description

technical field [0001] The invention relates to the technical field of distributed data stream management, in particular to a method for connecting distributed equivalent data streams. Background technique [0002] In recent years, with the explosive growth of data volume, traditional databases have shown relatively large limitations, and the analysis of massive data has become a new topic. Among them, streaming data is very important for data analysis due to its characteristics of infinity, real-time, and variability. presented new challenges. As early as 2002, Babcock proposed a streaming data management system for streaming data. Typical prototype systems include TelegraphCQ, Aurora, STREAM and other management systems. In these stream management systems, stream connection calculation has always been one of the very important data processing methods due to its complex calculation and large resource consumption. [0003] Over the past decade, streaming connectivity compu...

AI Technical Summary

Problems solved by technology

However, the operation of stream connection computing in a distributed environment has brought new problems: (1) Due to the interdependence between different data streams, in a distributed environment, distributing data to different nodes will lead to (2) The intermediate results of the multi-stream connection are huge and difficult to maintain, which poses a big challenge to the memory; (3) The processing pressure of a single node is high, and effective measures are required to ensure the results when the node is overloaded the accuracy of

[0004] Traditional solutions focus on the realization of data stream connections, using multi-way connection methods based on two-way connection calculations, maintaining the intermediate results of more connections, ignoring memory overhead, low data connection efficiency, or using load reduction The method does not fully consider the characteristics of the stream connection, resulting in a large error in the data connection

Images