Multi-path data flow connection system based on data inclination

Abstract

The invention discloses a multipath data flow connection system based on data inclination. A data acquisition module is connected to the input of the preprocessing module through the data caching module; output of a preprocessing module is connected to input of the conversion module through different windows, output of a conversion module is connected to input of the connection module through different windows, output of a connection module is connected to the result storage module through the expansion module, and output of the expansion module is connected to the operation interface. Beforedata streams are connected, tuples with the same key value in the data streams are aggregated into the same tuple model and then sent to the connection module, so that a subsequent distributed systemcan process the tuples in the same tuple model at the same time. According to the method, the characteristics of the streaming data in the sliding window can be counted, the connection of multiple paths of data streams is realized, the problem of key value redundancy in the connection process of the multiple paths of data streams is solved, and the connection optimization of the multiple paths ofdata streams is carried out by analyzing the characteristics of the streaming data and combining the invented group connection algorithm.

Description

technical field [0001] The invention relates to a data stream connection system and processing method, in particular to a system for performing multi-channel data stream connection according to the inclination degree of stream data. Background technique [0002] Today, with the further development of high-tech, especially the development of technologies such as artificial intelligence, cloud computing, and the Internet of Things, data has shown explosive growth. The computing and analysis services of massive data affect all aspects of society and serve the public. And provide services for the operational decision-making of enterprises. Under normal circumstances, these data are large in volume, strong in timeliness, and complex in structure. How to obtain accurate and comprehensive information in a timely manner from massive real-time data has become a hot direction in current big data research. [0003] Currently, data processing solutions for real-time data streams mainly...

AI Technical Summary

Problems solved by technology

[0008] However, there are still many deficiencies and limitations in the above-mentioned prior art.

[0009] On the one hand, existing technical solutions for streaming big data processing such as Spark, Flink, Storm, etc. do not connect multiple data streams based on data skew, or are only suitable for equivalent connections but not for non-equivalent connections, or For the connection sequence, only for three or more data stream connections

[0010] On the other hand, with the large-scale growth of data volume and the complexity of query structure, it has been found in practice that when querying massive streaming data, it is necessary to perform query and connection operations on multiple data streams in the shortest time To obtain complete and accurate results, there is no solution to improve the efficiency of join queries by reducing key-value redundancy in existing solutions

Images