Flink-based hydrological sensor data analysis system and construction method thereof

Abstract

The invention discloses a hydrological sensor data analysis system based on Flink and a construction method thereof. The construction method comprises the following steps: (1) installing and configuring Apache Flink, Apache Kafka and Apache IoTDB; (2) performing format conversion on the hydrological flow data of different sensors in a data sending layer, and sending the hydrological flow data to an upstream message transmission layer; (3) connecting the data sending layer and the data processing layer in an upstream message transmission layer by using Kafka and adopting an asynchronous message transmission mode; (4) in a data processing layer, performing statistical analysis on the data of the hydrological sensor by using Flink; (5) connecting the data processing layer, the persistence layer and the visualization layer in a downstream message transmission layer by using Kafka and adopting an asynchronous message transmission mode; (6) in a persistence layer, performing persistence on an analysis result by utilizing IoTDB; and (7) displaying a real-time analysis result and querying a historical analysis result in a visualization layer. According to the invention, real-time analysis of high-speed unbounded hydrological sensor data is realized.

Description

technical field [0001] The invention relates to a Flink-based hydrological sensor data analysis system and a construction method thereof, belonging to the technical field of stream data processing and statistical analysis. Background technique [0002] As an important part of the water conservancy research and construction process, the hydrological sensor collects and processes the hydrological data and plays a vital role in flood and drought prevention and water damage prevention. Usually, hydrological sensors collect data at certain time intervals, and send it to the analysis platform in the form of stream data, and finally form a hydrological time series for storage and calculation. Statistical analysis is the basis for studying hydrological sensor data. Currently, many hydrological analyses are based on the assumption that the data has certain characteristics. It is expected that the deviation of these assumptions from the actual situation will not affect the analysis re...

AI Technical Summary

Problems solved by technology

Hydrological sensor data has the characteristics of large volume and fast flow rate, so the memory resources of a single computer often cannot meet the demand, and it is necessary to continuously store new data into the database to free up memory space, and then take out the data for calculation

This process brings additional time overhead. When the calculation speed is lower than the flow rate of streaming data, the analysis results will not be given in time.

In addition, the current mainstream big data frameworks Spark and Flink do not provide good support for statistical analysis in streaming scenarios

Images