Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Flink-based hydrological sensor data analysis system and construction method thereof

A technology of a data analysis system and a construction method, which is applied to the hydrological sensor data analysis system and its construction field, and can solve the problems of fast flow rate, good support for statistical analysis, and inability to give analysis results in a timely manner.

Pending Publication Date: 2022-07-29
HOHAI UNIV
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Hydrological sensor data has the characteristics of large volume and fast flow rate, so the memory resources of a single computer often cannot meet the demand, and it is necessary to continuously store new data into the database to free up memory space, and then take out the data for calculation
This process brings additional time overhead. When the calculation speed is lower than the flow rate of streaming data, the analysis results will not be given in time.
In addition, the current mainstream big data frameworks Spark and Flink do not provide good support for statistical analysis in streaming scenarios

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Flink-based hydrological sensor data analysis system and construction method thereof
  • Flink-based hydrological sensor data analysis system and construction method thereof
  • Flink-based hydrological sensor data analysis system and construction method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0065] Below in conjunction with specific embodiments, the present invention will be further illustrated, and it should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. The modifications all fall within the scope defined by the appended claims of this application.

[0066] like figure 1 As shown, a Flink-based hydrological sensor data analysis system includes: a data sending layer, an upstream message transmission layer, a data processing layer, a downstream message transmission layer, a persistence layer, and a visualization layer, which are arranged in order from top to bottom.

[0067] The data sending layer is used to collect the stream data from different hydrological sensors, convert the data into a unified format, and aggregate it into a data stream and send it to the upstream message transmission layer to provide the system with a data source in a unified format.

[0068] The up...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a hydrological sensor data analysis system based on Flink and a construction method thereof. The construction method comprises the following steps: (1) installing and configuring Apache Flink, Apache Kafka and Apache IoTDB; (2) performing format conversion on the hydrological flow data of different sensors in a data sending layer, and sending the hydrological flow data to an upstream message transmission layer; (3) connecting the data sending layer and the data processing layer in an upstream message transmission layer by using Kafka and adopting an asynchronous message transmission mode; (4) in a data processing layer, performing statistical analysis on the data of the hydrological sensor by using Flink; (5) connecting the data processing layer, the persistence layer and the visualization layer in a downstream message transmission layer by using Kafka and adopting an asynchronous message transmission mode; (6) in a persistence layer, performing persistence on an analysis result by utilizing IoTDB; and (7) displaying a real-time analysis result and querying a historical analysis result in a visualization layer. According to the invention, real-time analysis of high-speed unbounded hydrological sensor data is realized.

Description

technical field [0001] The invention relates to a Flink-based hydrological sensor data analysis system and a construction method thereof, belonging to the technical field of stream data processing and statistical analysis. Background technique [0002] As an important part of the water conservancy research and construction process, the hydrological sensor collects and processes the hydrological data and plays a vital role in flood and drought prevention and water damage prevention. Usually, hydrological sensors collect data at certain time intervals, and send it to the analysis platform in the form of stream data, and finally form a hydrological time series for storage and calculation. Statistical analysis is the basis for studying hydrological sensor data. Currently, many hydrological analyses are based on the assumption that the data has certain characteristics. It is expected that the deviation of these assumptions from the actual situation will not affect the analysis re...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2458G06F16/2455G06F16/2457G06F16/248H04L47/50
CPCG06F16/2474G06F16/248G06F16/24552G06F16/24578H04L47/50
Inventor 潘晗韬叶枫陈龙金光球兰林
Owner HOHAI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com