Collection system and method of environment quality online monitoring data

Abstract

The invention discloses a collection system and method of environment quality online monitoring data. The collection system comprises a data collection client, a data receiving server-side, a data computing center and a data storage center. The technical scheme provided by the invention takes a Netty cluster and a Kafka message queue cluster as the data receiving server-side, so that the collection has high availability and can support a data concurrence request of over-ten-thousands stations, the data integrity is guaranteed, and the pressure on the data computing center by a client request is delayed; the data can be analyzed and checked in real time by using Redis cache equipment information, station information, data rule and like data and using the Flink cluster as a computing engine;the Hbase cluster is used as a database so that the system can satisfies mass data storage. The technical scheme disclosed by the invention greatly improves the data receiving capacity and the data computing capacity.

Description

technical field [0001] The invention relates to the application field of big data for environmental protection, in particular to an online monitoring data collection system and method for environmental quality. Background technique [0002] With the increasingly prominent ecological and environmental problems, it is necessary to promote the comprehensive integration and sharing of data resources, improve data acquisition capabilities, innovate data collection methods, and improve comprehensive perception and real-time monitoring capabilities of various environmental elements and various pollution sources. However, the current real-time monitoring data collection of environmental quality monitoring stations is still at a relatively traditional stage. For example, the existing collection system directly uses the socket server to receive monitoring station client data. When the number of concurrent data sending requests is large, it is easy to Cause blocking; use ordinary java ...

AI Technical Summary

Problems solved by technology

However, the current real-time monitoring data collection of environmental quality monitoring stations is still at a relatively traditional stage. For example, the existing collection system directly uses the socket server to receive monitoring station client data. When the number of concurrent data sending requests is large, it is easy to Cause blockage; use ordinary java algorithm to analyze and verify the monitoring data, and use ordinary relational database to store the monitoring data. When the amount of data is relatively large, the acquisition system cannot meet the computing and storage requirements

When a function of the acquisition system is abnormal, it is easy to cause the entire system to function abnormally and cannot receive data, resulting in the loss of all data sent by the environmental quality monitoring station before the system is restored.

With the continuous development of environmental protection business and the increasing number of environmental quality monitoring sites, the traditional collection system cannot handle the increasing amount of data

To sum up, the existing data acquisition system can no longer meet the actual demand for environmental quality data acquisition, and there is an urgent need for a more professional data acquisition system that can support the business needs of big data

Images