System and method for collecting automatic weather station data

Abstract

The present invention relates to a system and a method for collecting automatic weather station data. The system comprises a message production module, a message queue module and a message consumption module; the message production module and the message consumption module are independent from each other; the message production module is provided with a compatible interface connected with different automatic weather stations; and the message consumption module is provided with a data analysis class and a data warehouse class that are matched with different automatic weather stations. According to the system for collecting automatic weather station data disclosed by the present invention, a proprietary system is constructed for the collection work of the automatic weather station data, more automatic station data formats are supported, the compatibility with vendors of different databases is increased, persistence processing is carried out on the collected data, the risk of data loss is reduced, human intervention in the data collection process is reduced, and the hourly peak pressure is buffered, so that even by using the database with lower configuration, the hourly peak can be processed stably.

Description

technical field [0001] The invention relates to the fields of meteorological science and big data technology application, in particular to a system and method for collecting data from automatic weather stations. Background technique [0002] At present, the meteorological department mainly requests data from the data interface of the automatic weather station and writes it into the database by writing programs in various projects in the aspect of automatic weather station data collection. Due to the large number of automatic weather stations, all station data are updated at the same time on the hour, and data write and read requests peak at the hour, resulting in an increased burden on the database and slow access at the hour. The current solution is to deal with the peak by allocating more CPU and memory on the database hardware resources; in addition, since the data request and storage are completed by the same program, for the frequent abnormal stop of the program, it can...

AI Technical Summary

Problems solved by technology

Due to the large number of automatic weather stations, all station data are updated at the same time on the hour, and the data write and read requests peak at the hour, resulting in an increased burden on the database and slow access

The current solution is to deal with the peak by allocating more CPU and memory on the database hardware resources; in addition, since the data request and storage are completed by the same program, for the frequent abnormal stop of the program, it can only be done manually Analyze the log collection of the collection program to understand the time when the interruption occurred, and run the data supplementary recording program at an appropriate time to fill in the missing data when the program is abnormal; at the same time, there is no automatic weather station data collection in China that can complete a large amount of automatic station data The proprietary system of collection and the data collection of automatic weather stations are only a subsidiary task of independent project development. During the project development process, different manufacturers use different technical means to implement repeated temporary development.

[0003] From the analysis of the existing technical solutions, there are the following defects as a whole: 1. The solution puts the request and storage of the automatic weather station data in the same module, which leads to a high degree of coupling between the collection program and the database manufacturer. The database cannot be transplanted; 2. In some cases, an exception may occur during the process of requesting data or data storage. This solution does not persist the data, which will cause data loss. The lost data can only be supplemented by manual intervention. 3. This solution directly transfers the peak pressure of the whole point to the database, forcing the database to allocate resources with the peak access volume, resulting in waste of resources; 4. The collection program has been repeatedly developed in different projects, but they are all limited to temporary solutions The analysis and warehousing of one or two automatic weather station data formats encountered in the paper did not provide good support for the automatic weather station data itself

Images