Integrated astronomical data acquisition and storage system based on persistent memory

Abstract

The invention discloses an integrated astronomical data acquisition and storage system based on a persistent memory. The system comprises a data acquisition end, a communication module and a data storage end which are in communication connection in sequence, the data acquisition end is used for acquiring astronomical data observed by the astronomical telescope for persistent storage; the communication module is used for transmitting the astronomical data in buffer to the data storage end after the astronomical data in persistent storage is fully buffered; and the data storage end is used for persistently storing the astronomical data transmitted by the communication module. According to the invention, a persistent storage mode is used to realize high-speed acquisition of the data in the memory and transmission of the data to the storage end, so that the data is not lost in case of abnormality, the reliability of the acquisition system is greatly improved, and the safety of astronomicalobservation data is also improved.

Description

technical field [0001] The invention relates to the technical field of data storage, and more specifically relates to an integrated astronomical data collection and storage system based on persistent memory. Background technique [0002] In recent years, with the rapid development of large-scale telescope equipment manufacturing technology and information technology, astronomy has entered the era of big data driven by data. On the one hand, big data has brought new horizons and new research methods to scientific research. On the one hand, it also brings huge challenges to basic information technologies such as data storage and processing that support scientific research. As a key link, data storage directly restricts the front-end data collection and back-end data processing, and plays an important role as a link. [0003] At present, the front-end acquisition system of the domestic acquisition server runs on the Windows platform, while the distributed storage system uses t...

AI Technical Summary

Problems solved by technology

Since the file sharing protocol largely depends on the architecture of the operating system platform itself, the data storage performance of the acquisition system will inevitably be affected by the protocol

Although the CIFS method can expand the performance through the CTDB cluster, it will make the management of the collection server very complicated.

Therefore, other methods have to be used instead. At present, the most suitable technology for high-speed communication between platforms is Remote Memory Access-RDMA (Remote Direct Memory Access). Persistence capability, when an exception occurs, such as a sudden hardware failure, data will be lost

In the case of massive, high-speed, long-term persistent storage, even if it is an exception at the second level, this problem will cause a large amount of observation data to fail to be saved

Although NVST technicians try to write custom programs and components to solve the performance barriers in the transmission process, they cannot avoid the inherent characteristics of TCP / IP, I / O and system cache at the operating system level, so they all face problems similar to CIFS. The difficulty of low performance of cross-platform high-speed data transmission

Images