Astronomical metadata archiving method and system based on streaming data processing architecture

Abstract

The invention requests for protecting an astronomical metadata archiving method and system based on a streaming data processing architecture, belonging to the field of astronomical data processing. Firstly, an Agent is deployed in a data source; the Agent monitors the data source in real time; once new data is generated, the monitoring module of the Agent can instantly analyze new observation data, and generate corresponding metadata; secondly, the data acquisition module of the Agent can acquire the metadata generated by the monitoring module in real time, and transmit the metadata to an appointed metadata archiving platform through a network; then, the cache module of the archiving platform can receive the metadata transmitted from the data source in real time, and write the metadata in a distributed cache queue; and finally, the streaming data processing component of the archiving platform can read the metadata from the cache queue in real time, process the metadata into an associated sub-graph in the time-space dimension, and write the associated sub-graph in a graphic database. Due to real-time metadata archiving, the network bandwidth occupied by redundant data can be reduced; the cost of a physical cache occupied by the system can be reduced; and simultaneously, real-time sharing of the observation data can be ensured.

Description

technical field [0001] The invention belongs to the field of astronomical data processing, and in particular relates to a real-time astronomical data archiving system based on stream processing architecture. Background technique [0002] With the establishment of astronomical observatories around the world and the use of various high-precision observation instruments, astronomical observation data has shown an explosive growth trend. In the face of huge data sets, how to achieve effective management and retrieval is one of the key issues to improve scientific research output. In actual astronomical observations, FITS files are the most important storage format for astronomical observation data. FITS includes a data header and a data body. The FITS header contains rich descriptive metadata information, which is stored in the form of basic key / value. Compared with the huge original FITS dataset, FITS metadata not only has rich semantic information, but also is much smaller ...

AI Technical Summary

Problems solved by technology

However, the current methods of archiving astronomical observation data, especially remote data archiving, often require users to upload original data sets, or the archiving platform periodically scans data sources for offline archiving, resulting in a large amount of observation data occupying too much network bandwidth. Occupies a large cache space, and the problem of excessive data release delay for some applications with high real-time requirements

Images