Data processing system and method for radio astronomical data intensive scientific operation

Abstract

The invention provides a data processing system for radio astronomical data intensive scientific operation, which comprises at least one data constellation, and each data constellation is an extensible comprehensive data unit installed on a cabinet or an adjacent cabinet. The system is composed of an extensible distributed storage system, a hybrid heterogeneous computing node system and a network system. Each computing node is physically integrated with an ultra-large-capacity memory and a local storage unit of a flash memory type, and the storage system is composed of the local storage unit corresponding to each computing node and a distributed file system composed of the storage nodes; each data constellation has an independent distributed file system. The invention further provides a corresponding method. According to the data processing system, a data constellation architecture is adopted to cope with big data, each data constellation has an independent shared file system, the requirements of astronomical big data calculation and storage are met, and the influence caused by a traditional global file system is greatly reduced.

Description

technical field [0001] The invention belongs to the fields of radio astronomy data processing technology, data-intensive scientific computing, big data, and high-performance computing, and in particular relates to a data processing system and method for radio astronomical data-intensive scientific computing. Background technique [0002] With the construction and operation of advanced astronomical observation equipment, the astronomy community is faced with the challenges of very large-scale data and data-intensive scientific computing. For example, the globally cooperative Square Kilometre Array (SKA) telescope is the largest astronomical observation facility planned to be built by the international astronomy community, and it is the largest international cooperative scientific program in the field of astronomy that China has participated in. SKA brings together a large number of small-diameter antennas to achieve comprehensive aperture radio interference imaging. Its total...

AI Technical Summary

Problems solved by technology

[0004] Traditional supercomputing platforms have small local storage, low shared memory capacity, long data call time, and a single system architecture, which is not suitable for pipelines for this emerging ultra-large-scale data processing

In addition, the traditional supercomputing platform relies heavily on the shared file system in the storage architecture, and there will be a high system failure rate or even system paralysis in the SKA-scale data processing

Images