Data recording system for VGOS, data processing method and electronic equipment

Abstract

The invention provides a data recording system, a data processing method and electronic equipment for VGOS. The system comprises a control module, a front-end interface module, a synchronization module and a storage module, the front-end interface module is used for receiving multiple paths of observation data streams through a preset number of Ethernet interfaces; a programmable logic unit and a time reference interface are arranged in the synchronization module, the time reference interface is used for receiving an external clock signal, and the programmable logic unit is used for performing nanosecond frame synchronization on the multiple observation data streams based on the external clock signal; the control module is used for generating a write-in control strategy based on the state parameters of the storage module and writing the synchronously aligned data into the storage module based on the write-in control strategy; and the storage module forms a disk array by at least 32 hard disks. According to the invention, the long-time continuous recording capability that the single channel is 8 Gbps and the whole machine is not lower than 32 Gbps can be supported, and the requirements of VGOS on high-bandwidth and high-precision data acquisition are met.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data recording system, data processing method, and electronic device for VGOS. Background Technology

[0002] VLBI (Very Long Baseline Interferometry) uses globally distributed radio telescopes to conduct collaborative observations, creating a virtual telescope with an equivalent aperture on the Earth's scale. It plays an irreplaceable role in fields such as astrophysics, geodesy, and deep space exploration.

[0003] With the deployment of the next-generation VLBI data observation system, VGOS (VLBI Global Observing System), the raw data rate per station has generally reached 16–32 Gbps, and some deep space exploration or Event Horizon Telescope (EHT) missions even require continuous and stable recording of data streams exceeding 32 Gbps. This massive data deluge places higher demands on the capacity and continuous transmission of traditional VLBI data recording systems. Traditional magnetic tape recorders are insufficient to meet these requirements. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data recording system, data processing method and electronic device for VGOS, so as to improve the throughput of the data recording system and the data transmission capacity.

[0005] In a first aspect, a data recording system for VGOS is provided, the system comprising: a control module, a front-end interface module, a synchronization module and a storage module, wherein the front-end interface module, the synchronization module and the storage module are respectively communicatively connected to the control module; The front-end interface module is used to receive multiple observation data streams through a preset number of Ethernet interfaces; wherein the total effective payload rate of the preset number of Ethernet interfaces is not less than 32Gbps. The synchronization module has a built-in programmable logic unit and a time base interface. The time base interface is used to receive external clock signals, and the programmable logic unit is used to perform nanosecond-level frame synchronization of multiple observation data streams based on the external clock signals. The control module is used to generate a write control strategy based on the status parameters of the storage module, and to write the synchronized and aligned data into the storage module based on the write control strategy. The storage module consists of a disk array of at least 32 hard drives.

[0006] Optionally, the control module is also used to allocate an independent receive thread for each Ethernet interface; the execution logic of each receive thread includes: It receives observation data streams from the digital backend and parses the data streams according to the preset frame format to extract data frames containing frame numbers and timestamps. Based on an external time reference signal, frequency reference and time mark, a UTC timestamp is generated for each frame of data and written into the frame header of that frame; Based on the frame sequence number and UTC timestamp, out-of-order data frames are rearranged to form a frame sequence arranged in chronological order; The rearranged data frames are encapsulated according to a preset output format and written to the corresponding circular buffer.

[0007] Optionally, the synchronization module is specifically used for: Receive the encapsulated data stream from each circular buffer; each encapsulated data stream includes the frame sequence number and UTC timestamp; The phase offset of each channel relative to the global time base is determined based on the rising edge of the external time reference signal. Based on phase offset, dynamic synchronization and alignment processing is performed on the encapsulated data streams between each path; Write the synchronized and aligned multiple data streams into the shared buffer.

[0008] Optionally, the control module is specifically used for: The synchronized data stream is divided into continuous data blocks according to a preset block size; Real-time monitoring of the status parameters of each hard drive in the disk array, including at least the current write speed, I / O latency, and remaining storage space; Based on state parameters, the target hard drive with the best performance is dynamically determined, and the next data block to be written is allocated to the target hard drive.

[0009] Optionally, the data recording system for VGOS includes a master system and several slave systems; the master system and the slave systems have the same composition; the master system establishes communication with at least one slave system and completes time base synchronization before data reception; the control module in the master system is also used for: When the available storage space in the main system is detected to be lower than a preset threshold, the breakpoint position of the currently recorded data stream is determined. The breakpoint position includes the frame number of the last frame and the corresponding UTC timestamp. The breakpoint location is sent to the target slave system, triggering its start recording so that the target slave system can receive and record subsequent observation data streams starting from the next frame after the breakpoint location.

[0010] Optionally, the control module is also used to perform data playback operations, which include: Receive the playback time range or frame sequence number range specified by the user; Based on the time range or frame number interval, read the corresponding raw data from the disk array; The raw data read is reassembled into a multi-channel synchronous data stream according to the original observation time sequence; Send multi-channel synchronous data streams to the playback device.

[0011] Optionally, the control module is also used for: A hash value is embedded in a reserved field of each encapsulated data frame as an integrity fingerprint of that frame; The integrity fingerprints of multiple consecutive data frames are summarized in chronological order to generate a periodic verification list, which is then written to a read-only log file. Before data playback, the hash value of each frame is recalculated and compared with the integrity fingerprint embedded in the frame; If the hash value of any frame does not match, an alarm is triggered, and that frame and its associated data segment are marked as untrusted.

[0012] Optionally, the control module is also used for: Real-time collection of system operating status parameters, including at least CPU load, memory usage, throughput of each data processing thread, disk health status, and operating temperature; When any parameter exceeds the corresponding preset threshold, a graded alarm mechanism is triggered according to the severity of the anomaly. Simultaneously, when a disk failure or performance degradation is detected, the write task is switched to the hot spare hard drive, and a complete fault log containing the fault time, device identifier, and context information is generated.

[0013] Secondly, a data processing method for a VGOS data recording system is provided, the method comprising: It receives multiple observation data streams, with each data stream corresponding to an independent receiving channel; Based on externally provided time reference signals and frequency standards, the data streams within and between channels are synchronized and aligned. The synchronized and aligned multi-channel data streams are encapsulated according to a preset encapsulation format; Write control strategies are generated based on the disk array's status parameters. According to the write control policy, the encapsulated data stream is written to the corresponding disk in the disk array.

[0014] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements any of the methods of the first aspect.

[0015] This invention provides a data recording system, data processing method, and electronic device for VGOS. The system includes a control module, a front-end interface module, a synchronization module, and a storage module, all of which are communicatively connected to the control module. By employing multi-channel and independent front-end interface modules, this invention can stably receive observation data streams with a total rate exceeding 32 Gbps. Data from each channel is aligned to nanosecond-level time via the synchronization module, ensuring strict consistency in frame and timing among multiple signals. Simultaneously, the control module dynamically adjusts the writing strategy based on the real-time status of the disk array, achieving highly reliable and uninterrupted continuous data writing. Therefore, the system can support long-term continuous recording capabilities of 8 Gbps per channel and no less than 32 Gbps for the entire system, meeting the requirements of VGOS for high-bandwidth and high-precision data acquisition.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This diagram illustrates the structure of a data recording system for VGOS provided in an embodiment of the present invention. Figure 2 A flowchart of a data processing method for a VGOS data recording system provided by an embodiment of the present invention is shown; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] This invention provides a data recording system for VGOS. The VGOS observation system mainly consists of a radio source, an antenna, a digital back-end, and a digital recording system. The radio source refers to a natural radio radiation source from the universe, such as a quasar, an active galactic nucleus (AGN), or other distant compact objects. The radio signals emitted by these sources have high stability and point source characteristics, and can be used as a reference standard for VLBI interferometry. The antenna refers to a large radio telescope used to receive the weak radio signals from the radio source. The antenna is typically equipped with a high-sensitivity receiver and a low-noise amplifier (LNA) to down-convert the signal to the intermediate frequency (IF) band (e.g., 500 MHz–2 MHz). The digital back-end performs high-speed analog-to-digital conversion, digital down-conversion, multi-channel filtering, quantization (usually 2-bit complex number), and time stamping on the intermediate frequency analog signal output by the antenna, and encapsulates it into data frames according to a standard format (such as VDIF), and outputs continuous observation data streams through high-speed Ethernet (such as 10G / 25G); the digital recording system, namely the system described in the embodiments of the present invention, is used to receive multiple observation data streams from one or more digital back-ends and store and manage the data.

[0021] like Figure 1 As shown, the digital recording system includes a control module, a front-end interface module, a synchronization module, and a storage module. Each module communicates with the control module via a high-speed bus to collaboratively complete the reception, synchronization, storage, and management of multiple high-frequency observation data.

[0022] The front-end interface module is used to receive multiple observation data streams through a preset number of Ethernet interfaces; wherein the total effective payload rate of the preset number of Ethernet interfaces is not less than 32Gbps.

[0023] In this embodiment of the invention, the front-end interface module can be configured with four independent Ethernet interfaces, each of which connects to a digital backend (DBE) device; each interface can use a 10 G / 25 G / 40 G Ethernet interface, which can at least meet the data flow of 8 Gbps effective payload rate, thereby meeting the VGOS ≥32 Gbps continuous observation requirement.

[0024] The synchronization module has a built-in programmable logic unit and a time base interface. The time base interface is used to receive external clock signals, and the programmable logic unit is used to perform nanosecond-level frame synchronization of multiple observation data streams based on the external clock signals.

[0025] The synchronization module is based on a high-performance FPGA (such as Xilinx Kintex Ultrascale+ XCKU15P) and communicates with the control module through a PCIe Gen3 x8 interface.

[0026] The control module is used to generate a write control strategy based on the status parameters of the storage module, and to write the synchronized and aligned data into the storage module based on the write control strategy. The storage module consists of a disk array of at least 32 enterprise-grade hard drives, with a single-machine storage capacity exceeding 1 PB, supporting continuous 24-hour monitoring. In one example, it is connected to the control module via a miniSAS HD connector.

[0027] This invention employs multi-channel and independent front-end interface modules, enabling stable reception of observation data streams with a total rate exceeding 32 Gbps. Data from each channel undergoes nanosecond-level time alignment via a synchronization module, ensuring strict consistency in frame and timing among multiple signals. Simultaneously, the control module dynamically adjusts the write strategy based on the real-time status of the disk array, achieving highly reliable and uninterrupted continuous data writing. Therefore, the system supports long-term continuous recording capabilities of 8 Gbps per channel and no less than 32 Gbps for the entire system, meeting the requirements of VGOS for high-bandwidth and high-precision data acquisition.

[0028] Based on the above embodiments, the control module is further configured to allocate an independent receive thread for each Ethernet interface; the execution logic of each receive thread includes: The digital back-end performs high-speed analog-to-digital conversion, quantization, and time stamping on the radio signals received by the antenna, and outputs encapsulated data frames conforming to international standards. In this embodiment, the standard format is, for example, VDIF, but can also be Mark5B or a custom RAW format. Each frame contains a fixed-length frame header and payload, wherein the frame header has embedded a frame sequence number and a coarse time stamp generated by the digital back-end.

[0029] Each receiving thread continuously receives data packets from the corresponding Ethernet interface via the UDP protocol, and parses the data stream according to the preset frame format (such as the VDIF v1.0 specification) to extract the complete data frame and its frame sequence number and timestamp.

[0030] Based on an external time reference signal, frequency reference, and time stamp, a UTC timestamp is generated for each frame of data, and the UTC timestamp is written into the frame header of that frame.

[0031] In a specific example, the external time reference signal is, for example, 1PPS (One Pulse Per Second), a digital square wave signal with extremely precise rising edges, typically at TTL or LVDS level, with each rising edge strictly aligned to the exact second of UTC time; the frequency reference is, for example, 10MHz.

[0032] 1PPS provides an absolute time reference for UTC whole second moments; 10MHz provides a highly stable frequency reference for sub-second time interpolation.

[0033] 10 MHz is used to drive the internal counter of the front-end interface module, counting once every 100 ns (because 1 / 10 MHz = 100 ns). The signal coordination principle between the two is as follows: When the rising edge of 1PPS arrives, the front-end interface module will clear the counter or set it to a known value (such as the start of the corresponding UTC second). Thereafter, the precise UTC time at any given moment = whole second time (from the previous 1PPS) + counter value × 100 ns; Therefore, UTC time accurate to the nanosecond level can be calculated at any given moment. The receiving thread then maps each frame of data to its actual UTC timestamp (e.g., 2025-11-17T10:00:00.123456789 UTC) and writes this timestamp into the extended field of the frame header.

[0034] Based on the frame sequence number and UTC timestamp, out-of-order data frames are rearranged to form a frame sequence arranged in chronological order.

[0035] Although the digital backend sends data frames in strict timing, jitter in network transmission, switch buffering delays, or differences in operating system scheduling can cause data frames on the same channel to arrive at the recording system out of order (e.g., frame #1002 arrives before frame #1001). Without rearrangement, subsequent synchronization modules will be unable to correctly identify the time order of the frames, leading to timestamp misalignment and inter-frame phase distortion, severely impacting VLBI processing results. Therefore, it is essential to buffer and rearrange out-of-order frames based on frame sequence numbers and UTC timestamps to restore the original observation timing.

[0036] In a specific example, a receiving thread receives three consecutive frames with sequence numbers #1000, #1002, and #1001. Upon detecting the absence of frame #1001, the thread temporarily stores #1002, waiting for #1001 to arrive (the timeout threshold is set to 1 ms). Once #1001 arrives, the three frames are reassembled in the order #1000, #1001, and #1002, forming a frame sequence strictly arranged in ascending order of time.

[0037] The rearranged data frames are encapsulated according to a preset output format and written to the corresponding circular buffer.

[0038] In one example, the preset output format can be VDIF, Mark5B, or a custom RAW format. This embodiment of the invention supports multiple data formats such as VDIF, Mark5B, and RAW, is compatible with existing Mark 6 command protocols and the IVS Field System, and can directly interoperate with domestic and international digital back-ends and correlators.

[0039] The rearranged data frames are repackaged according to a preset output format (such as VDIF) (ensuring the timestamp has been updated) and written to a thread-specific circular buffer (e.g., a lock-free queue based on DPDK with a capacity of 64 MB). This circular buffer acts as a decoupling bridge between the front-end interface module and the synchronization module, absorbing burst traffic while ensuring a continuous supply of data.

[0040] This invention employs a multi-threaded buffering mechanism in the front-end interface module. Specifically, each Ethernet interface is allocated an independent receiving thread, and the data frames received by this thread are written to its dedicated circular buffer. This mechanism effectively decouples the rate difference between high-speed network data reception and subsequent data writing, thereby ensuring lossless reception and low-jitter transmission of multiple observation data streams under high load.

[0041] Specifically, since each data stream in the VGOS observation system has a rate exceeding 8 Gbps, using single-threaded polling or multi-channel shared buffers can easily lead to packet loss or processing bottlenecks due to thread scheduling delays or buffer contention. However, by configuring independent threads and private buffers for each interface, data reception from each channel does not interfere with each other, the operating system scheduling is more granular, and memory access locality is stronger, significantly improving the system's concurrent processing capabilities and real-time performance.

[0042] Based on the above embodiments, the synchronization module is specifically used for: Receive the encapsulated data stream from each circular buffer; each encapsulated data stream includes the frame sequence number and UTC timestamp.

[0043] In this embodiment of the invention, the synchronization module reads rearranged data frame streams from multiple circular buffers in a polling manner. Based on an external time reference signal, the phase offset of each channel relative to the global time base is determined.

[0044] In one example, with the rising edge of 1 PPS as the global time zero point, the synchronization module calculates the phase offset (in frames) of each data frame relative to the ideal frame sequence based on the UTC timestamp of each data frame.

[0045] For example, ideally, frame number #1000000 should be output at UTC = 10:00:00.000; if channel 3 actually outputs #999998 at this moment, its phase lags by 2 frames.

[0046] Based on phase offset, dynamic synchronization and alignment processing is performed on the encapsulated data streams between each channel.

[0047] In this embodiment of the invention, for channels with phase lag, the synchronization module dynamically inserts a corresponding number of empty frames into its output stream: the empty frame structure conforms to the corresponding frame format specification (e.g., VDIF specification); the frame header is marked with a "fill" flag (e.g., invalid_data = 1); the payload is all 0, does not participate in scientific processing, but retains the continuity of frame order.

[0048] For leading channels, their data is temporarily stored and output uniformly after other channels catch up. Write the synchronized and aligned multiple data streams into the shared buffer.

[0049] The aligned multiple data streams are interleaved and written to a shared buffer for subsequent write scheduling by the control module. Through the circular buffer of the above embodiment and the shared buffer of this embodiment, data reception and disk writing are effectively decoupled; even during disk writing, subsequent data frames can continue to be received, ensuring uninterrupted high-bandwidth data flow. Furthermore, even if disk writing is slow or delayed, it will not cause buffer overflow in the front-end interface module, thus avoiding data frame loss and damage to the integrity of the observation data.

[0050] This invention dynamically aligns data frames between channels, ensuring that all channels have the same frame number at any observation time. This guarantees strict alignment of baseline data during subsequent processing, preventing phase blurring or fringe loss. This allows the system to operate continuously for hours or even days, meeting long-term stable operation requirements and making it suitable for ultra-long baseline VLBI (e.g., Earth-Moon space geodesy) missions.

[0051] Based on the above embodiments, the control module of the present invention further implements an intelligent data distribution mechanism for efficiently and reliably writing synchronized and aligned multi-channel observation data into a large-scale disk array.

[0052] Specifically, the control module performs the following operations: The synchronized data stream is divided into continuous data blocks according to a preset block size; In one example, the preset block size is, for example, 64M, and each data block includes several frames of data.

[0053] Real-time monitoring of the status parameters of each hard drive in the disk array, including at least the current write speed, I / O latency, and remaining storage space; Based on state parameters, the target hard drive with the best performance is dynamically determined, and the next data block to be written is allocated to the target hard drive.

[0054] In this embodiment of the invention, the status parameters of each hard drive can be weighted and fused to obtain a score, and the hard drive with the highest score can be used as the target hard drive.

[0055] In one example, during system operation, for instance, hard drive #15 experienced a sudden drop in write speed from 550 MB / s to 180 MB / s due to aging, with I / O latency increasing to 800 μs. In the next scheduling round, the control module automatically allocated the next 64 MB data block to hard drive #37, without the user noticing, and recording continued.

[0056] This invention, through dynamic adjustment of the write control strategy, ensures that the system can switch over in milliseconds when single-disk performance degrades or a sudden failure occurs, without interrupting the recording task. This meets the high availability requirements of VGOS for continuous observation for several hours and improves the long-term reliability of the system.

[0057] In one feasible implementation, the control module establishes an index directory for the recorded data blocks to track the distribution and order of data across multiple disks. Whenever the disk write module successfully writes a data block, the index module records metadata such as the observation it belongs to, the time range, the disk number, and the block offset. This index information is persistently stored as a log, forming a data directory index. This mechanism enables the system to resume interrupted downloads / writes: if an interruption occurs during observation (e.g., temporary recording stoppage or device restart), the control module can consult the index to determine the last complete write position and resume writing from that point, avoiding data overwriting or omissions. Furthermore, the index module supports data reassembly and playback. When it is necessary to recombine observation data scattered across multiple disks into a continuous data stream, the index can guide the reassembly program to read the data blocks in the correct chronological order, essentially implementing a software-level "gather" reassembly function without the need for additional copying of the entire data. Index management also maintains metadata for each observation (observation ID, station name, scan segment number, etc.), facilitating integration with related processing software and data retrieval.

[0058] Based on the above embodiments, a complete workflow example for data reception, synchronization, and writing is given below: Suppose a VGOS observation mission begins at 10:00:00 UTC on November 17, 2025: Step 1: The control module broadcasts a start command, and the four digital back-end units simultaneously begin sending observation data (8Gbps per channel). Step 2: The front-end interface module receives 4 UDP streams and sends them to their respective receiving threads.

[0059] Each receiving thread parses the VDIF frame, extracts the frame sequence number (e.g., #1000000) and a coarse timestamp; combining 1PPS and 10MHz, it injects a precise UTC timestamp (e.g., 2025-11-17T10:00:00.123456789) into the frame; and based on the frame sequence number, it performs out-of-order frame buffering and rearrangement, and writes the result into a circular buffer.

[0060] Step 3: The synchronization module reads data from the 4-channel circular buffer; For example, if channel 3 is detected to be lagging by 2 frames (due to network jitter), 2 empty frames are immediately inserted into its stream; all channels output frame #1000001 at t=10:00:00.124 to achieve nanosecond-level alignment.

[0061] Step 4: The control module will divide the aligned data into 64 MB blocks; The write speed of hard drive #15 was detected to have dropped to 180 MB / s (below the threshold of 200 MB / s), and it was marked as to be replaced; subsequent data blocks were automatically allocated to hot spare drive #37.

[0062] In another embodiment of the invention, the data logging system for VGOS adopts a master-slave distributed architecture to overcome the limitations of single-machine storage capacity and support ultra-long-term continuous observation tasks (such as 24-hour geodesy or deep space exploration). The system includes a master system and several slave systems; all systems have identical hardware configurations, including a control module, a front-end interface module, a synchronization module, and a RAID disk array composed of 36 enterprise-grade SSDs (total usable capacity of approximately 500 TB / unit).

[0063] The main system and each slave system are interconnected via dedicated 25G SFP28 fiber optic links to form a low-latency control network.

[0064] Before the observation mission begins, the master system, acting as the time master node, broadcasts a time synchronization message to all slave systems via PTP (Precision Time Protocol). The synchronization modules of each slave system lock the PTP clock sent by the master system and perform phase calibration by combining it with the 1PPS / 10 MHz signal output by the local GPSDO to ensure that the time reference deviation of the entire system is ≤ 100 ns. The control module in the main system is also used for: When the available storage space in the main system is detected to be lower than a preset threshold, the breakpoint position of the currently recorded data stream is determined. The breakpoint position includes the frame number of the last frame and the corresponding UTC timestamp. The breakpoint location is sent to the target slave system, triggering its start recording so that the target slave system can receive and record subsequent observation data streams starting from the next frame after the breakpoint location.

[0065] This invention, through multi-machine cascading, allows the total recording capacity of the system to be linearly expanded to the petabyte (PB) level, meeting the needs of cutting-edge sciences such as ultra-long baseline bipolar interferometry (VLBI) and deep space exploration. Furthermore, breakpoint positioning is accurate to a single frame, and the starting points for resuming recording are strictly aligned, with no data overlap or loss, ensuring the integrity of the interferometric data.

[0066] Based on the above embodiments, the control module of the present invention further supports high-fidelity data playback functionality for scenarios such as scientific verification, correlator debugging, system testing, or educational demonstrations. The playback operation includes the following steps: Step 1: Receive the playback time range or frame sequence number range specified by the user.

[0067] In a specific example, playback requests can be submitted via a web interface, command line (such as vgos-replay --start-time=...), or IVS standard directives (such as playback=on, start=..., stop=...).

[0068] For example: UTC time range (e.g., 2025-11-17T10:00:00.000 to 2025-11-17T10:01:00.000); frame sequence number range (e.g., #1000000 to #1060000).

[0069] Step 2: Based on the time range or frame number interval, read the corresponding raw data from the disk array.

[0070] Step 3: Reassemble the read raw data into a multi-channel synchronous data stream according to the original observation time sequence.

[0071] In this step, all data frames from all channels are merged into a single time axis according to UTC timestamps; missing frames (such as those skipped due to recording failures) are inserted to maintain frame sequence continuity; and all channels in the output stream have the same frame number at any given time, restoring the synchronization state at the time of the original observation.

[0072] Step 4: Send the multi-channel synchronous data stream to the playback device.

[0073] In this step, sendfile or the DPDK user-space network stack is used to directly send disk data to the network card, supporting simultaneous writing and playback of data with latency controllable to the millisecond level. This is suitable for e-commerce applications. VLBI real-time correlation.

[0074] Playback devices include, for example, VLBI software correlators (such as DiFX, SFXC); hardware correlator prototypes; network analyzers or signal simulators; and remote test stations for joint testing.

[0075] Based on the above embodiments, the control module is also used for: A hash value is embedded in a reserved field of each encapsulated data frame as an integrity fingerprint of that frame.

[0076] Before writing data to the disk array, the control module calculates the SHA-256 hash value in real time for the payload portion (excluding the frame header) of each encapsulated data frame (such as VDIF format).

[0077] The integrity fingerprints of multiple consecutive data frames are summarized in chronological order to generate a periodic verification list, which is then written to a read-only log file.

[0078] In one example, the control module summarizes the integrity fingerprints of consecutive data frames in chronological order every 10 minutes (or every 60,000 frames) to generate a verification list record, which includes: start and end UTC time (e.g., 2025-11-17T10:00:00.000 – 10:10:00.000); start and end frame sequence numbers (e.g., #1000000 – #1060000); Merkle root hash of all frame fingerprints; and the list's own signature (for tamper-proof).

[0079] Before data playback, the hash value of each frame is recalculated and compared with the integrity fingerprint embedded in the frame.

[0080] When a user initiates a data playback request, the control module first starts the integrity verification process: Read all data frames for the target time period from the disk; Recalculate SHA-256 for each frame's payload; Compare with the fingerprint embedded in the frame.

[0081] If the hash value of any frame does not match, an alarm is triggered, and that frame and its associated data segment are marked as untrusted.

[0082] In one scenario: For example, a hard drive may have an error in one sample value due to cosmic rays. The hash check successfully captures the error, preventing it from entering the relevant processing and thus preventing phase shift of the interference fringes.

[0083] The embodiments of the present invention ensure that the data entering the correlator is faithful to the original observations, avoiding false astronomical conclusions or geodetic biases caused by storage errors.

[0084] Based on the above embodiments, the control module of the present invention further integrates a monitoring and fault management unit. This unit runs through the entire link of data reception, synchronization, storage and playback, realizing full-process, real-time, multi-dimensional monitoring of the system's operating status and automated fault response, ensuring high reliability of VGOS observation missions under long-term unattended conditions.

[0085] The control module is also used for: Real-time collection of system operating status parameters, including at least CPU load, memory usage, throughput of each data processing thread, disk health status, and operating temperature.

[0086] When any parameter exceeds the corresponding preset threshold, a graded alarm mechanism is triggered based on the severity of the anomaly.

[0087] For example, if the write speed of a disk drops significantly or an I / O error occurs, the fault management mechanism will trigger corresponding measures, including notifying the control module to enable the hot spare disk to take over the write, marking the faulty disk offline, dynamically adjusting the write load of other disks, and preventing the spread of a single point of failure.

[0088] For problems in network transmission, such as repeated timeouts of a transmission thread, the module can record the situation and notify the transmission engine to reduce the weight of that thread or switch to an alternative path.

[0089] Simultaneously, when a disk failure or performance degradation is detected, the write task is switched to the hot spare hard drive, and a complete fault log containing the fault time, device identifier, and context information is generated.

[0090] The fault log contains structured log entries with the following information: Failure time (UTC accurate to milliseconds); Device identifier (e.g., / dev / sdb, SN: S658NX0R812345); Context information (current CPU load, temperature, ongoing frame number, I / O error code); Automatic processing action ("Swapped to hot spare disk #37"); Log file path: / var / log / vgos / fault_20251117_143025.log.

[0091] This invention upgrades the traditional passive recording system into an intelligent observation platform with self-sensing, self-diagnosis, and self-recovery capabilities. This not only meets the stringent requirements of VGOS for long-term stability but also lays the technical foundation for the next generation of autonomous VLBI stations.

[0092] All the above modules operate through a unified control and communication interface, forming a complete software system. The modular design facilitates the addition, removal, or upgrading of functional modules as needed; for example, new transmission protocol modules (such as RDMA support) can be inserted, or the underlying storage media (such as NVMe SSD arrays) can be replaced without affecting the overall architecture. This ensures that the system of this invention meets the current high bandwidth requirements of VLBI while possessing the ability to continuously evolve for the future.

[0093] Based on the same inventive concept, embodiments of the present invention provide a data processing method for a VGOS data recording system, such as... Figure 2 The method includes the following steps: Step S201: Receive multiple observation data streams.

[0094] Each data stream corresponds to an independent receiving channel.

[0095] Before the observation begins, the control module receives the observation plan and configures the network parameters and data format of the front-end interface module. Network parameters include, for example, the speed, IP address, and port of each Ethernet interface.

[0096] Step S202: Based on the externally provided time reference signal and frequency reference, synchronize and align the data streams within and between channels.

[0097] Step S203: Encapsulate the synchronized multi-channel data streams according to a preset encapsulation format.

[0098] Step S204: Generate a write control strategy based on the disk array's status parameters.

[0099] Step S205: Write the encapsulated data stream to the corresponding disk in the disk array according to the write control policy.

[0100] The method provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned system embodiment. For the sake of brevity, any parts not mentioned in the method embodiment can be referred to the corresponding content in the aforementioned system embodiment. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process can be referred to the corresponding process in the above system embodiment, and will not be repeated here.

[0101] Based on the same technical concept, embodiments of the present invention also provide an electronic device, such as... Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0102] Memory 303 is used to store computer programs; The processor 301, when executing a program stored in the memory 303, implements the steps of a data processing method for a data recording system of VGOS.

[0103] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0104] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0105] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0106] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0107] The data recording system for VGOS provided in this embodiment of the invention can be specific hardware on the device or software or firmware installed on the device.

[0108] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces. Indirect couplings or communication connections between systems or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0113] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data recording system for VGOS, characterized in that, The system includes: a control module, a front-end interface module, a synchronization module, and a storage module, wherein the front-end interface module, the synchronization module, and the storage module are respectively communicatively connected to the control module; The front-end interface module is used to receive multiple observation data streams through a preset number of Ethernet interfaces; wherein the total effective payload rate of the preset number of Ethernet interfaces is not less than 32Gbps. The synchronization module has a built-in programmable logic unit and a time reference interface. The time reference interface is used to receive external clock signals, and the programmable logic unit is used to perform nanosecond-level frame synchronization of the multi-channel observation data stream based on the external clock signals. The control module is used to generate a write control strategy based on the status parameters of the storage module, and to write the synchronized and aligned data into the storage module based on the write control strategy. The storage module consists of a disk array composed of at least 32 hard drives.

2. The system according to claim 1, characterized in that, The control module is also used to allocate an independent receive thread for each Ethernet interface; the execution logic of each receive thread includes: The system receives observation data streams from a digital backend and parses the data streams according to a preset frame format to extract data frames containing frame numbers and timestamps. Based on the external time reference signal, frequency reference and the time mark, a UTC timestamp is generated for each frame of data and the UTC timestamp is written into the frame header of the frame; Based on the frame sequence number and UTC timestamp, the out-of-order data frames are rearranged to form a frame sequence arranged in chronological order; The rearranged data frames are encapsulated according to a preset output format and written to the corresponding circular buffer.

3. The system according to claim 2, characterized in that, The synchronization module is specifically used for: Receive the encapsulated data stream from each of the circular buffers; each encapsulated data stream includes a frame sequence number and a UTC timestamp; Based on an external time reference signal, determine the phase offset of each channel relative to the global time base; Based on the phase offset, dynamic synchronization and alignment processing is performed on the encapsulated data streams between each path; Write the synchronized and aligned multiple data streams into the shared buffer.

4. The system according to claim 1, characterized in that, The control module is specifically used for: The synchronized data stream is divided into continuous data blocks according to a preset block size; The status parameters of each hard drive in the disk array are monitored in real time, including at least the current write speed, I / O latency, and remaining storage space; Based on the state parameters, the target hard disk with the best performance is dynamically determined, and the next data block to be written is allocated to the target hard disk.

5. The system according to claim 1, characterized in that, The data recording system for VGOS includes a master system and several slave systems; the master system and the slave systems have the same composition; the master system establishes communication with at least one of the slave systems and completes time base synchronization before data reception; the control module in the master system is further used for: When the available storage space in the main system is detected to be lower than a preset threshold, the breakpoint position of the currently recorded data stream is determined. The breakpoint position includes the frame number of the last frame and the corresponding UTC timestamp. The breakpoint location is sent to the target slave system, and its start recording is triggered so that the target slave system can receive and record subsequent observation data streams starting from the next frame after the breakpoint location.

6. The system according to claim 2, characterized in that, The control module is also used to perform a data playback operation, the playback operation including: Receive the playback time range or frame sequence number range specified by the user; Based on the time range or frame number interval, the corresponding raw data is read from the disk array; The raw data read is reassembled into a multi-channel synchronous data stream according to the original observation time sequence; The multi-channel synchronous data stream is sent to the playback device.

7. The system according to claim 6, characterized in that, The control module is also used for: A hash value is embedded in a reserved field of each encapsulated data frame as an integrity fingerprint of that frame; The integrity fingerprints of multiple consecutive data frames are summarized in chronological order to generate a periodic verification list, and the verification list is written to a read-only log file. Before data playback, the hash value of each frame is recalculated and compared with the integrity fingerprint embedded in the frame; If the hash value of any frame does not match, an alarm is triggered, and that frame and its associated data segment are marked as untrusted.

8. The system according to claim 1, characterized in that, The control module is also used for: Real-time acquisition of system operating status parameters, including at least CPU load, memory usage, throughput of each data processing thread, disk health status, and operating temperature; When any of the parameters exceeds the corresponding preset threshold, a graded alarm mechanism is triggered according to the severity of the anomaly. Simultaneously, when a disk failure or performance degradation is detected, the write task is switched to the hot spare hard drive, and a complete fault log containing the fault time, device identifier, and context information is generated.

9. A data processing method for a VGOS data recording system based on any one of claims 1-8, characterized in that, The method includes: It receives multiple observation data streams, with each data stream corresponding to an independent receiving channel; Based on externally provided time reference signals and frequency standards, the data streams within and between channels are synchronized and aligned. The synchronized and aligned multi-channel data streams are encapsulated according to a preset encapsulation format; Write control strategies are generated based on the disk array's status parameters. According to the write control strategy, the encapsulated data stream is written to the corresponding disk in the disk array.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method of claim 9.

Images