Radio Astronomy Real-time Data Stream Processing Method and System Based on Distributed Acquisition Components

Abstract

The present invention discloses a method and system for processing real-time radio astronomy data streams based on distributed acquisition components, including: S1. Data transmission, the server receives User Datagram Protocol (UDP) data packets transmitted by a radio astronomy telescope through a network card, uses the open-source pulsar distributed acquisition component and data analysis system PSRDADA to create and manage a circular buffer, and transfers data streams between each circular buffer; S2. Data processing, utilize the unified computing device architecture (CUDA) parallel computing platform and programming model of a graphics processing unit (GPU) to write a calculation and unpacking program, thereby realizing parallel processing and data extraction of large-scale data; S3. Data writing to disk, use the data storage client of PSRDADA to write data blocks to the disk and split the data into files of arbitrary length. The present invention can efficiently and stably receive high-speed radio data streams, realize real-time processing of data, and provide a reliable solution for high-bandwidth signal transmission and processing in radio astronomy observations.

Description

Technical Field

[0001] The present invention relates to the technical fields of astronomical observation and signal processing, and mainly relates to a method and system for processing real-time radio astronomy data streams based on distributed acquisition components. Background Art

[0002] The phased array is a newly emerging array antenna receiving system. Due to its advantages in single-beam gain, multi-beam continuous sky coverage, and larger field of view, it is gradually being applied to radio telescopes and can significantly accelerate the sky survey rate in cooperation with large-aperture telescopes. However, the high-bandwidth radio signals received by the phased array pose higher requirements for the real-time performance, large storage capacity, high availability, and persistence of digital terminals. In a high input-output load environment, traditional digital terminals often experience problems such as packet loss, out-of-order, and duplication, significantly affecting the transmission quality. In addition, traditional data stream processing frameworks based on distributed acquisition components usually directly write the data transmitted by high-speed network cards into the video memory of a Graphics Processing Unit (hereinafter referred to as GPU), without introducing a circular buffer for data buffering. When the transmission rate exceeds the processing rate, this design is prone to data loss or processing delay and cannot meet the requirements of real-time data stream processing.

[0003] In response to the above challenges, this study developed a real-time stream data processing framework based on distributed acquisition components. This framework can efficiently and stably receive high-speed radio astronomy data streams observed by the phased array, and at the same time achieve real-time processing of the data, providing a reliable solution for high-bandwidth signal transmission and processing in radio astronomy observations.

[0004] Since the observed source signals in radio astronomy are extremely weak, generally 30 - 50 dB (decibels, Decibel) below the receiving system noise, extremely high receiving system sensitivity is required. To meet the high sensitivity and resolution requirements of radio astronomy observations, the usual solution is to increase the effective area of the antenna and reduce the system noise temperature (Wang Zhijie et al., 2008). However, increasing the effective area of the antenna results in a narrower beam width and reduces the single-beam spatial coverage range (Zhang Guangyi, 2007). Although traditional multi-beam feed groups can expand the observation field of view, they cannot achieve complete overlapping coverage of the beams.

[0005] Compared with traditional single-beam feeds and horn feed arrays, phased array technology performs better in terms of single-beam gain and multi-beam continuous sky coverage, has a larger field of view, and thus can achieve continuous sky coverage. The phased array can adjust the amplitude and phase of each array element through a beam synthesis network, and adjust the performance of the defocused beam to the level equivalent to that of the main focus beam. Compared with multi-horn feed arrays, the use of phased array feed technology can perfectly solve the problems of uneven illumination beam widths and incomplete overlapping coverage of densely distributed beams in the ultra-wideband range.

[0006] The Linux kernel (an open-source operating system kernel used as a server operating system) incurs significant overhead when processing broadband network traffic, mainly manifested in multiple packet copies, context switches, and complex processing flows in the protocol stack. Since digital terminals use User Datagram Protocol (UDP) packets, as network bandwidth continues to increase, the processing bottleneck of the kernel gradually emerges. In a high input / output load environment, this bottleneck can lead to problems such as network packet loss, out-of-order packets, and duplicate packets, which significantly affect the data transmission quality. For pulsar data, network packet loss, out-of-order packets, and duplicate packets have a serious impact on pulsar period prediction and folding.

[0007] In traditional radio data stream processing frameworks, data processing usually involves directly allocating space in the GPU video memory without creating buffers. This approach leads to frequent memory access, thereby reducing the computing rate. Especially when high-speed data transfer is required between the Central Processing Unit (CPU) and the GPU, during the stage where the graphics processing unit performs efficient real-time data processing, the communication and data access rate between the CPU and the GPU become the performance bottleneck. Summary of the Invention

[0008] Aiming at the problems existing in the prior art, the purpose of the present invention is to provide a radio astronomy real-time data stream processing method based on distributed acquisition components, and a real-time stream data processing framework including data transmission, processing, and disk writing is developed based on the distributed acquisition components. The present invention can efficiently and stably receive high-speed radio data streams observed by phased arrays, and at the same time realize real-time data processing, providing a reliable solution for high-bandwidth signal transmission and processing in radio astronomy observations.

[0009] To achieve the above object, the present invention provides a radio astronomy real-time data stream processing method based on distributed acquisition components, and the method includes the following steps:

[0010] S1. Data transmission, the server receives UDP packets transmitted by the radio telescope through the network card, uses the open-source pulsar distributed acquisition component and data analysis system (Pulsar Distributed Acquisition and DataAnalysis, hereinafter referred to as PSRDADA) to create and manage circular buffers, and transfer data streams between each circular buffer;

[0011] S2. Data processing: Using the Compute Unified Device Architecture (CUDA) parallel computing platform and programming model of the GPU, write a calculation and unpacking program to achieve parallel processing and data extraction of large-scale data.

[0012] S3. Data writing to disk: Use the data storage client (dada_dbdisk) of PSRDADA to write data blocks to the disk and split the data into files of arbitrary length.

[0013] Furthermore, in step S1, the distributed acquisition component PSRDADA creates multiple circular buffers. Each circular buffer is allocated as a shared memory resource and logically divided into a file header describing the observation information and multiple data blocks storing data.

[0014] Furthermore, in step S1, the data stream transmission process includes:

[0015] S1.1 At the start of the observation, after the server receives UDP data packets from the front-end radio telescope through the network card, it first creates a circular buffer in the server memory using the data stream control software (dada_db) of PSRDADA and reads the network card data using direct memory access; each data block in the buffer is initially empty, and its file header is initialized with information related to the observation; the information includes bandwidth, center frequency, signal source, and start time.

[0016] S1.2 When the circular buffer in the server memory receives data, parse the file header of the buffer to obtain the observation information; subsequently, create a circular buffer on the GPU using the data stream control software and start the transfer script transfer_cputogpu from CPU memory to GPU video memory to transfer the data in the server memory to the circular buffer on the GPU.

[0017] S1.3 After the GPU finishes data processing, first create a circular buffer in the server memory using the data stream control software, and then transfer the result data to the server memory using transfer_gputocpu.

[0018] S1.4 During the entire observation process, data is written to data blocks in sequence. When the observation ends, a data end flag is set, and at the same time, the last complete data block and the number of valid bytes written to it are recorded. After that, data will be read from the data blocks in sequence until the data end flag is encountered and the current data block is the last complete data block, at which point the reading stops.

[0019] Further, in step S2, the GPU is used to process parallel tasks, and the CPU is used to process latency-sensitive tasks.

[0020] Further, in step S2, the following steps are included:

[0021] S2.1 The GPU parses the observation data by reading the data in the circular buffer in the video memory and based on the meta-information in the file header; the parsing process includes extracting the observation timestamp and the UDP data size; and locating the data block address corresponding to the spectrum according to the unpacked meta-information.

[0022] S2.2 On the basis of the parsed data, beam calibration is pre-performed on the parsed spectrum information; in the calibration process, the covariance matrix is calculated for each observation point.

[0023] S2.3 According to the calculated covariance matrix, estimate the direction vectors of each pointing of the beam :

[0024] S2.4 After the direction vector estimation is completed, based on the maximum signal-to-noise ratio criterion, calculate the beamforming weights.

[0025] S2.5 According to the weight matrix obtained by the pre-beam calibration, combined with the information of each beam in the array, calculate the beam after beam synthesis.

[0026] S2.6 According to the synthesized beam, integrate and enhance the signal in the direction of the target signal source, and transmit the result to the server memory.

[0027] Further, it is characterized in that in step S2.2, the covariance matrix formula is as follows:

[0028]

[0029] Where in formula 1.1 represents the nth signal vector, represents the conjugate transpose of the nth signal vector; represents the expectation value of multiple signal vectors in the time domain; represents the "on" covariance matrix.

[0030] Where in formula 1.2 represents the nth noise vector, represents the conjugate transpose of the nth noise vector; represents the expectation value of multiple noise vectors in the time domain; represents the "off" covariance matrix.

[0031] Further, in step S3, the data writing stage, the beginning of each file contains a file header from the data block; after each file is written to the disk, a record is added to the ASCII (American Standard Code for Information Interchange) text log file.

[0032] Further, each record contains the complete path of the file, the writing time, the file size, the time required to write the file, and the observation identifier.

[0033] Further, when the data writing disk module needs to access other server computing nodes or storage nodes, it is sent to the switch through the network card.

[0034] On the other hand, the present invention provides a radio astronomy real-time data stream processing system based on a distributed acquisition component, and the system is used to implement the above-mentioned radio astronomy real-time data stream processing method based on a distributed acquisition component. The system includes a data transmission module, a data processing module, and a data writing disk module.

[0035] Beneficial effects:

[0036] The present invention designs and implements a framework program, called a pipeline, for digital terminals that can support high-speed data packet transmission, reception, and preprocessing. The pipeline realizes the efficient transfer of data from reception to processing and then to writing to the disk through a circular buffer, ensuring that the processing of each stage can be completed under real-time requirements. By calling the program modules of different stages, the pipeline can ensure the efficient connection between various links such as data transmission, processing, and storage, thereby meeting the strict requirements for real-time performance. The pipeline not only needs to effectively handle the data flow under high input-output loads, but also should have flexible scalability and high-performance computing capabilities, and can perform real-time transmission and processing of massive data. For this reason, the pipeline should be able to optimize the processing path, reduce kernel intervention, and efficiently allocate computing resources while ensuring smooth data transmission. This not only helps to improve the response speed of the system, but also significantly improves the overall operation efficiency, ensuring the stability and efficiency of the system during long-term operation, and thus supporting radio astronomy observation tasks. Description of the drawings

[0037] Figure 1 is the overall architecture diagram of the radio astronomy real-time data stream processing method and system based on a distributed acquisition component of the present invention;

[0038] Figure 2 is the schematic diagram of the data transmission data stream;

[0039] Figure 3 is the schematic diagram of the GPU heterogeneous computing principle;

[0040] Figure 4 It is a schematic diagram of the data processing process;

[0041] Figure 5 It is a schematic diagram of the data writing process to the disk. Specific embodiments

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0044] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installation", "connection", "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0045] The following is combined with Figures 1 - 5 The specific embodiments of the present invention will be described in detail. It should be understood that the specific embodiments described here are only for the purpose of illustrating and explaining the present invention, and are not used to limit the present invention.

[0046] Based on the current requirements of radio astronomy observations for high sensitivity and high resolution, as well as the urgent need to address the real-time processing of high-bandwidth radio signals collected by phased arrays in digital terminals, the primary research content of this paper is to construct a real-time data stream processing framework for radio astronomy data using distributed acquisition components. This research aims to solve the problem of data quality degradation caused by data packet loss in a high input-output network environment, thereby achieving high-speed processing of real-time transmitted radio astronomy data streams. In addition, by abstracting data blocks and processing processes using distributed acquisition components, the real-time data stream processing framework is highly modularized, making each processing stage more flexible. This approach not only optimizes the development process but also enables efficient flow of real-time data streams across data blocks, thus enhancing the overall performance and scalability of the processing framework.

[0047] As Figure 1 shown, Figure 1 Figure 1 is the overall architecture diagram of the radio astronomy real-time data stream processing method and system based on distributed acquisition components. The process starts with the data transmission phase at the beginning of the observation. After the server receives UDP data packets from the front end through the Network Interface Card (NIC), it first creates a circular buffer in the server memory using the data stream control software of PSRDADA. When the circular buffer in the server memory receives data, it parses the file header of the buffer to obtain observation information. Subsequently, a circular buffer is created on the GPU using the data stream control software, and a transfer script from CPU memory to GPU video memory is started to transfer the data in the server memory to the circular buffer on the GPU.

[0048] After entering the data processing phase, the GPU circular buffer receives data and parses the observation data based on the meta-information in the file header. Then, beam calibration is performed in advance to calculate the weight matrix . According to the weight matrix obtained from the pre-beam calibration and combined with the information of each beam in the array, the following formula is used to calculate the beam after beam synthesis. Based on the synthesized beam, integration is performed and the signal in the direction of the target signal source is enhanced. A circular buffer is created in the server memory using the data stream control software, and a transfer script from GPU memory to CPU video memory is started to transfer the data in the server memory to the circular buffer on the CPU.

[0049] Finally, in the data writing phase, the data storage client of PSRDADA is used to split the data into files of arbitrary length. The beginning of each file will contain the file header from the data block. Subsequently, if other server computing nodes or storage nodes need to be accessed, it can be sent to the switch through the network card.

[0050] The radio astronomy real-time data stream processing system based on distributed acquisition components according to the present invention includes the following three modules: a data transmission module (data transmission stage), a data processing module (data processing stage), and a data writing module (data writing stage).

[0051] The main function of the data transmission module is to create and manage a circular buffer, build a transmission link based on the circular buffer and a device usage script, and be responsible for managing the flow of real-time data. Its input is a UDP data packet from a switch, which contains a digital signal formed by encapsulating a radio signal processed by a radio telescope through FFT (Fast Fourier Transform) using an FPGA (Field-Programmable Gate Array) and specified observation information. The output is the result data transferred from the GPU video memory to the CPU memory.

[0052] The main function of the data processing module is to parse the file header in the circular buffer, extract specific observation information, and process the observation data in the circular buffer according to this information. Its input is the data transferred from the CPU memory to the GPU video memory of the data transmission module, and its output is the result data after integration and enhancement by beam synthesis.

[0053] The main function of the data writing module is to write multiple data blocks in the circular buffer to a disk, split the data into files of any length, and if it is necessary to access other server computing nodes or storage nodes, it can be sent to the switch through a network card. Its input is multiple data blocks in the circular buffer and a file header. The output is a FITS (Flexible Image Transport System) file or an encapsulated UDP data packet.

[0054] Corresponding to these three modules, accordingly, the radio astronomy real-time data stream processing method based on distributed acquisition components according to the present invention includes the following three steps:

[0055] S1. Data transmission: The server receives the UDP data packet transmitted by the radio telescope through the network card, and uses the open-source distributed acquisition component PSRDADA. Its core function is to create and manage a circular buffer and transfer the data stream between each circular buffer.

[0056] S2. Data processing: Utilize the CUDA parallel computing platform and programming model of the GPU to write a calculation and unpacking program, so as to realize the parallel processing of large-scale data and the rapid data extraction.

[0057] S3. Write data to disk. Use the data storage client of PSRDADA. Its core function is to write data blocks to the disk and split the data into files of arbitrary lengths.

[0058] Specifically, in step S1, in the data transmission module, PSRDADA creates multiple circular buffers. Each circular buffer will be allocated as a shared memory resource and logically divided into a file header (header block) describing the observation information and multiple data blocks storing data. Each data block will have an associated byte counter for calculating the time offset from the start of the observation. At any moment, only a continuous data stream can be represented in the data block; therefore, the size of the data block will determine the maximum time required to empty the circular buffer between stopping one observation and starting the next observation.

[0059] The overall data stream transmission is as Figure 2 shown. The process is that the switch sends UDP data packets to the server. After the server network card receives the incoming UDP data packets, it first creates a circular buffer in the server memory using the data stream control software of PSRDADA and reads the network card data using the Direct Memory Access (DMA) method. Each data block in this buffer is initially empty, and its file header will be initialized with information related to the observation (information related to the observation includes bandwidth, center frequency, signal source, start time, etc.). Before the data is officially valid, in order to start data acquisition in advance, data blocks are allowed to write data before the data start flag is set. Only when the data start flag is set, is it allowed to read data from the data block.

[0060] When the circular buffer in the server memory receives data, it will parse the file header of the buffer to obtain the observation information. Subsequently, a circular buffer is created on the GPU using the data stream control software, and a transfer script from CPU memory to GPU video memory is started to transfer the data in the server memory to the circular buffer on the GPU.

[0061] Similarly, after the GPU finishes data processing, it first creates a circular buffer in the server memory using the data stream control software, and then creates a circular buffer in the server memory using the data stream control software, and starts a transfer script from GPU memory to CPU video memory to transfer the data in the server memory to the circular buffer on the CPU.

[0062] Step S1 specifically includes:

[0063] S1.1 At the beginning of the observation, after the server receives UDP (User Datagram Protocol) data packets transmitted from the front end (radio telescope) through the Network Interface Card (NIC), it first creates a circular buffer in the server memory using the data stream control software of PSRDADA and reads the NIC data using direct memory access. Each data block in this buffer is initially empty, and its file header will be initialized with information related to the observation (such as bandwidth, central frequency, signal source, start time, etc.). Before the data becomes officially valid, in order to start data acquisition in advance, data blocks are allowed to be written with data before the data start flag is set. Only when the data start flag is set, is it allowed to read data from the data block.

[0064] S1.2 When the circular buffer in the server memory receives data, it parses the file header of the buffer to obtain the observation information. Subsequently, a circular buffer is created on the GPU using the data stream control software, and the transfer script transfer_cputogpu from CPU memory to GPU video memory is started to transfer the data in the server memory to the circular buffer on the GPU.

[0065] S1.3 After the GPU finishes data processing, it first creates a circular buffer in the server memory using the data stream control software, and then transfers the result data to the server memory using transfer_gputocpu.

[0066] S1.4 During the entire observation process, data is written into data blocks in sequence. When the observation ends, the data end flag is set, and at the same time, the last complete data block and the number of valid bytes written to it are recorded. After that, data will be read from the data block in sequence until the data end flag is encountered and the current data block is the last complete data block, at which point the reading stops.

[0067] Step S2: In the data processing module, during the data processing stage, an efficient computing and unpacking program is written using the CUDA (Compute Unified Device Architecture) parallel computing platform and programming model of the GPU, so as to achieve parallel processing of large-scale data and rapid data extraction. This system adopts a heterogeneous computing architecture, in which the GPU is dedicated to accelerating the parts of the code with parallel characteristics. Since the GPU can efficiently execute parallel computing tasks, it is usually called an "accelerator". In this mode, the GPU is responsible for processing parallel tasks, including beamforming, beam calibration, and parallel processing of the unpacking of the ring buffer file header. The CPU, on the other hand, focuses on processing latency-sensitive tasks, including real-time reception of UDP data packets, parsing UDP packet headers to obtain observation information, and real-time data transmission. By combining a high-performance CPU and a high-throughput GPU, the overall performance of the application can be significantly improved. Figure 3 Figure Figure 3 shows a schematic diagram of parallel computing based on the heterogeneous computing mode. The graphics processing unit (GPU) has multiple parallel computing units and is mainly responsible for parallel tasks and most matrix operations in the server. The central processing unit (CPU) has multiple computing cores and is responsible for processing sequential tasks, including real-time data transmission, as well as system and sequential tasks such as receiving UDP data packets and parsing UDP packet headers.

[0068] The CPU is good at processing the parts of the code limited by latency, while the GPU is good at running the code in a single-instruction multiple-data (SIMD) parallel manner. If only optimizing a certain part of the CPU code or GPU code to run faster does not necessarily significantly improve the speed of the overall application. To maximize performance, the two processors, the CPU and the GPU, need to work together in their optimal states. This approach of offloading certain types of operations from the processor to the GPU is called heterogeneous computing.

[0069] The overall process of data processing is as shown in Figure 4As shown, the process is as follows: When the GPU ring buffer receives data for data transmission, the GPU reads the data in the ring buffer in the video memory and parses the observation data based on the meta-information in the file header. The parsing process includes extracting key information such as the observation timestamp (Julian day, MJD_start), the number of feed array elements (nelement), the number of synthesized beams (nbeam), the UDP packet header size (pkt_header), and the UDP data size (pkt_data). Subsequently, based on the unpacked meta-information, the data block address corresponding to the spectrum is located. Since the spectrum data is in complex form, the parsed spectrum matrix information is stored and processed in cuComplex format. Based on the parsed data, beam calibration is pre-performed on the parsed spectrum information. The beam calibration calculates the covariance matrix of the signal and noise, then calculates each beam direction vector, and calculates the beam weight matrix according to each beam direction vector. In the beam synthesis stage, according to the weight matrix obtained from the pre-beam calibration and combined with the information of each beam in the array, the beam after beam synthesis is calculated. Finally, based on the synthesized beam, integration is performed and the signal in the direction of the target signal source is enhanced, and the result is transmitted to the server memory.

[0070] Step S2 specifically includes:

[0071] S2.1 The GPU reads the data in the ring buffer in the video memory and parses the observation data based on the meta-information in the file header. The parsing process includes extracting key information such as the observation timestamp (Julian day, MJD_start), the number of feed array elements (nelement), the number of synthesized beams (nbeam), the UDP packet header size (pkt_header), and the UDP data size (pkt_data). Subsequently, based on the unpacked meta-information, the data block address corresponding to the spectrum is located. Since the spectrum data is in complex form, the parsed spectrum matrix information is stored and processed in cuComplex format.

[0072] S2.2 Based on the parsed data, beam calibration is pre-performed on the parsed spectrum information. The calibration process calculates the covariance matrix for each observation point, and the formula is as follows:

[0073]

[0074] Where in formula 1.1 represents the nth signal vector, which contains multiple signal components and represents the signals received by different sensors or receiving antennas. represents the conjugate transpose of the nth signal vector. The conjugate transpose means taking the conjugate and transposing each element in the complex signal vector, which is usually used to process data in the complex domain in signal processing; It represents taking the expected value of multiple signal vectors in the time domain, and this expected value represents the average correlation between signal vectors in a statistical sense; It represents the "on" covariance matrix. This matrix reflects the correlation between signal vectors and provides the basis for subsequent direction vector estimation and beamforming. The calculation result of the covariance matrix can help us understand the propagation characteristics of signals and the interference situation between different signal channels.

[0075] The calculation purpose and principle of Formula 1.1 are as follows. By calculating the covariance matrix of the signal , the correlation between signals at different time points or different receiving channels can be understood. This is crucial for beamforming because it provides the basis for signal weighting processing (such as the calculation of beamforming weights). The calculation of the covariance matrix takes into account the statistical characteristics of the signal and can effectively reduce errors caused by multipath propagation and interference.

[0076] The technical effect of Formula 1.1 is as follows. This matrix provides the structural information of the signal, helps design a more accurate beam calibration method, improves the accuracy of beam weights, and thus optimizes the overall performance of the system.

[0077] Among them, in Formula 1.2 represents the nth noise vector. The noise vector represents the received noise signal, which is usually random interference unrelated to the real signal. represents the conjugate transpose of the nth noise vector, which has the same function as the conjugate transpose of the signal vector, that is, conjugating and transposing the complex noise signal elements; represents taking the expected value of multiple noise vectors in the time domain, and this expected value represents the average correlation between noise vectors in a statistical sense; It represents the "off" covariance matrix. The noise covariance matrix reflects the correlation between noise vectors and is also crucial for beam calibration and signal processing.

[0078] The calculation purpose and principle of Formula 1.2 are as follows: By calculating the covariance matrix of the noise , the correlation between noises at different time points or receiving channels can be understood. The noise covariance matrix is used to distinguish signals and noises and provides the noise statistical characteristics for subsequent beamforming and signal processing. The calculation of the noise covariance matrix provides a reliable noise model for subsequent signal enhancement, interference suppression, and direction estimation.

[0079] The technical effect of Formula 1.2 is as follows: This matrix is used to accurately distinguish noises and signals, provides detailed information about noise interference for the system, and then optimizes the beamforming algorithm, enabling it to effectively reduce the influence of noises and improve the performance and reliability of the system.

[0080] The calculation result of Formula 1.2 is to obtain the covariance matrix of the signal required for beam calibration based on the antenna information of each observation point. and the covariance matrix of the noise .

[0081] The calculation of the covariance matrix provides a basis for subsequent direction vector estimation and beamforming, ensuring the accuracy of beam weights and the optimization of system performance.

[0082] S2.3 According to the calculated covariance matrix, estimate the direction vectors of each pointing through the following formula :

[0083]

[0084] where in Formula 1.3 is the inverse matrix of the noise covariance matrix. By performing an inverse operation on the noise covariance matrix, it is used to find the principal eigenvector in the subsequent process; represents the estimated value of the covariance matrix of the signal, reflecting the correlation between the signals in different receiving channels or time points. It is calculated based on the observed data; is the direction vector, indicating the arrival direction of the target signal The spatial distribution on. It is usually related to the geometric structure of the array antenna and is used to describe the direction characteristics of the signal; is a scalar constant, representing the eigenvalue of this equation.

[0085] The calculation purpose and principle of Formula 1.3 are as follows: Based on the maximum signal-to-noise ratio rule, this formula provides a calculation basis for finding the principal eigenvector in the subsequent process. The aim is to improve the signal quality and maximize the received signal strength by optimizing the direction vector. Under this criterion, the goal is to maximize the ratio of signal to noise by adjusting the beamforming weights, thereby improving the overall signal quality. By adopting the maximum signal-to-noise ratio criterion, Formula 1.3 can effectively enhance the receptivity and processing accuracy of the signal, especially in a complex noise environment. This method can enhance the directivity of the signal, improve the direction estimation accuracy of the system, and maximize the gain of the target signal while suppressing the influence from noise and interference.

[0086] The calculation result of Formula 1.3 is to construct a formula for finding the principal eigenvector according to the antenna information of each observation point and the direction vector of each pointing based on the maximum signal-to-noise ratio rule.

[0087] where let , then the above formula becomes:

[0088]

[0089] Through eigenvalue decomposition, is the found main eigenvector. Therefore, the estimated beam direction vector is:

[0090]

[0091] where in Equation 1.4 is the inverse matrix of the noise covariance matrix. By performing the inverse operation on the noise covariance matrix, it is used to find the main eigenvector subsequently; represents the estimated value of the covariance matrix of the signal, which reflects the correlation between the signals in different receiving channels or time points. It is calculated based on the observed data; is the optimized direction vector, which is obtained by taking the inverse of the noise covariance matrix on the direction vector and is the main eigenvector of the direction vector; is a scalar constant representing the eigenvalue of this equation.

[0092] The calculation purpose and principle of Equation 1.4 are as follows: By introducing the main eigenvector , the influence of noise on signal processing is reduced, the direction vector is optimized, thereby improving the accuracy of signal reception. Under the maximum signal-to-noise ratio criterion, by using the inverse of the noise covariance matrix, the system can more accurately align the target signal and maximize its gain. By adopting the maximum signal-to-noise ratio criterion, the equation can effectively improve the receptivity and processing accuracy of the signal, especially in a complex noise environment. This method can enhance the directivity of the signal, improve the direction estimation accuracy of the system, and maximize the gain of the target signal while suppressing the influence from noise and interference.

[0093] The calculation result of Equation 1.4 is based on the antenna information of each observation point and the direction vector of each pointing, as well as introducing the found main eigenvector , to further derive the direction vector .

[0094] where in Equation 1.5 is the noise covariance matrix, which represents the correlation between the noises at different time points or receiving channels; is the direction vector representing the arrival direction of the target signal and the spatial distribution on it. It is usually related to the geometric structure of the array antenna and is used to describe the direction characteristics of the signal; is the optimized direction vector, which is obtained by taking the inverse of the noise covariance matrix on the direction vector and is the main eigenvector of the direction vector.

[0095] The calculation purpose and principle of Equation 1.5 are as follows: The direction vector is derived from the found main eigenvector: Equation 1.5 shows that the estimated beam direction vector is obtained by multiplying the optimized direction vector by the noise covariance matrix. The role of the noise covariance matrix is to adjust the direction vector so that it can accurately reflect the actual direction of the signal. Through this correction, the finally obtained beam direction vector is an optimized estimate, which can more accurately align with the direction of the target signal. This process effectively improves the accuracy and directivity of signal reception. Especially in an environment with noise and interference, it can significantly improve the quality and reliability of the signal.

[0096] The calculation result of Equation 1.5 is the direction vector calculated according to the antenna information of each observation point and the direction vector of each pointing according to the maximum signal-to-noise ratio rule. 。

[0097] S2.4 After the direction vector estimation is completed, based on the criterion of maximum signal-to-noise ratio (SNR), the beamforming weight is calculated. The weight formula is:

[0098]

[0099] where in Equation 1.6 the inverse matrix of the noise covariance matrix. The inverse of the covariance matrix is used to eliminate the interference of noise on the signal direction, so that the beamforming process can enhance the target signal as much as possible and suppress noise; is the direction vector representing the arrival direction of the target signal The spatial distribution on it. It is usually related to the geometric structure of the array antenna and is used to describe the direction characteristics of the signal; is the beamforming weight, which represents the weight coefficient of each array channel in the beamforming process. The beamforming weight adjusts the phase and amplitude of the signal to ensure that the signal is maximally enhanced along a specific direction (i.e., the direction of the target signal).

[0100] The purpose and principle of Formula 1.6 are as follows: It describes how to calculate the beamforming weight according to the maximum signal-to-noise ratio (SNR) criterion. The beamforming weight is obtained by multiplying the inverse matrix of the noise covariance matrix by the direction vector. The core goal of this process is to concentrate the gain of the beam in the direction of the target signal while suppressing the influence of noise as much as possible. Under the maximum SNR criterion, the beamforming weight maximizes the signal-to-noise ratio of the received signal in the target direction. The inverse operation of the noise covariance matrix can effectively eliminate the interference of noise on the beam direction and improve the accuracy of beamforming. Through the calculated beamforming weight, the beam can be optimized according to the direction of the target signal, thus ensuring the best signal gain. Through the calculated beamforming weight, the beam can effectively focus on the direction of the target signal and suppress the signals from noise and interference. This method provides higher signal gain and direction accuracy in a noisy environment, improving the performance of the system. The maximum SNR criterion ensures that the received gain of the target signal is optimized during the beamforming process, thereby improving the detection ability and signal quality of the system.

[0101] The calculation result of Formula 1.6 is the beamforming weight 。

[0102] S2.5 According to the weight matrix obtained from the pre-beam calibration and combined with the information of each beam in the array, use the following formula to calculate the beam after beam synthesis.

[0103]

[0104] where in Formula 1.7 The output signal of the th beam after beam synthesis in the direction. It is obtained by linearly combining the beam synthesis weight and the received signal; is the conjugate transpose of the weight vector of the th beam in the direction. This weight vector has been pre-beam calibrated and optimized, representing the amplitude and phase weights of each channel in the array, and is used to adjust the signal reception direction; is the signal vector received by the th beam, which contains the signals received by all antenna elements in the array.

[0105] The purpose and principle of Formula 1.7 are as follows: The core idea of beamforming is to perform weighted summation on the received signal vector through the weight vector to enhance the signal in the target direction while suppressing the noise and interference in other directions. Through the conjugate transpose of the weight vector, it multiplies with the received signal vector to obtain a scalar value, which is the synthesized beam output signal. The target adjusts the weight vector to make the directivity of the beam concentrated in the target direction, thereby maximizing the signal gain and reducing the influence of noise and interference. The application of Formula 1.7 can effectively enhance the signal in the target direction while suppressing the noise and interference from other directions, improving the signal reception performance of the system. The signal after beamforming is more concentrated in the target signal direction, which can provide higher-quality input for subsequent data processing and analysis.

[0106] The calculation result of Formula 1.7 is the beam vector information after beamforming. 。

[0107] S2.6 Integrate based on the synthesized beam and enhance the signal in the direction of the target signal source, and transmit the result to the server memory.

[0108] Step S3, in the data writing to disk module, during the data writing to disk stage, use the data storage client (dada_dbdisk) of PSRDADA. The core function of the data writing to disk module is to write data blocks to the disk and split the data into files of arbitrary length. The beginning of each file contains the file header from the data block. After each file is written to the disk, a record is added to the ASCII text log file; each record contains the full path of the file, the writing time, the file size, the time required to write the file, and the observation identifier. After the server memory receives the data passed in from the GPU circular buffer, it will use the data storage client to write the data to the disk for storage. If other server computing nodes or storage nodes need to be accessed subsequently, it can be sent to the switch through the network card. The overall process of writing to disk is as Figure 5 shown. The process is that when the CPU circular buffer receives the processed result data from the GPU, it uses the data storage client of PSRDADA to write the data block to the disk and split the data into files of arbitrary length. The beginning of each file will contain the file header from the data block. If other server computing nodes or storage nodes need to be accessed subsequently, it can be sent to the switch through the network card.

[0109] The performance evaluation of the overall real-time data stream processing system mainly depends on three core metrics: data transmission rate, data processing rate, and data disk writing rate. These metrics directly affect the real-time performance and processing ability of the system. Therefore, the present invention conducts in-depth tests and analyses from these aspects. Specifically, the research will evaluate the transmission and processing efficiency of data at each stage of the system to ensure that the system can still operate stably under high input-output load environments. At the same time, the efficiency of writing data to the storage medium will also be investigated to verify the reliability of the system under long-term operation. Through these tests, the feasibility analysis of the entire system can be supported.

[0110] The list of test environment parameters for the embodiments of the present invention is as follows:

[0111] Table 1 Test Environment Parameters

[0112]

[0113] The test method includes the following steps:

[0114] 1. First, create two 9GB circular buffers in the server memory, one for receiving UDP packet data and the other for receiving the data after GPU data processing. Each circular buffer contains a default 4096-byte file header and is divided into 8 data blocks of 1.05GB each.

[0115] 2. Then, create a 9GB circular buffer in the GPU video memory. This buffer includes a default 4096-byte file header and 8 data blocks of 1.05GB each.

[0116] 3. Next, use the PSRDADA network simulation sending tool (dada_junkdb) to simulate the UDP packet transmission process. Set the PSRDADA network simulation sending tool to use the zero-copy technology, transmit 5000MB of data per second for 20 seconds, and the transmitted content is Gaussian noise data.

[0117] 4. Start the written transfer_cputogpu script to test the data transmission rate from the CPU circular buffer to the GPU circular buffer.

[0118] 5. Start the written transfer_gputocpu script to test the data transmission rate from the GPU circular buffer to the CPU circular buffer.

[0119] 6. Finally, use the data storage client tool to write the processed data transferred from the GPU to the CPU to the hard disk.

[0120] The test results are as follows:

[0121] 1. Test the data transfer rate from the CPU circular buffer to the GPU circular buffer. It takes 22 seconds to transfer 100.7 GB of data, and the data transfer rate is approximately 4.2 GB / s.

[0122] 2. Test the data transfer rate from the GPU circular buffer to the CPU circular buffer. It takes 21 seconds to transfer 101.7 GB of data, and the data transfer rate is approximately 4.6 GB / s.

[0123] 3. Test the data transfer rate of the CPU circular buffer writing to disk. It takes 73 seconds to write 101.7 GB of data, and the data transfer rate is approximately 1.3 GB / s.

[0124] Based on the current requirements of radio astronomy observations for high sensitivity and high resolution, and to address the urgent problem of the need for real-time processing of high-bandwidth radio signals collected by phased arrays in digital terminals, the present invention constructs a real-time data stream processing framework for radio astronomy data using distributed acquisition components. The aim is to solve the problem of data quality degradation caused by data packet loss in a high input / output network environment, thereby achieving high-speed processing of real-time transmitted radio astronomy data streams. In addition, by abstracting data blocks and processing processes using distributed acquisition components, the real-time data stream processing framework is highly modularized, making each processing stage more flexible. This approach not only optimizes the development process but also enables the efficient flow of real-time data streams across data blocks, thereby enhancing the overall performance and scalability of the processing framework.

[0125] Based on the constructed real-time data stream processing framework, the present invention integrates and optimizes the beamforming algorithm to complete the real-time data stream processing system. The phased array samples the electromagnetic field of the reflector focal plane through a large-scale feed array and uses the terminal beamforming network to perform weighted superposition on the sampled data, ultimately generating a beam distribution covering a continuous sky region. Therefore, it is necessary to integrate the beamforming algorithm to achieve the required spatial response pattern; at the same time, perform the correlation matrix operation of the array element signals for subsequent beam calibration and generation of the optimal weight factor.

[0126] The present invention conducts a comprehensive performance test on the implemented real-time data stream processing system to verify its real-time performance and usability in practical applications. Through performance testing, the stability and data processing efficiency of the system in a high input / output load environment can be evaluated, thereby ensuring that the system can meet the requirements of radio astronomy observations.

[0127] Any process or method description described in the flowchart of the present invention or otherwise herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process, which can be implemented in any computer scale medium for an instruction execution system, apparatus, or device. The computer-readable medium can be any medium that contains, communicates, propagates, or transports a program for use by an instruction execution system, apparatus, or device. It includes read-only memory, magnetic disks, or optical discs, etc.

[0128] In the description of this specification, the descriptions referring to the terms "embodiment", "example", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without contradiction.

[0129] Although the above has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can perform update operations such as changes, modifications, substitutions, and variations on the above embodiments within the scope of the present invention.

Claims

1. A method for processing real-time radio astronomy data stream, characterized in that: The method comprises the following steps: S1. Data transmission: The server receives UDP data packets from the radio telescope through the network card, uses the open source pulsar distributed acquisition component and data analysis system PSRDADA to create and manage ring buffers, and transmit data streams between each ring buffer; the UDP data packet contains the digital signal and observation information encapsulated by the radio telescope after the radio telescope performs fast Fourier transform processing through FPGA; S2. Data processing: using the CUDA parallel computing platform and programming model of the graphics processing unit GPU, write computing and unpacking programs to achieve parallel processing and data extraction of data; GPU is used to process parallel tasks and delay-sensitive tasks; S3. Data writing: Use PSRDADA's data storage client to write data blocks to disk and split the data into files of any length. Step S2 includes the following steps: S2.1 GPU reads the data in the ring buffer in the video memory and parses the observation data based on the metadata in the file header; the parsing process includes extracting the observation timestamp and UDP data size; locating the data block address corresponding to the spectrum according to the unpacked metadata; S2.2 Based on the analyzed data, the analyzed spectrum information is pre-beam-calibrated; the calibration process includes calculating the signal covariance matrix required for beam calibration based on the antenna information of each observation point. and the noise covariance matrix ; S2.3 Based on the calculated signal covariance matrix and the noise covariance matrix According to the maximum signal-to-noise ratio rule, the direction vectors of each beam direction are estimated by the following formula: ； in, The inverse matrix of the noise covariance matrix is ​​used to calculate the main eigenvectors later by performing an inverse operation on the noise covariance matrix; Represents the estimated value of the signal covariance matrix, which reflects the correlation between the signals in different receiving channels or time points. It is calculated based on the observed data. is the direction vector, indicating the arrival direction of the target signal The spatial distribution on the antenna array is related to the geometric structure of the antenna array and is used to describe the directional characteristics of the signal. is a scalar constant representing the eigenvalue of the equation; Among them , then the above formula is: ; Through feature decomposition, is the main eigenvector found, and the estimated beam direction vector is: ； S2.4 After the direction vector is estimated, the beamforming weight is calculated based on the maximum signal-to-noise ratio criterion. The beamforming weight represents the weight coefficient of each array channel in the beamforming process. The beamforming weight adjusts the phase and amplitude of the signal to ensure that the signal is maximized along the direction of the target signal. The beamforming weight is obtained by multiplying the inverse matrix of the noise covariance matrix by the direction vector. S2.5 calculating the beam after beam synthesis according to the weight matrix obtained by the above-mentioned pre-beam calibration and the signal vector received by each beam; S2.6 integrates and enhances the signal in the direction of the target signal source according to the synthesized beam, and transmits the result to the server memory.

2. The method for processing radio astronomy real-time data stream according to claim 1, characterized in that: In step S1, PSRDADA creates multiple circular buffers, each of which is allocated as a shared memory resource and is logically divided into a file header describing observation information and multiple data blocks storing data.

3. The method for processing radio astronomy real-time data stream according to claim 1 or 2, characterized in that: In step S1, the data stream transmission process includes: S1.1 At the beginning of the observation, after the server receives the UDP data packet from the front-end radio telescope through the network card, it first uses the data flow control software of PSRDADA to create a circular buffer in the server memory, and uses direct memory access DMA to read the network card data; each data block in the buffer is initially empty, and its file header is initialized to information related to the observation; the information includes bandwidth, center frequency, signal source, and start time; S1.2 When the ring buffer in the server memory receives the data, the file header of the buffer is parsed to obtain the observation information; then, a ring buffer is created on the GPU using the data flow control software of PSRDADA, and the transfer script transfer_cputogpu from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the ring buffer of the GPU; After the S1.3 GPU completes data processing, it first creates a ring buffer in the server memory using the data flow control software of PSRDADA, and then transfers the result data to the server memory using the transfer script transfer_gputocpu; S1.4 During the entire observation process, data is written into data blocks in sequence; when the observation ends, the data end flag is set, and the last complete data block and the number of valid bytes written are recorded; after that, data is read from the data blocks in sequence until the data end flag is encountered and the current data block is the last complete data block.

4. The method for processing radio astronomy real-time data stream according to claim 1, characterized in that: In step S2.2, the covariance matrix formula is as follows: ； ； In formula 1.1, represents the nth signal vector, represents the conjugate transpose of the nth signal vector; It represents the expected value of multiple signal vectors in the time domain; represents the signal covariance matrix, which reflects the correlation between signal vectors; In formula 1.2 represents the nth noise vector, represents the conjugate transpose of the nth noise vector; Represents the expected value of multiple noise vectors in the time domain; Represents the noise covariance matrix, which reflects the correlation between noise vectors.

5. The method for processing radio astronomy real-time data stream according to claim 1, characterized in that: Step S3, data writing stage, the beginning of each file contains a file header from the data block; after each file is written to the disk, a record is added to the ASCII text log file.

6. The method for processing radio astronomy real-time data stream according to claim 5, characterized in that: Each record contains the full path to the file, the time it was written, the file size, the time it took to write the file, and an observation identifier.

Images