Radio astronomy real-time data stream processing method and system based on distributed acquisition component

Abstract

The invention discloses a radio astronomical real-time data stream processing method and system based on a distributed acquisition component, and the method comprises the steps: S1, data transmission: a server receives a user datagram protocol UDP data packet transmitted by a radio astronomical telescope through a network card, and carries out the data transmission through an open-source pulsar distributed acquisition component and a data analysis system PSRDADA; creating and managing annular buffers, and transmitting data streams among the annular buffers; s2, data processing: using a CUDA (Unified Computing Device Architecture) parallel computing platform and a programming model of a GPU (Graphics Processing Unit) to write a computing and unpacking program so as to realize parallel processing and data extraction of large-scale data; and S3, writing data into a disk, writing the data blocks into the disk by using a data storage client of the PSRDADA, and dividing the data into files with any length. According to the invention, high-speed radio data streams can be received efficiently and stably, real-time processing of data is realized, and a reliable solution is provided for high-bandwidth signal transmission and processing in radio astronomical observation.

Description

Technical Field

[0001] The present invention relates to the technical field of astronomical observation and signal processing, and mainly to a method and system for processing real-time data stream of radio astronomy based on distributed acquisition components. Background Art

[0002] Phased array is an emerging array antenna receiving system. Due to its advantages in single-beam gain, multi-beam continuous sky coverage and larger field of view, it is gradually being used in radio telescopes, and can significantly accelerate the survey rate in conjunction with large-aperture telescopes. However, the high-bandwidth radio signals received by the phased array place higher requirements on the real-time, large storage capacity, high availability and durability of digital terminals. Under high input and output load environments, traditional digital terminals often have problems such as packet loss, disorder and duplication, which significantly affect the transmission quality. In addition, the traditional data stream processing framework based on distributed acquisition components usually writes the data transmitted by the high-speed network card directly into the graphics processor (Graphic Process Unit, hereinafter referred to as GPU) video memory without introducing a ring buffer for data buffering. When the transmission rate exceeds the processing rate, this design is prone to data loss or processing delays, and cannot meet the needs of real-time data stream processing.

[0003] In response to the above challenges, this study developed a real-time streaming data processing framework based on distributed acquisition components, including data transmission, processing and disk writing. This framework can efficiently and stably receive high-speed radio data streams observed by phased arrays, and realize real-time data processing, providing a reliable solution for high-bandwidth signal transmission and processing in radio astronomy observations.

[0004] Since the observation source signal of radio astronomy is extremely weak, generally 30-50dB (decibel) below the noise of the receiving system, a very high sensitivity of the receiving system is required. In view of the high sensitivity and resolution requirements of radio astronomy observations, the usual solution is to increase the effective area of ​​the antenna and reduce the system noise temperature (Wang Zhijie et al., 2008). However, increasing the effective area of ​​the antenna leads to a narrower beam width and reduces the spatial coverage of a single beam (Zhang Guangyi, 2007). Although the traditional multi-beam feed group can expand the observation field of view, it cannot achieve complete coverage of beam overlap.

[0005] Compared with traditional single-beam feed and horn feed arrays, phased array technology performs better in single-beam gain and multi-beam continuous sky coverage, and has a larger field of view, so continuous sky coverage can be achieved. Phased arrays can control the amplitude and phase of each array element through a beam synthesis network to adjust the performance of the defocused beam to a level equivalent to that of the main focus beam. Compared with multi-horn feed arrays, the use of phased array feed technology can perfectly solve the problems of uneven illumination beam width and incomplete coverage of densely distributed and overlapping beams within the ultra-wideband range.

[0006] The Linux kernel (an open source operating system kernel used as a server operating system) has a large overhead when processing broadband network traffic, which is mainly manifested in multiple replications of data packets, context switching, and complex processing flows of the protocol stack. Since digital terminals use User Datagram Protocol (UDP) data packets, as network bandwidth continues to increase, the kernel's processing bottleneck gradually emerges. Under high input and output load environments, this bottleneck can lead to network data packet loss, disorder, duplication, and other problems, which significantly affects the quality of data transmission. For pulsar data, network data packet loss, disorder, and duplication problems will have a serious impact on pulsar period prediction and folding.

[0007] In the traditional radio data stream processing framework, data processing is usually performed by directly allocating space in the GPU video memory without creating a buffer. This practice leads to frequent memory access, which reduces the computing rate. Especially when high-speed data transmission is required between the central processing unit (CPU) and the GPU, and the graphics processor performs efficient real-time data processing, the communication and data access rate between the CPU and the graphics processor becomes a performance bottleneck. Summary of the invention

[0008] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method for processing real-time data streams in radio astronomy based on distributed acquisition components, and to develop a real-time stream data processing framework including data transmission, processing and disk writing based on distributed acquisition components. The present invention can efficiently and stably receive high-speed radio data streams observed by phased arrays, and realize real-time data processing, providing a reliable solution for high-bandwidth signal transmission and processing in radio astronomy observations.

[0009] To achieve the above object, the present invention provides a method for processing real-time radio astronomy data stream based on a distributed acquisition component, the method comprising the following steps: S1. Data transmission: The server receives UDP data packets from the radio telescope through the network card, uses the open source Pulsar Distributed Acquisition and Data Analysis (PSRDADA) system to create and manage ring buffers, and transmits data streams between each ring buffer. S2. Data processing: using the GPU's unified computing device architecture (CUDA) parallel computing platform and programming model to write computing and unpacking programs, thereby achieving parallel processing and data extraction of large-scale data; S3. Write data to disk. Use PSRDADA's data storage client (dada_dbdisk) to write data blocks to disk and split the data into files of any length.

[0010] Furthermore, in step S1, the distributed acquisition component PSRDADA creates multiple circular buffers, each of which is allocated as a shared memory resource and is logically divided into a file header describing the observation information and multiple data blocks storing data.

[0011] Further, in step S1, the data stream transmission process includes: S1.1 At the beginning of the observation, after the server receives the UDP data packet from the front-end radio telescope through the network card, it first uses the PSRDADA data flow control software (dada_db) to create a circular buffer in the server memory and uses direct memory access to read the network card data; each data block in the buffer is initially empty, and its file header is initialized to information related to the observation; the information includes bandwidth, center frequency, signal source, and start time; S1.2 When the ring buffer in the server memory receives the data, the file header of the buffer is parsed to obtain the observation information; then, a ring buffer is created on the GPU using the data flow control software, and the transfer script transfer_cputogpu from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the ring buffer of the GPU; After the S1.3 GPU completes data processing, it first creates a ring buffer in the server memory using the data flow control software, and then transfers the result data to the server memory using transfer_gputocpu; S1.4 During the entire observation process, data is written to the data block in sequence. When the observation ends, the data end flag is set, and the last complete data block and the number of valid bytes written are recorded. After that, data is read from the data block in sequence until the data end flag is encountered and the current data block is the last complete data block.

[0012] Furthermore, in step S2, the GPU is used to process parallel tasks, and the CPU is used to process delay-sensitive tasks.

[0013] Further, step S2 includes the following steps: S2.1 GPU reads the data in the ring buffer in the video memory and parses the observation data based on the metadata in the file header; the parsing process includes extracting the observation timestamp and UDP data size; locating the data block address corresponding to the spectrum according to the unpacked metadata; S2.2 Based on the analyzed data, the analyzed spectrum information will be pre-calibrated for beams; the calibration process calculates the covariance matrix for each observation point; S2.3 Estimate the direction vectors of each beam pointing based on the calculated covariance matrix : S2.4 after the direction vector estimation is completed, the beamforming weight is calculated based on the maximum signal-to-noise ratio criterion; S2.5 calculating the beam after beam synthesis according to the weight matrix obtained by the pre-beam calibration and combining the information of each beam in the array; S2.6 integrates and enhances the signal in the direction of the target signal source according to the synthesized beam, and transmits the result to the server memory.

[0014] Further, it is characterized in that, in step S2.2, the covariance matrix formula is as follows: In formula 1.1, represents the nth signal vector, represents the conjugate transpose of the nth signal vector; It represents the expected value of multiple signal vectors in the time domain; Represents the "on" covariance matrix.

[0015] In formula 1.2 represents the nth noise vector, represents the conjugate transpose of the nth noise vector; Represents the expected value of multiple noise vectors in the time domain; Represents the "off" covariance matrix.

[0016] Furthermore, in step S3, the data writing stage, the beginning of each file includes a file header from the data block; after each file is written to the disk, a record is added to the ASCII (American Standard Code for Information Interchange) text log file.

[0017] Furthermore, each record contains the full path to the file, the time it was written, the file size, the time it took to write the file, and the observation identifier.

[0018] Furthermore, when the data writing module needs to access other server computing nodes or storage nodes, it is sent to the switch through the network card.

[0019] On the other hand, the present invention provides a radio astronomy real-time data stream processing system based on distributed acquisition components, the system is used to implement the above-mentioned radio astronomy real-time data stream processing method based on distributed acquisition components, and the system includes a data transmission module, a data processing module and a data disk writing module.

[0020] Beneficial effects: The present invention designs and implements a set of framework programs that can support high-speed data packet transmission, reception and preprocessing for digital terminals, which is called a pipeline. The pipeline realizes efficient flow of data from reception to processing and then to writing to disk through a ring buffer, ensuring that the processing of each stage can be completed under real-time requirements. By calling program modules at different stages, the pipeline can ensure the efficient connection between various links such as data flow in transmission, processing, and storage, thereby meeting the strict requirements for real-time performance. The pipeline not only needs to effectively cope with data traffic under high input and output loads, but also should have flexible scalability and high-performance computing capabilities, and can transmit and process massive data in real time. To this end, the pipeline should be able to optimize the processing path, reduce kernel intervention and efficiently allocate computing resources while ensuring smooth data transmission. This not only helps to improve the response speed of the system, but also significantly improves the overall operating efficiency, ensures the stability and efficiency of the system under long-term operation, and thus supports radio astronomy observation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall architecture diagram of the radio astronomy real-time data stream processing method and system based on the distributed acquisition component of the present invention; Figure 2 It is a schematic diagram of the data flow of data transmission; Figure 3 This is a schematic diagram of the GPU heterogeneous computing principle; Figure 4 It is a schematic diagram of the data processing process; Figure 5 This is a schematic diagram of the data writing process. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0024] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0025] The following combination Figure 1-Figure 5 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0026] Based on the current demand for high sensitivity and high resolution in radio astronomy observations, as well as the urgent need for real-time processing of high-bandwidth radio signals observed by phased arrays collected by digital terminals, the primary research content of this paper is to use distributed acquisition components to build a real-time data stream processing framework for radio astronomy data. This study aims to solve the problem of data quality degradation caused by data packet loss in a high-input and output network environment, thereby achieving high-speed processing of real-time transmitted radio astronomy data streams. In addition, by abstracting data blocks and processing processes using distributed acquisition components, the real-time data stream processing framework is highly modularized, making each processing stage more flexible. This approach not only optimizes the development process, but also enables the real-time data stream to flow efficiently in each data block, thereby improving the overall performance and scalability of the processing framework.

[0027] like Figure 1 As shown, Figure 1The figure is the overall architecture diagram of the radio astronomy real-time data stream processing method and system based on distributed acquisition components. The process is to enter the data transmission stage at the beginning of the observation. After the server receives the UDP data packet from the front end through the network interface card (NIC), it first uses the PSRDADA data flow control software to create a ring buffer in the server memory. When the ring buffer in the server memory receives the data, it will parse the file header of the buffer to obtain the observation information. Subsequently, a ring buffer is created on the GPU using the data flow control software, and the transmission script from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the GPU's ring buffer.

[0028] After entering the data processing stage, the GPU ring buffer receives the data and parses the observation data based on the metadata in the file header. Then the beam calibration is performed in advance to calculate the weight matrix . Based on the weight matrix obtained by the pre-beam calibration and combined with the information of each beam in the array, the beam after beam synthesis is calculated using the following formula. Based on the synthesized beam, the signal in the direction of the target signal source is integrated and enhanced. A ring buffer is created in the server memory using the data flow control software, and the transfer script from the GPU memory to the CPU video memory is started to transfer the data in the server memory to the CPU ring buffer.

[0029] Finally, in the data writing stage, the data is split into files of any length using the PSRDADA data storage client. The beginning of each file will contain the file header from the data block. If it is necessary to access other server computing nodes or storage nodes later, it can be sent to the switch through the network card.

[0030] The radio astronomy real-time data stream processing system based on distributed acquisition components according to the present invention includes the following three modules: a data transmission module (data transmission stage), a data processing module (data processing stage), and a data writing module (data writing stage).

[0031] The main function of the data transmission module is to create and manage the ring buffer, and to build the transmission link based on the ring buffer and the device using the script, and to manage the flow of real-time data. Its input is the UDP data packet from the switch, which contains the radio signal encapsulated into a digital signal and the specified observation information after the radio telescope performs FFT (Fast Fourier transform) processing through FPGA (Field-Programmable Gate Array). The output is the result data transmitted from the GPU video memory to the CPU memory.

[0032] The main function of the data processing module is to parse the file header in the ring buffer, extract specific observation information, and process the observation data in the ring buffer based on this information. Its input is the data transmitted from the CPU memory of the data transmission module to the GPU memory, and its output is the result data of integration and enhancement after beam synthesis.

[0033] The main function of the data writing module is to write multiple data blocks in the ring buffer to the disk and split the data into files of any length. If it needs to be connected to other server computing nodes or storage nodes, it can be sent to the switch through the network card. Its input is multiple data blocks in the ring buffer and a file header. The output is a FITS (Flexible Image Transport System) file or an encapsulated UDP data packet.

[0034] According to these three modules, correspondingly, the method for processing radio astronomy real-time data stream based on distributed acquisition components according to the present invention includes the following three steps: S1. Data transmission, the server receives UDP data packets from the radio telescope through the network card, using the open source distributed acquisition component PSRDADA, whose core function is to create and manage ring buffers and transmit data streams between each ring buffer.

[0035] S2. Data processing: Use the GPU's CUDA parallel computing platform and programming model to write computing and unpacking programs to achieve parallel processing of large-scale data and fast data extraction.

[0036] S3. Data writing to disk, using PSRDADA’s data storage client, whose core function is to write data blocks to disk and split the data into files of any length.

[0037] Specifically, in step S1, in the data transmission module, PSRDADA creates multiple ring buffers, each of which will be allocated as a shared memory resource and logically divided into a header block describing the observation information and multiple data blocks storing data. Each data block will have an associated byte counter for calculating the time offset from the start of the observation. At any time, only one continuous data stream can be represented in the data block; therefore, the size of the data block will determine the maximum time required to clear the ring buffer from the end of one observation to the start of the next observation.

[0038] The overall data flow transmission is as follows Figure 2 As shown in the figure, the process is that the switch sends a UDP data packet to the server. After the server network card receives the UDP data packet, it first uses the data flow control software of PSRDADA to create a circular buffer in the server memory, and uses direct memory access (DMA) to read the network card data. Each data block of the buffer is initially empty, and its file header will be initialized to information related to the observation (information related to the observation includes bandwidth, center frequency, signal source, start time, etc.). Before the data is officially valid, in order to start data collection in advance, the data block allows data to be written before the data start flag is set. Only when the data start flag is set, data can be read from the data block.

[0039] When the ring buffer in the server memory receives the data, it will parse the file header of the buffer to obtain the observation information. Then, a ring buffer is created on the GPU using the data flow control software, and the transfer script from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the ring buffer of the GPU.

[0040] Similarly, after the GPU completes data processing, it first creates a ring buffer in the server memory with the data flow control software, and then starts the transfer script from the GPU memory to the CPU video memory to transfer the data in the server memory to the CPU's ring buffer.

[0041] Step S1 specifically includes: S1.1 At the beginning of the observation, after the server receives the UDP (User Datagram Protocol) data packet from the front end (radio telescope) through the network interface card (NIC), it first uses the data flow control software of PSRDADA to create a circular buffer in the server memory and reads the network card data using direct memory access. Each data block in the buffer is initially empty, and its file header will be initialized to information related to the observation (such as bandwidth, center frequency, signal source, start time, etc.). Before the data is officially valid, in order to start data collection in advance, the data block allows data to be written before the data start flag is set. Only when the data start flag is set, data can be read from the data block.

[0042] S1.2 When the ring buffer in the server memory receives the data, it will parse the file header of the buffer to obtain the observation information. Then, a ring buffer is created on the GPU using the data flow control software, and the transfer script transfer_cputogpu from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the ring buffer of the GPU.

[0043] After the S1.3 GPU completes data processing, it first creates a ring buffer in the server memory using the data flow control software, and then transfers the result data to the server memory using transfer_gputocpu.

[0044] S1.4 During the entire observation process, data is written into the data block in sequence. When the observation ends, the data end flag is set, and the last complete data block and the number of valid bytes written are recorded. After that, data will be read from the data block in sequence until the data end flag is encountered and the current data block is the last complete data block.

[0045] Step S2, in the data processing module, the data processing stage uses the GPU's CUDA (Compute Unified Device Architecture) parallel computing platform and programming model to write efficient computing and unpacking programs, thereby realizing parallel processing of large-scale data and fast data extraction. This system adopts a heterogeneous computing architecture, in which the GPU is dedicated to accelerating the part of the code with parallel characteristics. Because the GPU can efficiently perform parallel computing tasks, it is often called an "accelerator." In this mode, the GPU is responsible for processing parallel tasks, including beam synthesis, beam calibration, and parallel processing of ring buffer file header unpacking. The CPU focuses on processing delay-sensitive tasks, including real-time reception of UDP data packets, parsing UDP data packet headers to obtain observation information, and real-time transmission of data. By combining a high-performance CPU and a high-throughput GPU, the overall performance of the application can be significantly improved. Figure 3 The schematic diagram of parallel computing based on heterogeneous computing mode is shown. The graphics processing unit (GPU) has multiple parallel computing units, which are mainly responsible for parallel tasks and most matrix operations in the server. The central processing unit (CPU) has multiple computing cores and is responsible for processing continuous tasks, including real-time data transmission, as well as system and sequential tasks, such as receiving UDP data packets and parsing UDP packet headers.

[0046] CPUs excel at processing latency-bound parts of the code, while GPUs excel at running code in parallel in a Single Instruction Multiple Data (SIMD) manner. If optimizations only make one part of the CPU code or GPU code run faster, this will not necessarily significantly increase the speed of the overall application. To maximize performance, both processors, the CPU and the GPU, need to work together at their optimal state. This practice of offloading certain types of operations from the processor to the GPU is called heterogeneous computing.

[0047] The overall process of data processing is as follows Figure 4As shown in the figure, the process is that when the GPU ring buffer receives the data of data transmission, the GPU reads the data in the ring buffer in the video memory and parses the observation data based on the metadata in the file header. The parsing process includes extracting the key information of the observation timestamp (Julian day, MJD_start), the number of feed array elements (nelement), the number of synthesized beams (nbeam), the size of the UDP header (pkt_header), and the size of the UDP data (pkt_data). Subsequently, the data block address corresponding to the spectrum is located according to the unpacked metadata. Since the spectrum data is in complex form, the cuComplex format is used to store and process the parsed spectrum matrix information. Based on the parsed data, the parsed spectrum information will be pre-beam calibrated. Beam calibration calculates the covariance matrix of the signal and noise, and then calculates each beam direction vector, and calculates the beam weight matrix according to each beam direction vector. In the beam synthesis stage, the beam synthesized beam is calculated according to the weight matrix obtained by the pre-beam calibration and combined with the information of each beam in the array. Finally, according to the synthesized beam, the signal in the direction of the target signal source is integrated and enhanced, and the result is transmitted to the server memory.

[0048] Step S2 specifically includes: The S2.1 GPU reads the data in the ring buffer in the video memory and parses the observation data based on the metadata in the file header. The parsing process includes extracting key information such as the observation timestamp (Julian day, MJD_start), the number of feed array elements (nelement), the number of synthesized beams (nbeam), the UDP header size (pkt_header), and the UDP data size (pkt_data). Subsequently, the data block address corresponding to the spectrum is located according to the unpacked metadata. Since the spectrum data is in complex form, the cuComplex format is used to store and process the parsed spectrum matrix information.

[0049] S2.2 will pre-calibrate the beam on the parsed spectrum information based on the parsed data. The calibration process calculates the covariance matrix for each observation point. The formula is as follows: In formula 1.1, represents the nth signal vector, which contains multiple signal components, representing the signals received by different sensors or receiving antennas. Represents the conjugate transpose of the nth signal vector. The conjugate transpose refers to taking the conjugate and transposing each element in the complex signal vector. It is usually used to process complex domain data in signal processing. It represents the expected value of multiple signal vectors in the time domain, which represents the average correlation between signal vectors in a statistical sense; Represents the "on" covariance matrix. This matrix reflects the correlation between signal vectors, which provides the basis for subsequent direction vector estimation and beamforming. The calculation results of the covariance matrix can help us understand the propagation characteristics of the signal and the interference between different signal channels.

[0050] The purpose and principle of calculation of formula 1.1 are as follows: by calculating the covariance matrix of the signal , we can understand the correlation of signals at different time points or between different receiving channels. This is crucial for beamforming because it provides a basis for weighted processing of signals (such as the calculation of beamforming weights). The calculation of the covariance matrix takes into account the statistical characteristics of the signal and can effectively reduce errors caused by multipath propagation and interference.

[0051] The technical effect of formula 1.1 is as follows: the matrix provides structural information of the signal, helps design a more accurate beam calibration method, improves the accuracy of the beam weight, and thus optimizes the overall performance of the system.

[0052] In formula 1.2 represents the nth noise vector, which represents the received noise signal, usually a random interference that has nothing to do with the real signal. represents the conjugate transpose of the nth noise vector, which has the same effect as the conjugate transpose of the signal vector, i.e., conjugate and transpose the complex noise signal elements; It represents the expected value of multiple noise vectors in the time domain, which represents the average correlation between noise vectors in a statistical sense; Represents the "off" covariance matrix. The noise covariance matrix reflects the correlation between noise vectors and is also crucial for beam calibration and signal processing.

[0053] The purpose and principle of calculation of formula 1.2 are as follows: By calculating the covariance matrix of the noise , we can understand the correlation of noise at different time points or between receiving channels. The noise covariance matrix is ​​used to distinguish between signals and noise and provide noise statistical characteristics for subsequent beamforming and signal processing. The noise covariance matrix calculation provides a reliable noise model for subsequent signal enhancement, interference suppression and direction estimation.

[0054] The technical effect of formula 1.2 is as follows: This matrix is ​​used to accurately distinguish between noise and signals, provide the system with detailed information on noise interference, and then optimize the beamforming algorithm so that it can effectively reduce the impact of noise and improve the performance and reliability of the system.

[0055] The calculation result of formula 1.2 is to find the covariance matrix of the signal required for beam calibration based on the antenna information of each observation point. and the noise covariance matrix .

[0056] The calculation of the covariance matrix provides a basis for subsequent direction vector estimation and beamforming, ensuring the accuracy of beam weights and optimization of system performance.

[0057] S2.3 Based on the calculated covariance matrix, estimate the direction vector of each direction using the following formula : In formula 1.3, The inverse matrix of the noise covariance matrix. The inverse operation of the noise covariance matrix is ​​used to find the main eigenvector later; Represents the estimated value of the covariance matrix of the signal, reflecting the correlation of the signal between different receiving channels or time points. It is calculated based on the observed data; is the direction vector representing the arrival direction of the target signal It is usually related to the geometric structure of the array antenna and is used to describe the directional characteristics of the signal; is a scalar constant representing the eigenvalue of the equation.

[0058] The purpose and principle of calculation of formula 1.3 are as follows: This formula provides a calculation basis for the subsequent main eigenvector according to the maximum signal-to-noise ratio rule. It aims to improve the quality of the signal and maximize the received signal strength by optimizing the direction vector. Under this criterion, the goal is to maximize the ratio of signal to noise by adjusting the beamforming weights, thereby improving the overall signal quality. By adopting the maximum signal-to-noise ratio criterion, formula 1.3 can effectively improve the receivability and processing accuracy of the signal, especially in complex noise environments. This method can enhance the directionality of the signal, improve the direction estimation accuracy of the system, and maximize the gain of the target signal while suppressing the influence of noise and interference.

[0059] The calculation result of formula 1.3 is based on the antenna information of each observation point and the direction vector of each pointing, and the formula for calculating the main eigenvector is constructed according to the maximum signal-to-noise ratio rule.

[0060] Among them , then the above formula is: Through feature decomposition, is the main eigenvector found. Therefore, the estimated beam direction vector is: In formula 1.4, The inverse matrix of the noise covariance matrix. The inverse operation of the noise covariance matrix is ​​used to find the main eigenvector later; Represents the estimated value of the covariance matrix of the signal, reflecting the correlation of the signal between different receiving channels or time points. It is calculated based on the observed data; The optimized direction vector is obtained by The inverse of the noise covariance matrix is ​​obtained, which is the main eigenvector of the direction vector; is a scalar constant representing the eigenvalue of the equation.

[0061] The purpose and principle of calculation of formula 1.4 are as follows: By introducing the main eigenvector , reduce the impact of noise on signal processing, optimize the direction vector, and thus improve the accuracy of signal reception. Under the maximum signal-to-noise ratio criterion, by using the inverse of the noise covariance matrix, the system can more accurately align the target signal and maximize its gain. By adopting the maximum signal-to-noise ratio criterion, the formula can effectively improve the receivability and processing accuracy of the signal, especially in complex noise environments. This method can enhance the directionality of the signal, improve the system's direction estimation accuracy, and maximize the gain of the target signal while suppressing the impact of noise and interference.

[0062] The calculation result of formula 1.4 is based on the antenna information of each observation point and the direction vector of each pointing, as well as the main eigenvector found by introducing , to further deduce the direction vector .

[0063] In formula 1.5, The noise covariance matrix represents the correlation of noise at different time points or between receiving channels; is the direction vector indicating the arrival direction of the target signal It is usually related to the geometric structure of the array antenna and is used to describe the directional characteristics of the signal; The optimized direction vector is obtained by The inverse of the noise covariance matrix is ​​obtained, which is the main eigenvector of the direction vector.

[0064] The purpose and principle of calculation of formula 1.5 are as follows: The direction vector is derived from the main eigenvector found: Formula 1.5 shows that the estimated beam direction vector is obtained by multiplying the optimized direction vector with the noise covariance matrix. The role of the noise covariance matrix is ​​to adjust the direction vector so that it can accurately reflect the actual direction of the signal. Through this correction, the final beam direction vector is an optimized estimate, which can more accurately align the direction of the target signal. This process effectively improves the accuracy and directivity of signal reception, especially in an environment with noise and interference, it can significantly improve the quality and reliability of the signal.

[0065] The calculation result of formula 1.5 is calculated based on the antenna information of each observation point and the direction vector of each pointing, and the direction vector is calculated according to the maximum signal-to-noise ratio rule. .

[0066] S2.4 After the direction vector estimation is completed, the beamforming weight is calculated based on the maximum signal-to-noise ratio (SNR) criterion. The weight formula is: In formula 1.6, The inverse matrix of the noise covariance matrix. The inverse of the covariance matrix is ​​used to eliminate the interference of noise on the signal direction, so that the beamforming process can enhance the target signal and suppress the noise as much as possible; is the direction vector indicating the arrival direction of the target signal It is usually related to the geometric structure of the array antenna and is used to describe the directional characteristics of the signal; is the beamforming weight, which represents the weight coefficient of each array channel in the beamforming process. The beamforming weight adjusts the phase and amplitude of the signal to ensure that the signal is maximized along a specific direction (i.e., the direction of the target signal).

[0067] The purpose and principle of the calculation of formula 1.6 are as follows: It describes how to calculate the beamforming weights according to the maximum signal-to-noise ratio (SNR) criterion. The beamforming weights are obtained by multiplying the inverse matrix of the noise covariance matrix with the direction vector. The core goal of this process is to concentrate the gain of the beam in the direction of the target signal while suppressing the influence of noise as much as possible. Under the maximum signal-to-noise ratio criterion, the beamforming weights maximize the signal-to-noise ratio of the received signal in the target direction. The inverse operation of the noise covariance matrix can effectively eliminate the interference of noise on the beam direction and improve the accuracy of beamforming. Through the calculated beamforming weights, the beam can be optimized according to the direction of the target signal to ensure the best signal gain. Through the calculated beamforming weights, the beam can be effectively focused in the direction of the target signal and suppress signals from noise and interference. This method provides higher signal gain and directional accuracy in a noisy environment, improving the performance of the system. The maximum SNR criterion ensures that the receiving gain of the target signal is optimized during the beamforming process, thereby improving the detection capability and signal quality of the system.

[0068] The result of formula 1.6 is the beamforming weight .

[0069] S2.5 uses the following formula to calculate the beam after beam synthesis based on the weight matrix obtained by the pre-beam calibration and the information of each beam in the array.

[0070] In formula 1.7, After beamforming The first beam The output signal of the direction. It is obtained by linearly combining the beamforming weights and the received signal; For the The first beam The conjugate transpose of the weight vector of the direction. This weight vector has been pre-beam calibrated and optimized, representing the amplitude and phase weights of each channel in the array, and is used to adjust the signal reception direction; For the The signal vector received by a beam contains the signals received by all antenna elements in the array.

[0071] The purpose and principle of calculation of formula 1.7 are as follows: The core idea of ​​beamforming is to perform weighted summation of the received signal vector through the weight vector to enhance the signal in the target direction while suppressing noise and interference in other directions. Through the conjugate transpose of the weight vector, it is multiplied by the received signal vector to obtain a scalar value, which is the synthesized beam output signal. The target adjusts the weight vector so that the directivity of the beam is concentrated in the target direction, thereby maximizing the signal gain while reducing the impact of noise and interference. The application of formula 1.7 can effectively enhance the signal in the target direction while suppressing noise and interference from other directions, thereby improving the signal reception performance of the system. The signal after beamforming is more concentrated in the target signal direction, which can provide higher quality input for subsequent data processing and analysis.

[0072] The result of formula 1.7 is the beam vector information after beam synthesis. .

[0073] S2.6 integrates and enhances the signal in the direction of the target signal source according to the synthesized beam, and transmits the result to the server memory.

[0074] Step S3, in the data writing module, during the data writing stage, the data storage client (dada_dbdisk) of PSRDADA is used. The core function of the data writing module is to write data blocks to disk and divide the data into files of arbitrary length. The beginning of each file contains a file header from the data block. After each file is written to disk, a record is added to the ASCII text log file; each record contains the full path of the file, write time, file size, time required to write the file, and observation identifier. After the server memory receives the data transmitted from the GPU ring buffer, the data storage client is used to write the data to the disk for storage. If you need to access other server computing nodes or storage nodes later, you can send it to the switch through the network card. The overall disk writing process is as follows: Figure 5As shown in the figure. The process is that when the CPU ring buffer receives the processing result data from the GPU, it uses the PSRDADA data storage client to write the data block to the disk and split the data into files of any length. The beginning of each file will contain the file header from the data block. If it is necessary to connect to other server computing nodes or storage nodes later, it can be sent to the switch through the network card.

[0075] The performance evaluation of the overall real-time data stream processing system mainly depends on three core indicators: data transmission rate, data processing rate and data write rate. These indicators directly affect the real-time performance and processing capacity of the system. Therefore, the present invention conducts in-depth testing and analysis from these aspects. Specifically, the study will evaluate the transmission and processing efficiency of data at each stage of the system to ensure that the system can still maintain stable operation under high input and output load environments. At the same time, the efficiency of writing data to the storage medium will also be examined to verify the reliability of the system under long-term operation. Through these tests, the feasibility analysis of the entire system can be supported.

[0076] The test environment parameters of the embodiment of the present invention are listed as follows: Table 1 Test environment parameters

[0077] The test method includes the following steps: 1. First, create two 9GB ring buffers in the server memory, one for receiving UDP packet data and the other for receiving GPU data after processing. Each ring buffer contains a default 4096-byte file header and is divided into 8 data blocks of 1.05GB each.

[0078] 2. Then, create a 9GB ring buffer in the GPU memory, which includes the default 4096-byte file header and 8 data blocks of 1.05GB each.

[0079] 3. Next, use the PSRDADA network simulation sending tool (dada_junkdb) to simulate the UDP packet transmission process. Set the PSRDADA network simulation sending tool to use zero copy technology, transmit 5000MB of data per second, last for 20 seconds, and the transmission content is Gaussian noise data.

[0080] 4. Start the written transfer_cputogpu script to test the data transfer rate from the CPU ring buffer to the GPU ring buffer.

[0081] 5. Start the written transfer_gputocpu script to test the data transfer rate from the GPU ring buffer to the CPU ring buffer.

[0082] 6. Finally, use the data storage client tool to write the processed data transferred from the GPU to the CPU to the hard disk.

[0083] The test results are as follows:

[0084] 1. Test the data transfer rate from the CPU ring buffer to the GPU ring buffer. It takes 22 seconds to transfer 100.7GB of data, and the data transfer rate is about 4.2GB / s.

[0085] 2. Test the data transfer rate from GPU ring buffer to CPU ring buffer. It takes 21 seconds to transfer 101.7GB of data, and the data transfer rate is about 4.6GB / s.

[0086] 3. Test the CPU ring buffer write disk data transfer rate. It takes 73 seconds to write 101.7GB of data to the disk, and the data transfer rate is about 1.3GB / s.

[0087] Based on the current demand for high sensitivity and high resolution in radio astronomy observations, and the urgent need for real-time processing of high-bandwidth radio signals observed by phased arrays collected by digital terminals, the present invention uses distributed acquisition components to construct a real-time data stream processing framework for radio astronomy data. It aims to solve the problem of data quality degradation caused by data packet loss in a high-input and output network environment, thereby achieving high-speed processing of real-time transmitted radio astronomy data streams. In addition, the real-time data stream processing framework is highly modularized by abstracting data blocks and processing processes using distributed acquisition components, making each processing stage more flexible. This approach not only optimizes the development process, but also enables the real-time data stream to flow efficiently in each data block, thereby improving the overall performance and scalability of the processing framework.

[0088] The present invention is based on the constructed real-time streaming data processing framework, integrates and optimizes the beamforming algorithm, and completes the real-time streaming data processing system. The phased array samples the electromagnetic field of the focal plane of the reflector through a large-scale feed source array, and uses the terminal beamforming network to perform weighted superposition on the sampled data, and finally generates a beam distribution covering a continuous sky area. Therefore, it is necessary to integrate the beamforming algorithm to achieve the required spatial response pattern; at the same time, the correlation matrix operation of the array element signal is performed to calibrate the beam and generate the optimal weight factor.

[0089] The present invention conducts a comprehensive performance test on the implemented real-time stream data processing system to verify its real-time performance and availability in practical applications. Through performance testing, the stability and data processing efficiency of the system under high input and output load environments can be evaluated, thereby ensuring that the system can meet the needs of radio astronomy observations.

[0090] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0091] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0092] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. A method for processing real-time data streams of radio astronomy based on distributed acquisition components, characterized in that: The method comprises the following steps: S1. Data transmission: The server receives UDP data packets from the radio telescope through the network card, uses the open source pulsar distributed acquisition component and data analysis system PSRDADA to create and manage ring buffers, and transmit data streams between each ring buffer; the UDP data packet contains the digital signal and observation information encapsulated by the radio telescope after the radio telescope performs fast Fourier transform processing through FPGA; S2. Data processing: using the CUDA parallel computing platform and programming model of the graphics processing unit GPU to write computing and unpacking programs, thereby realizing parallel processing and data extraction of data; S3. Data writing to disk, using PSRDADA's data storage client, writes data blocks to disk and splits the data into files of any length.

2. The method for processing radio astronomy real-time data stream based on distributed acquisition components according to claim 1, characterized in that: In step S1, the distributed acquisition component PSRDADA creates multiple ring buffers, each of which is allocated as a shared memory resource and is logically divided into a file header describing the observation information and multiple data blocks storing data.

3. The method for processing radio astronomy real-time data stream based on distributed acquisition components according to claim 1 or 2, characterized in that: In step S1, the data stream transmission process includes: S1.1 At the beginning of the observation, after the server receives the UDP data packet from the front-end radio telescope through the network card, it first uses the data flow control software of PSRDADA to create a circular buffer in the server memory, and uses direct memory access DMA to read the network card data; each data block in the buffer is initially empty, and its file header is initialized to information related to the observation; the information includes bandwidth, center frequency, signal source, and start time; S1.2 When the ring buffer in the server memory receives the data, the file header of the buffer is parsed to obtain the observation information; then, a ring buffer is created on the GPU using the data flow control software of PSRDADA, and the transfer script transfer_cputogpu from the CPU memory to the GPU video memory is started to transfer the data in the server memory to the ring buffer of the GPU; After the S1.3 GPU completes data processing, it first creates a ring buffer in the server memory using the data flow control software of PSRDADA, and then transfers the result data to the server memory using transfer_gputocpu; S1.4 During the entire observation process, data is written into data blocks in sequence; when the observation ends, the data end flag is set, and the last complete data block and the number of valid bytes written are recorded; after that, data is read from the data blocks in sequence until the data end flag is encountered and the current data block is the last complete data block.

4. The method for processing radio astronomy real-time data stream based on distributed acquisition components according to claim 1, characterized in that: In step S2, the GPU is used to process parallel tasks, and the CPU is used to process delay-sensitive tasks.

5. The method for processing real-time radio astronomy data stream based on distributed acquisition components according to claim 4, characterized in that: Step S2 includes the following steps: S2.1 GPU reads the data in the ring buffer in the video memory and parses the observation data based on the metadata in the file header; the parsing process includes extracting the observation timestamp and UDP data size; locating the data block address corresponding to the spectrum according to the unpacked metadata; S2.2 Based on the analyzed data, the analyzed spectrum information will be pre-calibrated for beams; the calibration process calculates the covariance matrix for each observation point; S2.3 Estimate the direction vectors of each beam pointing based on the calculated covariance matrix : S2.4 after the direction vector estimation is completed, the beamforming weight is calculated based on the maximum signal-to-noise ratio criterion; S2.5 calculating the beam after beam synthesis according to the weight matrix obtained by the pre-beam calibration and combining the information of each beam in the array; S2.6 integrates and enhances the signal in the direction of the target signal source according to the synthesized beam, and transmits the result to the server memory.

6. The method for processing real-time radio astronomy data stream based on distributed acquisition components according to claim 5, characterized in that: In step S2.2, the covariance matrix formula is as follows: In formula 1.1, represents the nth signal vector, represents the conjugate transpose of the nth signal vector; It represents the expected value of multiple signal vectors in the time domain; represents the "on" covariance matrix; In formula 1.2 represents the nth noise vector, represents the conjugate transpose of the nth noise vector; Represents the expected value of multiple noise vectors in the time domain; Represents the "off" covariance matrix.

7. The method for processing real-time radio astronomy data stream based on distributed acquisition components according to claim 1, characterized in that: Step S3, data writing stage, the beginning of each file contains a file header from the data block; after each file is written to the disk, a record is added to the ASCII text log file.

8. The method for processing radio astronomy real-time data stream based on distributed acquisition components according to claim 7, characterized in that: Each record contains the full path to the file, the time it was written, the file size, the time it took to write the file, and an observation identifier.

9. The method for processing real-time radio astronomy data stream based on distributed acquisition components according to claim 8, characterized in that: When the data writing module needs to access other server computing nodes or storage nodes, it is sent to the switch through the network card.

10. A radio astronomy real-time data stream processing system based on distributed acquisition components, characterized in that: The system is used to implement the radio astronomy real-time data stream processing method based on distributed acquisition components according to any one of claims 1 to 9, and the system includes a data transmission module, a data processing module and a data disk writing module.

Images