Method for extracting and storing electromagnetic interference data of radio astronomy

Abstract

The invention provides a radio astronomy electromagnetic interference data extraction and storage method. The method comprises the steps of obtaining original baseband data and storing the original baseband data in a PSRFITS format; defining a storage data structure; according to the original baseband data, the basic observation attribute and the file path of the PSRFITS format file, extracting to obtain an original electromagnetic interference data matrix and attribute data; the method comprises the following steps of: extracting and grouping an original electromagnetic interference data matrix to obtain a denoised data matrix with a frequency index and a time index and an attribute data key value pair set, wherein the denoised data matrix is divided into polarization and interference service frequency bands; enabling each HDF5 file to correspond to a combination of a polarization mode POL and an interference service frequency band; and storing the logic group matrix, the time index vector and the attribute data key value pair set in an HDF5 file. According to the method disclosed by the invention, the conversion from original baseband data to standardized storage is realized, and the access speed, the analysis flexibility and the long-term maintainability are considered while the data volume is obviously reduced.

Description

Technical Field

[0001] This invention relates to the field of radio astronomy observation technology, and in particular to a method for extracting and storing radio astronomy electromagnetic interference data. It is designed for multi-source heterogeneous radio astronomy electromagnetic compatibility data and is used to convert, group, and filter noise from observation data in the original PSRFITS format to structured HDF5 data. Background Technology

[0002] In the field of radio astronomy, radio frequency interference (RFI) refers to unwanted signals received by a radio telescope other than the signal from the observed target. Because the signals received by radio telescopes are extremely weak, they are highly susceptible to radio frequency interference (RFI) from ground-based radio transmitting equipment (such as base stations, radar, and satellites). [1] To ensure the scientific validity of astronomical data, rigorous electromagnetic compatibility (EMC) analysis must be performed on the observational data to identify, quantify, and eliminate these human interferences. [2] Therefore, efficient and accurate electromagnetic compatibility data processing has become an indispensable key component in the field of radio astronomy.

[0003] Currently, radio astronomy electromagnetic compatibility (EMIC) data processing faces three major technical bottlenecks. First, the raw FITS data contains four-dimensional baseband data (sub-integral × sampling point × polarization × channel) and coordinate parameters (AZ / EL) scattered across different levels, resulting in significant data format heterogeneity. This makes effective integration of data from different devices and observation periods difficult. Second, in terms of noise interference processing, traditional fixed threshold methods are ill-suited to the dynamic characteristics of different frequencies. Research indicates that sliding window standard deviation analysis (such as the `find_clean_segment` function) is necessary to establish an accurate noise baseline. [3] Finally, in terms of analysis efficiency, the ungrouped full-band scanning method severely restricts processing speed; using a predefined frequency band grouping model (e.g., the 935-948MHz mobile communication band) can significantly improve analysis efficiency.

[0004] References:

[0005] [1] Wang Yue, Liu Qi, Liu Ye, Su Xiaoming. Software development and implementation of automated electromagnetic environment monitoring system. Astronomical Research and Technology, 2020.

[0006] [2] Liu Qi, Chen Maozheng, Li Ying, et al. Electromagnetic radiation assessment of electronic equipment at radio astronomy sites [J]. Astronomical Research and Technology, 2015, 12(3):292-298.

[0007] [3] Liu Qi, Feng Dongdong, Cai Minghui, et al. A signal-to-noise separation method based on neighbor comparison [J]. Science Technology and Engineering, 2021, 21(09):3656-3661. Summary of the Invention

[0008] The purpose of this invention is to provide a method for extracting and storing radio astronomy electromagnetic interference data, so as to realize the conversion process from raw baseband data to standardized storage, which significantly reduces the data volume while taking into account access speed, analysis flexibility and long-term maintainability.

[0009] To achieve the above objectives, the present invention provides a method for extracting and storing radio astronomy electromagnetic interference data, comprising:

[0010] S1: Acquire raw baseband data and store it in PSRFITS format;

[0011] S2: Define the storage data structure as including the original electromagnetic interference data matrix and the attribute data key-value pair set, where the attribute data key-value pair set includes multiple attribute data; for each PSRFITS format file, extract the original electromagnetic interference data matrix and attribute data based on the original baseband data, basic observation attributes, and file path of the PSRFITS format file.

[0012] S3: Further extraction and grouping of the original electromagnetic interference data matrix yields a denoised data matrix with polarization and interference service frequency bands, and with frequency and time indices, as well as a corresponding set of attribute data key-value pairs;

[0013] S4: Define each HDF5 file as a combination of a polarization mode POL and an interfering service frequency band; store the denoised and filtered logical group matrix and time index vector under multiple spatial coordinates as multiple core datasets, together with the corresponding attribute data key-value pair set in the corresponding HDF5 file.

[0014] The raw baseband data is obtained by observations at different azimuth and elevation angles using a radio telescope. When stored in PSRFITS format, each spatial coordinate corresponds to a folder, and each folder stores only one PSRFITS format file. Each PSRFITS format file includes the raw baseband data and basic observation attributes under a spatial coordinate, which is a combination of azimuth and elevation angles.

[0015] The definition of the storage data structure is as follows:

[0016] ,

[0017] Where P is the storage data structure, For attribute data key-value pairs, This represents the 42nd attribute data; The original electromagnetic interference data matrix is ​​extracted from the original baseband data. It is four-dimensional and the matrix size is the number of sub-integrals Ns × the number of sampling points per sub-integral Nt × the number of polarization modes of the original baseband data channel × the number of frequency channels n.

[0018] Extracting the original electromagnetic interference data matrix from the original baseband data refers to integrating heterogeneous data scattered in the PSRFITS format file to obtain an original electromagnetic interference data matrix with a size of Ns (number of sub-integrations) × Nt (number of sampling points per sub-integration) × number of polarization modes of the original baseband data (channel) × number of frequency channels (n).

[0019] In the attribute data key-value pair set, the attribute data is obtained from the header of the PSRFITS file or from the file path.

[0020] Step S3 specifically includes:

[0021] S31: Separate the original electromagnetic interference data matrix D according to the dimension of polarization to obtain the horizontally polarized electromagnetic interference data matrix D. H and vertically polarized electromagnetic interference data matrix D V ;

[0022] S32: Data matrix D for horizontally polarized electromagnetic interference H and vertically polarized electromagnetic interference data matrix D V Adding frequency and time indices yields a horizontally polarized electromagnetic interference data matrix with both frequency and time indices. and vertical polarization electromagnetic interference data matrix ;

[0023] S33: Analyze the horizontally polarized electromagnetic interference data matrix with frequency and time indices according to multiple predefined interference service frequency bands (freq_ranges). and vertical polarization electromagnetic interference data matrix Each is divided into logical group matrices for different interference service frequency bands. The logical group matrices correspond one-to-one with the interference service frequency bands, and the attribute data key-value pair set of each logical group matrix is ​​obtained.

[0024] S34: For each logical group matrix Perform noise reduction filtering; then process the noise-reduced logic group matrix. It compresses its time index vector.

[0025] In step S32, when calculating the frequency index, the frequency index is determined by using the start frequency START, end frequency STOP, and bandwidth CHAN_BW of each frequency channel in the basic observation attributes of the PSRFITS format file as the start point, end point, and step bandwidth of the frequency index; when calculating the time index, the time index starts from 1, the time index of each line is incremented by 1, and the time index of the last line is 0.

[0026] In step S33, the set of interfering service frequency bands is as follows: , For the i-th interfering service frequency band, each interfering service frequency band It has a corresponding frequency space , For the i-th interfering service frequency band Start and end frequency indices, band selection function Frequency index representing the q-th frequency channel Is it in the i-th interfering service frequency band? If so, then the frequency band selection function True;

[0027] For an arbitrary polarization electromagnetic interference data matrix with frequency and time indices, the i-th interfering service frequency band Logical group matrix for:

[0028] ,

[0029] Where M is an arbitrary polarization electromagnetic interference data matrix with frequency and time indices. , The frequency band selection function represents the electromagnetic interference data matrix M. The submatrix formed by the columns containing the true frequency indices. , is a vector of time indices. For the first Time index of sampling points;

[0030] Meanwhile, according to the set of interference service frequency bands Various interference frequency bands The frequency range, for each interfering service frequency band. The frequency-related attribute data is corrected to obtain the frequency bands of each interfering service. Logical group matrix The set of attribute data key-value pairs.

[0031] The logic group matrix after noise filtering The compression of the logic group matrix and its time index vector includes compressing the denoised and filtered logic group matrix and its time index vector into uint8 and uint32 types, respectively.

[0032] Each HDF5 file corresponds to a combination of polarization mode and interfering service frequency band, and the file name of the HDF5 file is the name of the interfering service frequency band.

[0033] The HDF5 file includes multiple core datasets. Each denoised and filtered logical group matrix and each time index vector are data items of a core dataset. The name of each core dataset includes spatial coordinates and timestamps, and the attribute data key-value pair set is stored in the attribute item of each core dataset in the HDF5 file.

[0034] This invention embeds multiple core attributes directly into the file as HDF5 attributes, optimizing the storage structure and making the data completely self-describing, allowing for traceability of its origin and meaning without external documentation. Furthermore, it treats signal data and time indexes as independent datasets and enables compression for each, thereby significantly compressing the data through optimized storage structure and noise filtering algorithms. In addition, this invention splits the data according to interfering service frequency bands and polarization methods, storing them in independent HDF5 files. This avoids excessively large single files, supports on-demand loading, lays the foundation for parallel processing, and improves interference analysis efficiency.

[0035] Furthermore, this invention uses HDF5 files, supports a block storage strategy, optimizes the read and write efficiency of large datasets, and supports incremental write mode, which facilitates batch processing of data while ensuring processing stability and data security. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method for extracting and storing radio astronomical electromagnetic interference data according to the present invention.

[0037] Figure 2 This is a schematic diagram of the storage structure of the PSRFITS format files and their folders according to the present invention.

[0038] Figure 3 This is a schematic diagram of the storage structure of HDF5 files and their folders according to the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0040] The method for extracting and storing radio astronomy electromagnetic interference data of the present invention includes data measurement, structure definition, extraction and classification, HDF5 format design and storage optimization.

[0041] like Figure 1 As shown, the method for extracting and storing radio astronomy electromagnetic interference data of the present invention includes the following steps:

[0042] Step S1: Acquire raw baseband data and store it in PSRFITS format;

[0043] The raw baseband data is obtained by observations made by radio telescopes at different azimuth (AZ) and elevation (EL) positions. Therefore, the raw baseband data includes a set of spectral data corresponding to each combination of azimuth and elevation (i.e., each observation position).

[0044] During storage, the file directory structure is organized according to spatial coordinates.

[0045] In this embodiment, as Figure 2 As shown, when stored in PSRFITS format, each spatial coordinate corresponds to a folder, and each folder stores only one PSRFITS file. Each PSRFITS file includes the raw baseband data and basic observation attributes under a spatial coordinate, which is a combination of azimuth and elevation angles. Therefore, this file directory structure means that the data file path itself contains spatial coordinate information, facilitating subsequent automated processing.

[0046] like Figure 2 As shown, the spectrum data is the raw baseband data; basic observation attributes, such as test time, start frequency, and end frequency, are stored in the header file of the PSRFITS format file. Figure 2 In the raw baseband data, N_subints is the number of sub-integrations, N_samples is the number of sampling points per sub-integration, channel0 is the horizontal polarization, channel 1 is the vertical polarization, and f n For frequency index, frequency index f n The maximum value of the subscript n is the number of frequency channels. N_subints, N_samples, and n are variables describing the different dimensions of the original baseband data, used to indicate what size the original baseband data is.

[0047] Step S2: Define the storage data structure as including the original electromagnetic interference data matrix and the attribute data key-value pair set, the attribute data key-value pair set including multiple attribute data; for each PSRFITS format file, extract the original electromagnetic interference data matrix D and attribute data according to the original baseband data, basic observation attributes and file path of the PSRFITS format file.

[0048] As mentioned above, each PSRFITS format file includes not only spectral data with a combination of azimuth and elevation angles, but also basic observation attributes located in the header file. To facilitate retrieval and retrieval during later processing, the data structure needs to be redefined so that each data set has a corresponding attribute dataset, forming a unified "spectral data + attribute data key-value pair set" structure. Therefore, the attribute data in the attribute data key-value pair set makes the data file highly self-descriptive, allowing users to understand the data source, processing procedure, and physical meaning without additional documentation, conforming to best practices in scientific data management.

[0049] To achieve self-descriptive and standardized data, the storage data structure P is defined as follows:

[0050] (1)

[0051] Where P is the storage data structure, For attribute data key-value pairs, This represents the 42nd attribute data; The original electromagnetic interference data matrix extracted from the original baseband data is four-dimensional, and its size is Ns (number of sub-integrals) × Nt (number of sampling points per sub-integral) × number of polarization modes (channel) of the original baseband data × number of frequency channels (n). The number of polarization modes (channel) of the original baseband data has a value of 4, which includes two polarizations and the real and imaginary parts after coherent processing of the two polarizations. Channel 0 is horizontal polarization, and channel 1 is vertical polarization.

[0052] Extracting the original electromagnetic interference data matrix from the original baseband data refers to integrating heterogeneous data scattered in the PSRFITS format file to obtain an original electromagnetic interference data matrix with a size of Ns (number of sub-integrations) × Nt (number of sampling points per sub-integration) × number of polarization modes of the original baseband data (channel) × number of frequency channels (n).

[0053] In this embodiment, the attribute data key-value pair set contains 42 attribute data items. All attribute data are shown in Table 1. Most attribute data in the attribute data key-value pair set is obtained from the header file of the PSRFITS file, and a small amount of attribute data is obtained from the file path. The attribute data key-value pair set is ultimately stored in the attribute items of each core dataset in the HDF5 file described below.

[0054] Table 1: Details of Attribute Data

[0055] Name of attribute data Property Definition SYS_M System Model SYS_N System Name POL Polarization mode (horizontal polarization H, vertical polarization V, linear, circular polarization...) AZ [degrees] Azimuth (0-360) EL [degrees] Pitch angle (0-180) ANT_H [meters] Antenna height TEST_D [Meters] Test distance DATE Test date (e.g., 20200101) TIME Test time (e.g., 00:00:00) REPEAT_T Number of repetitions (e.g., 1 / 2 / 3…) START [MHz] Starting frequency STOP [MHz] Cut-off frequency CF [MHz] Center frequency BW [MHz] Total bandwidth TBIN [Seconds] Width of each time slot CHAN_BW [MHz] Bandwidth per frequency channel NCHAN Number of frequency channels n in the file SA_RBW [kHz] Resolution bandwidth SA_VBW [KHz] Video bandwidth SA_SWT [Milliseconds] Scan time SA_SWP The number of scan points, i.e., the number of sampling points per sub-integral, N_samples SA_SWC Scan count, i.e., number of sub-integrals N_subints SA_DETECTOR Trajectory detector (SAMP, RMS, POS, NEG, AVG, etc.) SA_TRAC_M Track patterns (Average AVG, Maximum Hold MAXH, Minimum Hold MINH...) SA_ATT [dB] RF attenuation SA_REF [dBm] Reference Level SA_AMP Preamplifier (ON, OFF) CAL Calibration (ON, OFF) DATA_MODE Data pattern (binary, decimal, string) DATA_TYPE Data types (see table below) UNIT Data units (dBm, dBm / Hz, dBμV / m, dBW / m² / Hz) LON_E [degrees] East longitude LAT_N [degrees] North Latitude ALTITUDE [meters] Altitude TEMP [degrees] temperature HUMI [%] Humidity ATM [Pa] Atmospheric pressure EUT_NAME Telescope Name EUT_TYPE Telescope type EUT_CODE Telescope code

[0056] Step S3: Further extract and group the original electromagnetic interference data matrix D to obtain a denoised data matrix with polarization and interference service frequency bands, and with frequency and time indices, as well as the corresponding attribute data key-value pair set;

[0057] Thus, useful information is extracted from the raw baseband data and logically grouped to improve processing efficiency.

[0058] Step S3 specifically includes:

[0059] Step S31: Separate the original electromagnetic interference data matrix D according to the dimension of polarization to obtain the horizontally polarized electromagnetic interference data matrix D. H and vertically polarized electromagnetic interference data matrix D V :

[0060] (2)

[0061] (3)

[0062] in, The meaning is to separate the data set with a polarization dimension value of 0 (i.e., horizontal polarization data) from the original electromagnetic interference data matrix D. That is, to slice the four-dimensional original electromagnetic interference data matrix D, take all elements (:) in the dimension of the number of sub-integrals, take all elements (:) in the dimension of the number of sampling points per sub-integral, take the element with index 0 (0) in the dimension of polarization, and take all elements in the dimension of the number of frequency channels. The meaning is to separate the data set with a polarization dimension value of 1 (i.e., vertical polarization data) from the original electromagnetic interference data matrix D.

[0063] Step S32: For the horizontally polarized electromagnetic interference data matrix D... H and vertically polarized electromagnetic interference data matrix D V Adding frequency and time indices yields a horizontally polarized electromagnetic interference data matrix with both frequency and time indices. and vertical polarization electromagnetic interference data matrix ;

[0064] To provide the infrastructure for the subsequent recovery and restoration of the stored data after noise reduction and compression, the two types of polarized electromagnetic interference data matrices were constructed into composite matrices containing the original data, frequency index, and time index.

[0065] When calculating the frequency index, the frequency index is determined by using the start frequency (START), end frequency (STOP), and bandwidth (CHAN_BW) of each frequency channel in the basic observation attributes of the PSRFITS format file as the start point, end point, and step bandwidth of the frequency index. That is, the frequency index of the first column is START, the frequency index of the second column is START + CHAN_BW, ..., the frequency index of the nth column is START + n * CHAN_BW, ..., and the frequency index of the last column is STOP, consistent with the actual frequency. When calculating the time index, the time index starts from 1, i.e., the first second. The time index of each row is incremented by 1, representing an increase of 1 second (i.e., the time interval between adjacent sampling points), and recorded sequentially. The time index of the last row (i.e., the frequency index row) is 0. In actual application, when processing data in the HDF5 file, simply add the test date (DATE) and test time (TIME) in the basic observation attributes of the PSRFITS format file to the time index to obtain the actual time.

[0066] Horizontal polarized electromagnetic interference data matrix with frequency and time indexes for:

[0067] (4)

[0068] (5)

[0069] (6)

[0070] Vertically polarized electromagnetic interference data matrix with frequency and time indexes for:

[0071] (7)

[0072] (8)

[0073] (9)

[0074] Where, N s N is the number of sub-integrals. t This represents the number of sampling points per sub-integral, where n is the number of frequency channels. This is the data from the first sampling point and the first frequency channel of the horizontally polarized electromagnetic interference data matrix. This is the data from the first frequency channel of the second sampling point in the horizontally polarized electromagnetic interference data matrix. It is the first in the horizontally polarized electromagnetic interference data matrix. Data from the nth frequency channel at the sampling point. For the integrated horizontal polarization electromagnetic interference data matrix, This is a horizontally polarized electromagnetic interference data matrix with frequency index. A horizontally polarized electromagnetic interference data matrix with frequency and time indices; This is the data from the first sampling point and the first frequency channel of the vertically polarized electromagnetic interference data matrix. This is the data from the first frequency channel of the second sampling point in the vertically polarized electromagnetic interference data matrix. It is the first vertically polarized electromagnetic interference data matrix. Data from the nth frequency channel at the sampling point. This is the integrated vertical polarization electromagnetic interference data matrix. This is a vertically polarized electromagnetic interference data matrix with frequency index. This is a vertically polarized electromagnetic interference data matrix with frequency and time indices. , is a vector of frequency indices. This is the frequency index of the nth frequency channel; , is a vector of time indices. For the first Time index of sampling points.

[0075] Step S33: According to the predefined multiple interference service frequency bands freq_ranges, generate the horizontal polarization electromagnetic interference data matrix with frequency index and time index. and vertical polarization electromagnetic interference data matrix Each is divided into logical group matrices for different interference service frequency bands. The logical group matrices correspond one-to-one with the interference service frequency bands, and the attribute data key-value pair set of each logical group matrix is ​​obtained.

[0076] Interference service frequency bands refer to the operating frequency bands of radio services that act as sources of interference. A horizontally polarized electromagnetic interference data matrix with frequency and time indices. and vertical polarization electromagnetic interference data matrix It covers the entire frequency band. Electromagnetic interference (EMI) data typically covers a wide frequency band, but analysis often focuses only on the operating frequency bands of specific radio services (such as mobile communications and satellite bands). The radio signals operating in these bands are themselves interference signals for radio astronomy. The frequency bands where these nationally designated radio services operate are the interference frequency bands. Therefore, the purpose of grouping the interference frequency bands and dividing the full-band data into multiple logical groups is to allow for analysis of specific frequency bands when needed for future analysis of interference sources, thus improving analysis efficiency.

[0077] Logical groups can be, for example, CM-2G-DL, CU-4G-UL, etc. The specific meanings of each interference service frequency band and its logical group are explained in Table 2. The predefined interference service frequency bands freq_ranges are preset according to the radio service zones divided by the country. Currently, in the L-band, the service range of 900MHz-1.9GHz has been defined.

[0078] Table 2: Frequency Bands and Logical Groups of Various Interference Services

[0079] Interference service frequency bands (MHz) business Abbreviated name (English) 935-948 China Mobile / Downlink 2G CM-2G-DL 949-959 China Unicom / Downlink 2G CU-2G-DL 1025-1150 Aviation radio navigation (DME / ADS-B) A-RNAV 467-1492 Satellite broadcasting (digital broadcasting satellite uplink band service) DBS 1530-1550 Satellite phone: Inmarsat, Thuraya / downlink signal SATCOM 1560-1563 Satellite Navigation: BeiDou B1 BDS B1 or BeiDou GNSS 1574-1578 Satellite navigation: GPS L1 downlink, Galileo E1, QZSS L1 GNSS 1589-1593 Aviation mobile communications AMC 1618-1627 Satellite phones: Inmarsat, GlobalStar, Iridium SATCOM 1710-1725 China Mobile / Uplink 2G CM-2G-UL 1725-1735 China Mobile / Uplink 4G CM-4G-UL 1735-1755 China Unicom / Uplink 2G CU-2G-UL 1755-1765 China Unicom / Uplink 4G CU-4G-UL

[0080] In step S33, the set of interfering service frequency bands is as follows: , For the i-th interfering service frequency band, each interfering service frequency band It has a corresponding frequency space , For the i-th interfering service frequency band Start and end frequency indices, band selection function Frequency index representing the q-th frequency channel Is it in the i-th interfering service frequency band? If so, then the frequency band selection function True.

[0081] That is, frequency band selection function for:

[0082] (10)

[0083] For arbitrary polarization electromagnetic interference data matrices with frequency and time indices The i-th interfering service frequency band Logical group matrix for:

[0084] (11)

[0085] Where M is an arbitrary polarization electromagnetic interference data matrix with frequency and time indices. , The frequency band selection function represents the electromagnetic interference data matrix M. The submatrix formed by the columns containing the true frequency indices. The second sampling point of the horizontally polarized electromagnetic interference data matrix, with frequency index . Data, For the i-th interfering service frequency band The start frequency index and the end frequency index, i start i stopThese are the frequency channel ordinal numbers corresponding to the starting frequency index and the ending frequency index, respectively. , is a vector of time indices. For the first Time index of sampling points.

[0086] The expansion of formula (11) is illustrated here using the vertically polarized electromagnetic interference data matrix as an example. For the horizontally polarized formula, simply change the subscript in the formula from V to H.

[0087] Meanwhile, according to the set of interference service frequency bands Various interference frequency bands The frequency range, for each interfering service frequency band. The frequency-related attribute data is corrected to obtain the frequency bands of each interfering service. Logical group matrix attribute data key-value pair set .

[0088] In the collection of attribute data key-value pairs In the middle, the logical group matrix of different interference service frequency bands Corresponding attribute data key-value pair set These are different sets of attribute data key-value pairs, each containing 42 attribute data items. The logical group matrix is ​​for different interference service frequency bands. Corresponding attribute data key-value pair set These attribute data key-value pair sets The only difference lies in four attribute data: start frequency (START), stop frequency (STOP), center frequency (CF), and total bandwidth (BW), which are essentially different frequency ranges. All other attributes are the same as those in the logical group matrix. The attribute data of the corresponding original electromagnetic interference data matrix D are consistent.

[0089] The purpose of step S33 is to: 1. Store data of different interfering service frequency bands in different files to avoid excessively large single files and improve read and write efficiency; 2. Allow users to directly access the interfering service frequency bands of interest without loading all data; 3. Enable spectrum analysis and interference detection to be performed on specific interfering service frequency bands.

[0090] Step S34: For each logical group matrix Noise filtering is performed to retain the valid signal; then the denoised and filtered logic group matrix is ​​processed. It compresses its time index vector.

[0091] In step S34, the dynamic threshold calculation model based on sliding window standard deviation analysis (based on the statistical method described in reference [3] in the background section) is used to obtain the threshold array. Using formula For each interference service submatrix The data is denoised.

[0092] Among them, the sliding window is used to find high-quality data groups in the overall original data, and a threshold line is generated by algorithms such as window sliding and mean calculation to filter the data. For specific methods, please refer to the literature in the background technology section [3]. Because the threshold array The length of the threshold array is consistent with the frequency range of the entire original data. Therefore, following the method of dividing interference services mentioned above, the threshold array also needs to be divided into a sub-threshold array corresponding to each interference radio service. To remove useless segments from the business logic, that is, to use formulas right A segmentation was performed, which was done according to the corresponding radio interference service frequency range, the same as the previous grouping of interference service frequency bands for horizontal or vertical polarization data. That is, the i-th interfering service frequency band. The corresponding sub-threshold array. When filtering noisy data, does the frequency amplitude value of each row have at least one value greater than the corresponding sub-threshold array? Calculate the corresponding values ​​for the i-th interfering service frequency band. Corresponding logical group matrix If the frequency amplitude value of each row is greater than at least one of the corresponding thresholds, the row is retained; otherwise, it is deleted (i.e., the data corresponding to the time index is deleted).

[0093] That is, the screening matrix for:

[0094] (13)

[0095] in, For filtering matrix The element in the j-th row and k-th column, where j is the row number and k is the column number of the logical group matrix, is used to filter the matrix. =1 indicates that this row should be retained, filtering the matrix. =0 indicates that the row is deleted. It is the threshold of the k-th column in the sub-threshold array. This is the frequency index of the k-th frequency channel.

[0096] For each logical group matrix After noise reduction and filtering, the i-th interfering service frequency band is output. The denoised and filtered logical group matrix and its time index vector.

[0097] Assuming the remaining subscripts after filtering are If the matrix data has j rows in total, then the i-th interfering service frequency band is finally obtained. The denoised and filtered logical group matrix Its time index vector is:

[0098] (14)

[0099] (15)

[0100] in, It refers to the i-th interfering service frequency band. The denoised and filtered logical group matrix; Refers to the logical group matrix By filtering matrix The time index vector portion is removed from the new matrix formed by each row retained after filtering. It refers to the i-th interfering service frequency band. The time index vector of the denoised and filtered logical group matrix; Refers to the logical group matrix By filtering matrix The time index vector in the new matrix formed by each row retained after filtering.

[0101] It should be noted that this example only uses the formula for vertical polarization data, so the subscript is V. For horizontal polarization, simply change the subscript from V to H to get the formula for horizontal polarization.

[0102] The logic group matrix after noise filtering Compression of its time index vector includes: compressing the denoised and filtered logical group matrix. and its time index vector The data is compressed into uint8 and uint32 types respectively to improve compression efficiency. Simultaneously, the compression described below is performed, using the gzip format inherent to HDF5 files, for storage within the HDF5 file described below.

[0103] Therefore, the logical group matrix is ​​denoised and compressed to achieve the purpose of data compression. The time and frequency indices are retained to restore the data (during restoration, the useless data that was originally removed will be replaced with noise).

[0104] Step S4: Define each HDF5 file as a combination corresponding to a polarization mode (POL) and interfering service frequency band; for each combination of polarization mode (POL) and interfering service frequency band, denoise and filter logical group matrices under multiple spatial coordinates. Time index vectors serve as multiple core datasets, along with corresponding attribute data key-value pair sets. They are stored together in the corresponding HDF5 file.

[0105] Therefore, to facilitate storage and access, a standardized data structure for HDF5 files is designed. Each HDF5 file corresponds to a combination of polarization mode POL (horizontal polarization H or vertical polarization V) and interfering service frequency bands, and the file name of the HDF5 file is the name of the interfering service frequency band.

[0106] The HDF5 file includes multiple core datasets. Each denoised and filtered logical group matrix and each time index vector are data items of a core dataset. The name of each core dataset includes spatial coordinates and timestamps, and the attribute data key-value pair set (including polarization mode POL and interference service frequency band) is stored in the attribute item of each core dataset in the HDF5 file.

[0107] The data structure of an HDF5 file is as follows:

[0108] (16)

[0109] in, It is named the i-th interfering service frequency band. HDF5 file; It is the i-th interfering service frequency band. The denoised and filtered logical group matrix; It is the i-th interfering service frequency band. The denoised and filtered logical group matrix The time index vector; It is the i-th interfering service frequency band. The denoised and filtered logical group matrix The corresponding set of attribute data key-value pairs.

[0110] Therefore, the logic group matrix after noise filtering and its time index vector and corresponding attribute data set Store the data in the format corresponding to formula (16) and persist it to the HDF5 file in the DATA FILE folder.

[0111] This process is specifically designed for the massive and complex environment of radio astronomy data, aiming to optimize storage efficiency (through efficient compression and block strategies), access performance (supporting fast queries and partial reads), and long-term data maintainability (ensuring future readability and traceability). This provides a reliable data foundation for subsequent interference source analysis, spectrum management, and astronomical signal research. The stored procedure receives a denoised logical group matrix after polarization separation, frequency band grouping, and noise filtering. .like Figure 3 As shown, for each combination of interfering service frequency bands and polarization methods, the system creates a separate HDF5 file (e.g., SATCOM.hdf5, CM-2G-DL.hdf5) in the output directory. Inside the file, a hierarchical data model is used: the denoised and filtered logical group matrix (uint8 type) and the corresponding time index (uint32 type) are stored as data items in the core dataset (H_). <el> _ <az> _ <date>T represents the core dataset group name. <time>_data represents the suffix name of the core dataset storing the denoised and filtered logical group matrix at a specific point in time, T <time>`_time` represents the suffix name of the core dataset that stores the time index of the denoised and filtered logical group matrix at a specific time point, allowing a single HDF5 file to store multiple core datasets corresponding to different spatial coordinates. Simultaneously, for each core dataset, all 42 attributes are embedded as attributes in the file root group. The data shape corresponding to the logical group matrix (i.e., the final number of remaining data rows M_valid_samples, the number of frequency channels (i.e., the final number of data columns L_freq_points)) and the data shape corresponding to the time index (i.e., the final number of remaining data rows M_valid_samples) are also stored as part of the data item. Data writing uses HDF5's built-in gzip compression and block strategy, and supports incremental mode to avoid overwriting existing data.

[0112] The method for extracting and storing radio astronomy electromagnetic interference data of the present invention has the following characteristics:

[0113] 1) Attribute Embedded Management: 42 core attributes are directly embedded into the file as HDF5 attributes, optimizing the storage structure and making the data completely self-describing, allowing the source and meaning to be traced without external documents.

[0114] 2) Data Separation and Compression: The signal data (converted to uint8) and the time index are treated as independent datasets, and compression is enabled separately for each. By leveraging the different data characteristics of the two, an overall compression rate of up to 40% is achieved.

[0115] 3) Grouping and Physical Isolation: Data is split according to the interfering service frequency band and polarization, and stored in independent HDF5 files. This avoids excessively large single files, supports on-demand loading, lays the foundation for parallel processing, and improves the efficiency of interference analysis.

[0116] 4) Block Incremental Write: It adopts HDF5 files, supports block storage strategy, optimizes the read and write efficiency of large datasets; supports incremental write mode, which facilitates batch processing of data, while ensuring the stability of processing and data security.

[0117] This invention demonstrates significant performance optimization in both experimental and practical applications, primarily in three core aspects. First, by introducing an optimized storage structure and noise filtering algorithm, the invention successfully compressed data volume by 40%. This result, based on actual test data, proves that storage overhead is significantly reduced while maintaining data integrity. Second, in terms of interference analysis efficiency, the software achieves a 5-fold improvement. This is attributed to the application of a predefined frequency band grouping model, which focuses on the target service frequency band (such as the 935-948MHz mobile communication band) by avoiding full-band scanning, thus significantly accelerating the processing flow. Finally, since the preprocessed data is in psrfits file format, which is widely used in astronomical data storage, it is highly compatible with processing other astronomical data. Therefore, the software design possesses high compatibility, seamlessly supporting data input from large international radio telescopes such as FAST and SKA, ensuring standardized processing across different observation devices, thereby improving the universality and scalability of electromagnetic compatibility analysis. These advantages collectively address the storage, efficiency, and interoperability bottlenecks in radio astronomy data processing.

[0118] In summary, the method of the present invention significantly reduces data volume while taking into account access speed, analytical flexibility, and long-term maintainability.

[0119] Experimental results:

[0120] This example experiment, based on measured data from the Nanshan 26-meter radio telescope at the Xinjiang Astronomical Observatory, verifies the effectiveness of the proposed electromagnetic interference data processing method. The experiment selected 58 seconds of continuous observation data collected on December 18, 2024, within an azimuth angle of 10°–360° and an elevation angle of 10°. The original PSRFITS format file was approximately 60GB. Through a standardized processing workflow—including key steps such as polarization separation, frequency band grouping, and noise filtering—the data was efficiently compressed and converted to a unified HDF5 format for storage, reducing the total size to 13GB. This processing not only significantly improves data manageability and access efficiency but also ensures compatibility with data from large radio telescopes such as FAST and SKA, providing a reliable data foundation for subsequent electromagnetic interference analysis and radio environment monitoring.

[0121] The specifications of the raw baseband data and PSRFITS format files are as follows:

[0122] Observation equipment: Nanshan 26m radio telescope, Xinjiang Astronomical Observatory;

[0123] Data format: PSRFITS standard format;

[0124] Frequency range: 1000MHz-2024MHz;

[0125] Observation duration: 58 seconds of continuous observation;

[0126] File size: The original PSRFITS file is approximately 60GB.

[0127] PSRFITS is a standard data format in the field of radio astronomy, ensuring data compatibility with large telescopes such as FAST and SKA, and providing a unified input source for subsequent standardized processing.

[0128] The formats of folders and PSRFITS files are as follows:

[0129] data_ns /

[0130] ├── RFL_az_el_10_10 / / / folder

[0131] │ ├── xrb2024_1218_072140_0001.sf / / PSRFITS format file

[0132] ├── RFL_az_el_20_10 /

[0133] │ ├── xrb2024_1218_072237_0001.sf

[0134] ├── RFL_az_el_30_10 /

[0135] │ ├── xrb2024_1218_072435_0001.sf

[0136] │ ...

[0137] ├── RFL_az_el_150_10 /

[0138] │ ├── xrb2024_1218_074103_0001.sf

[0139] │ ...

[0140] ├── RFL_az_el_360_10 /

[0141] │ ├── xrb2024_1218_080110_0001.sf

[0142] Subsequently, the PSRFITS format file in the path is read, and the obtained raw baseband data, basic observation attributes and file paths are processed through a batch preprocessing program to achieve polarization separation, frequency band grouping, noise filtering and HDF5 format storage.

[0143] The original PSRFITS file (60GB) undergoes [polarization separation] to obtain horizontal / vertical polarized electromagnetic interference data (in-memory structure); then [frequency band grouping] to obtain a logical group matrix of interfering service frequency bands (in-memory structure); next, [noise filtering] to obtain a denoised logical group matrix + time index (in-memory structure); finally, [HDF5 compression storage] to obtain a standardized HDF5 file set (total 13GB). Each HDF5 file includes multiple core datasets, and each core dataset includes attribute items and data items.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. All simple and equivalent changes and modifications made in accordance with the claims and description of this application fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.< / time> < / time> < / date> < / az> < / el>

Claims

1. A method for extracting and storing electromagnetic interference data from radio astronomy, characterized in that, include: Step S1: Acquire raw baseband data and store it in PSRFITS format; Step S2: Define the storage data structure as including the original electromagnetic interference data matrix and the attribute data key-value pair set, the attribute data key-value pair set including multiple attribute data; for each PSRFITS format file, extract the original electromagnetic interference data matrix and attribute data according to the original baseband data, basic observation attributes and file path of the PSRFITS format file; Step S3: Further extract and group the original electromagnetic interference data matrix to obtain a denoised data matrix with polarization and interference service frequency bands, and with frequency and time indices, as well as the corresponding attribute data key-value pair set; Step S4: Define each HDF5 file as a combination corresponding to a polarization mode POL and an interfering service frequency band; store the denoised and filtered logical group matrix and time index vector under multiple spatial coordinates as multiple core datasets, together with the corresponding attribute data key-value pair set in the corresponding HDF5 file.

2. The method for extracting and storing radio astronomical electromagnetic interference data according to claim 1, characterized in that, The raw baseband data is obtained by observations at different azimuth and elevation angles using a radio telescope. When stored in PSRFITS format, each spatial coordinate corresponds to a folder, and each folder stores only one PSRFITS format file. Each PSRFITS format file includes the raw baseband data and basic observation attributes under a spatial coordinate, which is a combination of azimuth and elevation angles.

3. The method for extracting and storing radio astronomy electromagnetic interference data according to claim 1, characterized in that, The definition of the storage data structure is as follows: ， Where P is the storage data structure, For attribute data key-value pairs, This represents the 42nd attribute data; The original electromagnetic interference data matrix extracted from the original baseband data is four-dimensional, and its size is Ns × Nt × the number of sub-integrals × the number of sampling points per sub-integral × the number of polarization modes of the original baseband data × the number of frequency channels. ; Extracting the original electromagnetic interference data matrix from the raw baseband data refers to integrating heterogeneous data scattered in the PSRFITS format file to obtain a matrix with dimensions of Ns (number of sub-integrations) × Nt (number of sampling points per sub-integration) × number of polarization modes of the original baseband data × number of frequency channels. The original electromagnetic interference data matrix.

4. The method for extracting and storing radio astronomical electromagnetic interference data according to claim 1, characterized in that, In the attribute data key-value pair set, the attribute data is obtained from the header of the PSRFITS file or from the file path.

5. The method for extracting and storing radio astronomical electromagnetic interference data according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Separate the original electromagnetic interference data matrix D according to the dimension of polarization to obtain the horizontally polarized electromagnetic interference data matrix D. H and vertically polarized electromagnetic interference data matrix D V ; Step S32: For the horizontally polarized electromagnetic interference data matrix D... H and vertically polarized electromagnetic interference data matrix D V Adding frequency and time indices yields a horizontally polarized electromagnetic interference data matrix with both frequency and time indices. and vertical polarization electromagnetic interference data matrix ; Step S33: According to the predefined multiple interference service frequency bands freq_ranges, generate the horizontal polarization electromagnetic interference data matrix with frequency index and time index. and vertical polarization electromagnetic interference data matrix Each is divided into logical group matrices for different interference service frequency bands. The logical group matrices correspond one-to-one with the interference service frequency bands, and the attribute data key-value pair set of each logical group matrix is ​​obtained. Step S34: For each logical group matrix Perform noise reduction filtering; then process the noise-reduced logic group matrix. It compresses its time index vector.

6. The method for extracting and storing radio astronomical electromagnetic interference data according to claim 5, characterized in that, In step S32, when calculating the frequency index, the frequency index is determined by using the start frequency START, end frequency STOP, and bandwidth CHAN_BW of each frequency channel in the basic observation attributes of the PSRFITS format file as the start point, end point, and step bandwidth of the frequency index; when calculating the time index, the time index starts from 1, the time index of each line is incremented by 1, and the time index of the last line is 0.

7. The method for extracting and storing radio astronomy electromagnetic interference data according to claim 5, characterized in that, In step S33, the set of interfering service frequency bands is as follows: , For the i-th interfering service frequency band, each interfering service frequency band It has a corresponding frequency space , For the i-th interfering service frequency band Start and end frequency indices, band selection function Frequency index representing the q-th frequency channel Is it in the i-th interfering service frequency band? If so, then the frequency band selection function True; For an arbitrary polarization electromagnetic interference data matrix with frequency and time indices, the i-th interfering service frequency band Logical group matrix for: ， Where M is an arbitrary polarization electromagnetic interference data matrix with frequency and time indices. , The frequency band selection function represents the electromagnetic interference data matrix M. The submatrix formed by the columns containing the true frequency indices. , is a vector of time indices. For the first Time index of sampling points; Meanwhile, according to the set of interference service frequency bands Various interference frequency bands The frequency range, for each interfering service frequency band. The frequency-related attribute data is corrected to obtain the frequency bands of each interfering service. Logical group matrix The set of attribute data key-value pairs.

8. The method for extracting and storing radio astronomy electromagnetic interference data according to claim 1, characterized in that, The logic group matrix after noise filtering The compression of the logic group matrix and its time index vector includes compressing the denoised and filtered logic group matrix and its time index vector into uint8 and uint32 types, respectively.

9. The method for extracting and storing radio astronomy electromagnetic interference data according to claim 1, characterized in that, The HDF5 file name is the name of the interfering service frequency band.

10. A method for extracting and storing radio astronomical electromagnetic interference data according to claim 1, characterized in that, The HDF5 file includes multiple core datasets. Each denoised and filtered logical group matrix and each time index vector are respectively considered as a core dataset. The name of each core dataset includes spatial coordinates and timestamps, and the attribute data key-value pair set is stored in the attribute items of each core dataset in the HDF5 file.

Images