Radio astronomical data intelligent grading compression method and system based on deep learning

Abstract

The invention discloses an intelligent staged compression method and system for radio astronomical data based on deep learning. The method comprises the following steps: 1) constructing a celestial body signal frequency domain template library according to observation data in the radio astronomy field; 2) mapping current observation data to be processed to a frequency domain to generate a frequency point matrix; 3) extracting a frequency domain feature of each frequency point from the frequency point matrix, performing similarity calculation on the frequency domain feature and a feature corresponding to each template, if the similarity is greater than a set threshold value, determining that the corresponding frequency point is an effective frequency point, otherwise, determining that the corresponding frequency point is an invalid frequency point; (3) the effective frequency points are analyzed and processed according to channel dimensions, and if the difference between the adjacent channel data is smaller than or equal to a difference threshold value, the difference value between the adjacent channel data is recorded in a difference coding mode; 4) performing secondary compression on the output data in the step 3); and 5) generating a standard Fits format file from the secondary compressed coding data and adding metadata parameters. According to the method, the bottleneck problem of mass data storage and transmission caused by multi-beam observation is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of radio astronomy big data processing, and specifically relates to an intelligent hierarchical compression method and system that integrates blind source signal separation, spatiotemporal joint coding, and deep learning enhanced compressed sensing. It performs intelligent data compression processing on the massive celestial radio signal data collected by radio telescopes to solve key technical problems encountered in the data storage and transmission process, and promote the efficient development of scientific research in the field of radio astronomy. Background Art

[0002] In contemporary radio astronomy research, multi-beam observation technology, with its ability to simultaneously observe multiple regions of the sky, has become an important means of acquiring data on astronomical radio signals. For example, my country's Five-hundred-meter Aperture Spherical Radio Telescope (FAST) uses a multi-beam observation system that can simultaneously cover multiple observation areas, generating massive amounts of data in a single observation. This data encompasses radio signals from various celestial objects and across different frequency bands, and contains crucial information on a variety of astrophysical phenomena, including neutral hydrogen (HI) lines and fast radio bursts (FRBs).

[0003] However, with the improvement of observation accuracy and the increase in observation duration, data volume has experienced explosive growth. Traditional data compression algorithms, such as general-purpose lossless compression algorithms (such as ZIP and GZIP) and some specialized compression algorithms, have significant limitations when processing multi-beam radio astronomy observation data. Firstly, traditional compression algorithms struggle to fully exploit the inherent characteristics of radio astronomy data. Applications of traditional compression algorithms, such as Huffman coding and differential coding, in radio astronomy face the following challenges: Limited compression efficiency: Traditional methods achieve a maximum compression rate of only 40%, which is insufficient for processing multi-beam observation data. Risk of weak signal loss: Differential coding with a fixed threshold can misclassify weak signals as noise, resulting in the loss of critical information. High computational resource consumption: Processing massive amounts of data requires high-performance computing, and traditional algorithms lack the parallelization capability. This forces astronomical data centers to invest heavily in storage equipment, resulting in high storage costs. Secondly, due to insufficient compression efficiency, data transmission times are significantly extended, severely impacting researchers' ability to analyze and process data in a timely manner and hindering the rapid advancement of radio astronomy research.

[0004] Furthermore, the weak signal features contained in radio astronomy observation data are extremely susceptible to noise interference. Traditional compression algorithms struggle to accurately distinguish between noise and valid signals during the compression process, easily leading to the loss or misjudgment of valid signals, further impacting data availability and the accuracy of scientific research results. Therefore, there is an urgent need for an efficient and intelligent data compression processing method specifically for radio astronomy multi-beam observation data to overcome existing technological bottlenecks. The massive amount of data generated by radio astronomy observations places enormous pressure on storage and transmission. Therefore, there is an urgent need for an efficient and intelligent data compression processing method specifically for radio astronomy multi-beam observation data to overcome existing technological bottlenecks. Summary of the Invention

[0005] In response to the problems existing in the prior art, the present invention aims to provide a method and system for intelligent hierarchical compression of radio astronomy data based on deep learning. To address the problem of weak signals in radio astronomy data being easily overwhelmed by noise, the present invention constructs a dual-stream FastICA architecture with a supervised and unsupervised stream in parallel. Through the dual mechanism of "prior knowledge guidance + data-driven self-learning," the system achieves precise separation of signal and noise. At the same time, through the innovative fusion of frequency domain feature screening and the original dynamic threshold differential coding compression technology, the system effectively solves the bottleneck problem of massive data storage and transmission generated by multi-beam observations. This method achieves efficient compression of radio astronomy data, significantly saving storage space while ensuring efficient data transmission. By strictly controlling the missed detection rate and false detection rate, the integrity of weak signal characteristics is guaranteed, providing a high-quality, reliable data foundation for radio astronomy research and promoting research and development in the field of radio astronomy.

[0006] The technical solution of the present invention is:

[0007] A method for intelligent hierarchical compression of radio astronomy data based on deep learning, comprising the following steps:

[0008] 1) The template library management module builds a frequency domain template library of celestial body signals based on observation data in the field of radio astronomy;

[0009] 2) The data preprocessing module maps the current observation data to be processed into the frequency domain and generates a frequency matrix;

[0010] 3) The frequency domain feature screening module extracts the frequency domain feature of each frequency point from the frequency point matrix and calculates the similarity with the corresponding feature of each template in the celestial signal frequency domain template library. If the similarity is greater than a set threshold, the corresponding frequency point is a valid frequency point, otherwise it is an invalid frequency point;

[0011] 4) The primary compression module analyzes and processes the effective frequency points according to the channel dimension. If the difference between adjacent channel data is less than or equal to the difference threshold Δ max, the data of the adjacent channels are determined to be correlated and the difference between the data of the adjacent channels is recorded using differential coding; otherwise, the original data of the next channel in the adjacent channels is directly recorded;

[0012] 5) The secondary compression module performs secondary compression on the data output by the primary compression module; wherein the method for performing secondary compression on the difference values ​​after differential encoding is as follows: first, the frequency of occurrence of each difference value is counted; if the frequency of occurrence of the difference value is greater than a set threshold, it is regarded as a high-frequency symbol; otherwise, it is a low-frequency symbol; then, the high-frequency symbols are encoded using Huffman coding, and the low-frequency symbols are encoded using arithmetic coding, and shorter codes are assigned to the high-frequency symbols and longer codes are assigned to the low-frequency symbols;

[0013] 6) The result output module generates a standard Fits format file from the compressed coded data output by the secondary compression module and appends metadata parameters.

[0014] Furthermore, the dual-stream FastICA algorithm is used to identify effective frequency points from the frequency matrix. The method is as follows:

[0015] 21) For each frequency point of the frequency matrix, extract its power spectrum density vector and polarization degree vector as the eigenvector v of the frequency point, and calculate the cosine similarity between the eigenvector v of the frequency point and the eigenvector t corresponding to each template in the celestial signal frequency domain template library. If Sim(v,t) is greater than the set dynamic matching threshold, the frequency point is stored in the valid set as a valid frequency point, otherwise it is a non-matching frequency point; the feature vector v of the frequency point is saved in the candidate set;

[0016] 22) constructing a linear mixed signal X=AS based on the non-matching frequency points in the frequency matrix, and separating independent components from the linear mixed signal X using an unsupervised blind source separation method based on kurtosis optimization; calculating the mean μnoise and standard deviation σnoise of the independent components; if the eigenvector v of the frequency point satisfies |v-μnoise|<3σnoise, then marking the frequency point as a noise-dominant frequency point and filtering it out, and treating the remaining signal as an unknown signal; wherein A is a mixing matrix composed of non-matching frequency points, S is an independent component, and the independent component is a celestial signal or noise;

[0017] 23) According to the mean μnoise and standard deviation σnoise of each unknown signal and the eigenvalue S of the eigenvector t corresponding to each template template , calculate the signal-to-noise sensitivity factor When α≥3σnoise, the unknown signal is determined to be a valid frequency point.

[0018] Furthermore, the difference threshold Δ max =0.1×min(xi ,x i+1 ).

[0019] Furthermore, the data preprocessing module first converts the currently processed observation data into a unified format, and then decomposes the data in the unified format into low-frequency components and high-frequency components through wavelet transform; then suppresses the background noise of the low-frequency component through a threshold shrinkage algorithm, locates the weak signal in the high-frequency component through an attention mechanism algorithm and generates a gain coefficient of the weak signal, and uses the gain coefficient to enhance the weak signal in the high-frequency component; then reconstructs the denoised low-frequency component and the enhanced high-frequency component into a time domain signal through an inverse wavelet transform and inputs it into a multi-phase filter group for channelization processing, and then frames and windows each channel signal and performs a fast Fourier transform, calculates the frequency power and screens and calibrates the effective frequency; finally, organizes the data according to the "channel × frequency × time" dimension to generate the frequency matrix.

[0020] Furthermore, the celestial signal frequency domain template library includes multiple templates, each template corresponding to a type of representative celestial signal frequency domain characteristics.

[0021] Furthermore, the polarization characteristics of the observation data are extracted and fused with the frequency domain characteristics of the celestial body signal corresponding to each template to obtain a multi-dimensional feature vector of the corresponding template.

[0022] Furthermore, the metadata parameters include template library version and compression parameters; dynamic gain compensation parameters are attached to the header of the Fits format file to record the gain adjustment strategy for weak signals during the compression process to ensure that the original dynamic range of the signal is restored during decompression.

[0023] A radio astronomy data intelligent hierarchical compression system based on deep learning is characterized by comprising a template library management module, a data preprocessing module, a frequency domain feature screening module, a primary compression module, a secondary compression module and a result output module; wherein,

[0024] The template library management module is used to construct a celestial signal frequency domain template library based on observation data in the field of radio astronomy;

[0025] The data preprocessing module is used to map the currently processed observation data into the frequency domain to generate a frequency matrix;

[0026] The frequency domain feature screening module is used to extract frequency domain features from the frequency point matrix and match them with the corresponding features of each template in the celestial signal frequency domain template library. If they match, the current frequency point matrix is ​​valid frequency point data, otherwise it is invalid frequency point;

[0027] The primary compression module is used to analyze and process the effective frequency point data according to the channel dimension. If the difference between adjacent channel data is less than or equal to the difference threshold Δm ax, the data of the adjacent channels are determined to be correlated and the difference between the data of the adjacent channels is recorded using differential coding; otherwise, the original data of the next channel in the adjacent channels is directly recorded;

[0028] The secondary compression module is configured to perform secondary compression on the data output by the primary compression module. The secondary compression method for the differential values ​​after differential encoding is as follows: first, the frequency of occurrence of each differential value is counted; if the frequency of occurrence of the differential value is greater than a set threshold, it is considered a high-frequency symbol; otherwise, it is a low-frequency symbol; then, the high-frequency symbols are encoded using a Huffman coding method, and the low-frequency symbols are encoded using an arithmetic coding method, and shorter codes are assigned to the high-frequency symbols and longer codes are assigned to the low-frequency symbols;

[0029] The result output module is used to generate a standard Fits format file from the compressed coded data output by the secondary compression module and append metadata parameters

[0030] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above method.

[0031] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the above method when executed by a processor.

[0032] The solution includes six modules: template library management module 1, data preprocessing module 2, frequency domain feature screening module 3, primary compression module 4, secondary compression module 5 and result output module 6. Specifically:

[0033] Template Library Management Module 1 builds and stores a library of frequency-domain templates for astronomical signals based on existing radio astronomy observations and data. This module dynamically selects 10 representative frequency-domain signatures of astronomical signals, including HI lines and FRBs, as templates. HI line signatures reflect the distribution and motion of neutral hydrogen atoms in interstellar space, while FRB signatures are closely related to the mysterious fast radio burst phenomenon. These templates cover the frequency-domain signatures of common and important astrophysical phenomena in radio astronomy research, providing an accurate benchmark for subsequent data screening.

[0034] This module has two innovative designs: a. Dynamic template update mechanism. By introducing an online learning algorithm, when the matching degree between new observation data and existing templates is lower than the threshold, the template library update process is automatically triggered. New feature vectors are generated through incremental learning and added to the template library to ensure the timeliness and comprehensiveness of the template library. b. Multimodal feature fusion: Fusion of frequency domain features and polarization features. In the template library, the frequency domain features of the celestial signal of each template are fused with the same polarization features to obtain the multi-dimensional feature vectors corresponding to each template, thereby improving the robustness of template matching. Polarization features refer to key parameters that describe the polarization state of radio signals, including: polarization degree, which measures the proportion of the signal polarization component in the total signal intensity; polarization angle, which is the angle between the polarization direction of the linearly polarized signal and the reference axis. These features need to be extracted from the original observation data through data processing algorithms.

[0035] Data preprocessing module 2 converts the original observation data into a format and form suitable for algorithm processing. Perform format unification processing (such as converting the FAST fits format into a binary matrix) to solve the problem of heterogeneous data. Frequency domain conversion (FFT) maps the time domain signal to the frequency domain. By introducing the polyphase filter bank PFB for channelization processing before the FFT transformation, the spectrum leakage and fan-shaped loss are effectively reduced. At the same time, an improved Wave-U-Net model is designed. The input contains a multi-channel time domain waveform, which is output through the wavelet transform layer and the dual branch. The entire processing flow is as follows: the multi-channel time domain waveform is decomposed into low-frequency approximate components and high-frequency detail components through the wavelet transform layer; the background noise of the low-frequency component is suppressed by the threshold shrinkage algorithm (reference: Donoho, DL, & Johnstone, IM (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81 (3), 425-455), and the attention mechanism algorithm (reference: Woo, S., et al. (2018). CBAM: Convolutional block attention module. European Conference on Computer Vision (ECCV) locates weak signals in high-frequency components and generates gain coefficients for these weak signals (set to 1.5 when SNR ≤ 5, otherwise 1). The gain coefficients are then used to enhance the weak signals in the high-frequency components, and the gain coefficient matrix is ​​recorded in the data frame header (later used for dynamic gain compensation in the result output module 6). The denoised low-frequency components and the enhanced high-frequency components are reconstructed into a time-domain signal via an inverse wavelet transform. The time-domain signal reconstructed by Wave-U-Net is then channelized using a polyphase filter bank (PFB). This involves anti-aliasing filtering, polyphase decomposition, and decimation downsampling, dividing the broadband signal into multiple narrowband channels to suppress spectral leakage and scalloping. Each channel signal is then framed, windowed (e.g., with a Hanning window), and subjected to a fast Fourier transform (FFT). Frequency power is calculated and valid frequencies are selected for calibration. Finally, the data is organized into a "channel × frequency × time" matrix, forming a frequency matrix (e.g., 256 channels × 2048 frequencies × 100 time frames).

[0036] This process combines the efficient frequency band segmentation of PFB with the precise frequency domain conversion of FFT, achieving spectrum leakage suppression (less than -60dB) and frequency domain feature standardization while reducing computational complexity. It can also dynamically adjust the frequency power in combination with the gain coefficients generated by Wave-U-Net, providing structured input for subsequent frequency domain feature screening. The overall process forms a closed-loop process of "wavelet decomposition → dual-branch parallel processing (noise suppression + weak signal enhancement) → inverse transform reconstruction + gain coefficient storage", achieving a 30% reduction in background noise and a 1.5-fold increase in weak signal amplitude. This process breaks the traditional "single filtering / transformation" preprocessing mode and forms a multi-scale feature optimization for radio signals through the cascade processing of "deep learning time domain enhancement → classical signal processing frequency domain regularization". The deep learning time domain enhancement is reflected in the fact that Wave-U-Net achieves time domain noise suppression and weak signal enhancement through the wavelet transform layer and dual-branch network, solving the problem of time domain noise and weak signal aliasing. Multi-scale feature optimization of radio signals achieves frequency-domain regularization similar to classical signal processing through channelization and frequency-domain feature screening using a polyphase filter bank (PFB) + FFT. This ensures accurate frequency-domain segmentation (reducing feature misjudgment caused by spectrum leakage). The processing results generate a frequency matrix (dimensions: channel × frequency × time) that can be processed by frequency-domain feature screening module 3.

[0037] The frequency domain feature screening module 3 uses the template created by the template library management module 1 to screen the frequency matrix output by the data preprocessing module 2, that is, through the frequency domain feature screening module of the dual-stream FastICA (fast independent component analysis) algorithm, the dual-stream FastICA algorithm automatically learns and identifies the frequency domain feature patterns in the data. When processing data, the frequency matrix of the observed data is deeply compared and analyzed with the features in the template library. Through the dual-stream FastICA algorithm, the observed data can be separated into independent components, thereby identifying which components are caused by noise and which components are caused by celestial signals. The dual-stream FastICA algorithm can accurately identify invalid frequency points dominated by noise. For those frequency points that do not match the features in the template library and exhibit noise characteristics, the algorithm determines them as invalid frequency points and automatically performs filtering operations. Practical tests have verified that this method can effectively filter out 70% of invalid frequency points dominated by noise, significantly reducing the amount of data while retaining key valid signal frequency points.

[0038] Compared to traditional ICA algorithms, the dual-stream architecture is designed to run supervised and unsupervised streams in parallel, achieving weak signal preservation and accurate noise identification through three-dimensional optimization (algorithm-hardware-application). Dual-stream FastICA breaks through the bottleneck of traditional ICA in the field of radio astronomy through three-dimensional optimization of algorithm-hardware-application. The main innovations are as follows:

[0039] a. Algorithm level: The dual-stream architecture and dynamic threshold are used to solve the problem of weak signal loss. The algorithm process is as follows: Figure 2 As shown, the parallel dual-stream architecture is:

[0040] (1) Supervision flow: First, based on the supervised feature matching of the historical template library, 10 types of celestial body signal templates are dynamically loaded; for each frequency point of the input frequency matrix, its power spectrum density vector (frequency domain amplitude distribution) and polarization vector (signal polarization feature) are extracted, and the cosine similarity is calculated with the 10 types of feature vectors in the template library: Where v is the feature vector of the frequency point to be matched, and t is the feature vector of the template. Vector normalization is used to eliminate the dimension effect. A dynamic matching threshold is set (default ≥ 0.6) to filter out the frequency points that match any template and mark them as "potentially valid frequency points", retaining the candidate set with known signal features.

[0041] (2) Unsupervised flow: In the absence of prior knowledge, the noise components in the mixed signal are separated by a data-driven method to identify weak signals or new signals that do not match the template. The algorithm is based on unsupervised blind source separation with kurtosis optimization to achieve self-learning of noise features. Specifically: for the frequency points of the unmatched template, the observation data of each channel and each time point are regarded as a linear mixed signal X = AS, where X is the observation data (dimension: channel × frequency point × time), A is the mixed matrix of the unmatched frequency points, and S is the independent component (including celestial body signals and noise). The FastICA fixed point iteration algorithm is used to solve the separation matrix W based on kurtosis optimization (the objective function is to maximize the independent component kurtosis (Kurtosis) Kurtosis(s) = E{s 4}-3[E{s 2}] 2 ), the observation data X is transformed by W to obtain the estimated value of the independent component Y = WX (where the separation matrix W is the transformation matrix used to estimate the independent component S from the observation data. By optimizing the objective function, Y is made as close as possible to the true independent component S, realizing the separation of signal and noise. Moreover, due to the non-Gaussian nature of the celestial body signal, the absolute value of the kurtosis of the corresponding independent component is significantly higher than that of the noise (noise kurtosis ≈ 0)). The components with kurtosis values ​​falling within the interval of [-0.5, 0.5] are judged as noise, and their mean μnoise and standard deviation σnoise are calculated to construct a model. If the eigenvector v of the frequency point satisfies |v-μnoise|<3σnoise (i.e., it falls within the 99.7% confidence interval of the noise component, following the 3σ principle), it is marked as a "noise-dominated frequency point" and filtered out to avoid misjudging weak signals as noise.

[0042] Finally, the "potentially effective frequency points" output by the supervised stream are directly added to the effective set (screened from potential effective frequency point data to confirmed effective frequency point data), ensuring 100% recall of known signals; the unsupervised stream performs noise filtering on frequency points that do not match the template, retaining unknown signals with kurtosis significantly higher than the noise (such as new pulsar signals). At the same time, dynamic threshold optimization is performed to calculate the signal-to-noise ratio sensitivity factor: Among them S template is the eigenvalue of the template eigenvector t. When α ≥ 3σ noise (corresponding to statistical significance ≥ 99.7%), it is determined to be a valid signal, reducing the missed detection rate by 62% compared to the traditional fixed threshold method, and ensuring a 98% retention rate for weak signals with SNR ≥ 3.

[0043] b. Hardware: We designed a CUDA kernel function to execute dual-stream FastICA in parallel, reducing the time required for 2048-channel data separation from 58 seconds to 2.3 seconds. This supports real-time processing of TB-level data and can also adapt to the high-throughput requirements of next-generation radio telescopes such as the SKA.

[0044] This dual-stream architecture breaks through the limitations of traditional ICA that relies on a single prior or pure data-driven approach. Through the complementary mechanism of "supervised matching to preserve the known and unsupervised separation to mine the unknown", it accurately retains weak signal characteristics while filtering out 70% of noise frequencies, providing a high-purity data foundation for subsequent compression links. It solves the problem of weak signal loss caused by fixed thresholds in traditional algorithms from the source, and promotes the professional application of blind source separation technology in the field of radio astronomy.

[0045] The primary compression module 4 analyzes and processes the effective frequency point data after frequency domain feature screening according to the channel dimension. In radio astronomy observations, data is usually collected in the form of multiple channels, and each channel corresponds to a different frequency band or observation direction. When the difference between adjacent channel data is ≤10%, it indicates that the data of the two channels have a high correlation. At this time, a differential encoding method is used to record the difference value between adjacent channel data. When the difference between adjacent channel data is greater than 10%, the original data of the latter channel is directly recorded. In this way, data redundancy can be effectively reduced and the amount of data can be further reduced. For example, if the two data values ​​of adjacent channels at a certain moment are 100 and 105 respectively, if the differential encoding conditions are met, only their difference of 5 is recorded, thereby saving data storage space.

[0046] Based on this, this patent innovatively designs the spatiotemporal joint differential coding of astronomical data. Its processing flow is as follows: first, the channel dimension is considered, the dynamic threshold is calculated for each channel, and it is determined whether to use differential coding or record the original data. Then, an ARIMA model is constructed from the time dimension to predict the data at the next moment and calculate the residual. It is then determined whether to enable predictive coding. Finally, a hybrid coding protocol is used for compression based on the number of channels that continuously meet the conditions.

[0047] Specific implementation: channel dimension, dynamic adjustment of adjacent channel differential threshold (Δ max =0.1×min(x i ,x i+1 )); In the time dimension, an ARIMA model is constructed to predict the data trend at the next moment. When the residual is ≤ 5%, predictive coding is enabled. Differential hybrid coding protocol satisfies Δ≤ dynamic adjustment threshold Δ for N consecutive channels. m When using ax, (RLE,Δ) tuple encoding is used (e.g., (5,0.05) represents a 5% difference between five consecutive channels), resulting in an 18% improvement in compression compared to traditional differential encoding. While traditional differential encoding only processes two adjacent channels (N=2), this patented hybrid encoding protocol supports compression of long continuous segments. This design leverages the strong correlation between adjacent frequency channel data in radio astronomy multi-beam observations (e.g., consecutive frequency channels within the same beam often exhibit similar signal characteristics), improving efficiency through compression of long continuous segments.

[0048] Secondary Compression Module 5: After completing inter-channel differential encoding, it performs secondary compression on the data processed by Primary Compression Module 4. Adaptive hybrid entropy coding is used to perform secondary compression on the differential values ​​after differential encoding. This method combines Huffman coding and arithmetic coding, dynamically selecting the encoding method based on the context of the current symbol (such as the entropy value of the previous symbol). Specifically, Huffman coding is used for small differences that occur frequently, and arithmetic coding is used for large differences that occur less frequently. Different codes are assigned to each symbol based on their frequency of occurrence in the data. Frequently occurring symbols are assigned shorter codes, while less frequently occurring symbols are assigned longer codes. This optimizes coding efficiency and fully utilizes the statistical characteristics of the data to achieve efficient data compression. Compared to traditional single Huffman coding, the adaptive hybrid entropy coding algorithm can further improve the compression rate by 12%. After the data from Primary Compression Module 4 is processed by the adaptive hybrid entropy coding of Secondary Compression Module 5, the final compressed data is obtained.

[0049] Result output module 6 is used to generate and store the two-level compressed data in the standard Fits format for astronomical data. Metadata parameters (such as template library version and compression parameters) are also added to ensure algorithm compatibility during back-end decompression. Dynamic gain compensation metadata is innovatively implemented, adding dynamic gain compensation parameters to the FITS file header to record the gain adjustment strategy for weak signals during compression, ensuring that the original dynamic range of the signal is restored during decompression. The gain coefficient matrix is ​​appended to the FITS file header as metadata in the form of a binary table and managed in shards via the BOINC distributed storage system. During decompression, the signal is then inversely gain-calculated (multiplied by the gain coefficient) based on this matrix to restore the enhancement effect of the weak signal, ensuring that the signal dynamic range is consistent with the results of the data preprocessing module, forming a closed loop of "inverse compensation before compression → parameter optimization during compression → positive recovery after decompression." The BOINC-based distributed storage system stores the compressed data in shards across multiple nodes, supporting parallel reading and fault-tolerant recovery, improving storage efficiency and reliability.

[0050] The advantages of the present invention are as follows:

[0051] 1. Dual-stream Feature Separation and Dynamic Threshold Screening Technology: This pioneering dual-stream FastICA architecture integrates supervised template matching (using 10 types of celestial signal templates) with unsupervised blind source separation. Dynamic thresholding is used to calculate the signal-to-noise ratio sensitivity factor α ≥ 3σ noise (99.7% statistical significance) to accurately identify valid signals. This reduces the missed detection rate by 62% compared to traditional algorithms, while retaining 98% of weak signals (SNR ≥ 3). GPU acceleration reduces the time required for 2048-channel separation from 58 seconds to 2.3 seconds, filtering out 70% of noise frequencies and providing a high-purity data foundation for compression.

[0052] 2. Spatiotemporal Joint Differential Coding Technology: This technology proposes cross-dimensional joint coding, employs dynamic threshold adaptive adjustment in the channel dimension, and uses an ARIMA model to predict trends in the time dimension (predictive coding is enabled when the residual is ≤ 5%). Combined with differential hybrid protocols (such as (RLE, Δ) tuple coding), this technology improves the compression rate by 18% compared to traditional differential coding, and reduces inter-channel redundancy from 65% to 22%, becoming a key step in achieving an overall compression rate of 63%.

[0053] 3. Adaptive Hybrid Entropy Coding and Gain Compensation Technology: This technology employs an intelligent entropy coding system that dynamically switches between Huffman coding (high-frequency, small differences) and arithmetic coding (low-frequency, large fluctuations), resulting in a 12% improvement in compression compared to Huffman coding alone. Dynamic gain compensation parameters (such as amplifying weak signals by a factor of 2) are embedded to ensure that the amplitude error of the decompressed signal is less than 3%, and weak signal fidelity reaches 98%. Distributed storage supports 1024 nodes in parallel, adapting to terabyte-level data processing. The system has successfully detected pulsars using FAST data, establishing a closed-loop compression-recovery integrity loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a system diagram of the present invention.

[0055] Figure 2 This is the processing flow chart of the dual-stream FastICA algorithm.

[0056] Figure 3 This is a curve showing the change of missed detection rate with compression rate. DETAILED DESCRIPTION

[0057] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0058] This method is applicable to radio astronomy observation data processing scenarios and is suitable for implementation in high-performance computing cluster environments. Figure 1 As shown, it includes the following modules.

[0059] Template Library Management Module 1 screens data from 10 types of astronomical signals, including HI lines and FRBs, from international radio astronomy databases (such as the NRAO and ASKAP data centers) and historical FAST observation data, extracting frequency domain data from the 100MHz-8GHz range. Each signal type undergoes a Fourier transform to obtain a frequency domain power spectrum. A peak detection algorithm is used to extract characteristic frequencies and construct feature vectors (e.g., the 1420MHz characteristic peak of the HI line and its harmonics). These feature vectors are stored as JSON files. Each template contains metadata such as signal type, characteristic frequency, and bandwidth, and is stored in a MongoDB database for rapid retrieval and updating.

[0060] Data preprocessing module 2 uses the fits.open function of the Astropy library to read radio astronomy data in fits format, converting it into a two-dimensional NumPy array (shape [number of channels, number of sampling points]), and storing it in binary file format (.bin) to reduce file redundancy. The SciPy library's fftpack.fft function performs a Fast Fourier Transform (FFT) on each channel's time-domain data to generate a three-dimensional frequency matrix (shape [number of channels, number of frequencies, timestamp]), which serves as input for frequency-domain feature filtering module 3. A frequency-domain transform (FFT) maps the time-domain signal to the frequency domain. Channelization is performed by introducing a polyphase filter bank (PFB) before the FFT transform, effectively reducing spectral leakage and scalloping. A deep learning model based on Wave-U-Net is also used to preprocess the time-domain signal to suppress background noise and enhance weak signal features. The resulting frequency matrix (dimensions: channel × frequency × time) is then processed by frequency-domain feature filtering module 3.

[0061] Frequency domain feature screening module 3 uses the dual-stream FastICA algorithm to input the frequency matrix output by data preprocessing module 2 into the trained model. It then performs classification prediction on each frequency, sets a threshold of 0.85 (i.e., frequencies predicted to be valid are retained if their probability is ≥ 0.85), filters out invalid frequencies, and outputs a dataset of valid frequencies. Dual-stream FastICA is an improved blind source separation technique that assumes that the observed data is a linear mixture of multiple independent source signals. The dual-stream FastICA algorithm can separate the observed data into independent components, thereby identifying which components are caused by noise and which are caused by celestial signals. It does not require prior knowledge of the source signals and can automatically separate the independent components. It is highly effective in separating mixed signals and can effectively remove noise interference. For example, when radio astronomy data is a mixture of multiple independent signal sources, dual-stream FastICA can separate these signal sources, thereby identifying and filtering out noise-dominated frequencies. For example, in multi-beam observations, the signals received by different beams may be mixed with each other. Dual-stream FastICA can help separate the independent signal of each beam and remove noise interference.

[0062] Primary compression module 4, traverses each channel of the valid frequency point data set in order, and calculates the absolute difference Δ=|x corresponding to the frequency point data of adjacent channels i -x i+1 ∣, if Δ≤10%×max(x i ,x i+1 ), differential encoding is used and the Δ value is recorded; otherwise, the original data x is recorded. i+1 The encoding result is stored as a new one-dimensional array as the input of the secondary compression module 5. A spatiotemporal joint differential encoding of astronomical data is designed: channel dimension: dynamic adjustment of the differential threshold of adjacent channels (Δ max =0.1×min(x i ,x i+1 )); Time dimension: Build an ARIMA model to predict the data trend at the next moment, and enable predictive coding when the residual is ≤5%; Differential hybrid coding protocol: When N consecutive channels meet Δ≤ threshold, use (RLE,Δ) tuple encoding (such as (5,0.05) to indicate a 5% difference between 5 consecutive channels), which increases the compression rate by 18% compared to traditional differential coding.

[0063] Secondary compression module 5 performs secondary compression on the differentially coded data based on adaptive hybrid entropy coding. It uses the Python standard library to implement adaptive hybrid entropy coding tree construction, counts the frequency of occurrence of each symbol in the differentially coded array, builds a priority queue, adds each symbol and its frequency as a node to the queue, repeatedly merges the two nodes with the lowest frequency until a complete adaptive hybrid entropy coding tree is formed, traverses the tree from the root node, generates binary codes for each symbol, and finally converts the differentially coded data into a binary stream according to the generated coding table to complete the secondary compression.

[0064] Result output module 6 implements Fits format generation. Using the `fits.PrimaryHDU` class from the Astropy library, the compressed binary stream is encapsulated as the primary data unit of a Fits file, with header information such as the data dimensions and data type set. Metadata is also added, such as the template library version number, dual-stream FastICA model parameters, and differential encoding thresholds, to ensure algorithm compatibility during back-end decompression. The generated Fits file is ultimately stored in the distributed storage system of the Radio Astronomy Data Center for subsequent access by researchers.

[0065] The curve of missed detection rate and compression rate variation of the above implementation scheme is as follows: Figure 3 As shown, it can be determined through experimental data that the present invention can ensure stable and efficient operation in actual radio astronomy data processing and realize intelligent compression and reliable storage of massive observation data.

[0066] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A method for intelligent hierarchical compression of radio astronomy data based on deep learning, comprising the following steps: 1) The template library management module builds a frequency domain template library of celestial body signals based on observation data in the field of radio astronomy; 2) The data preprocessing module maps the current observation data to be processed into the frequency domain and generates a frequency matrix; 3) The frequency domain feature screening module extracts the frequency domain feature of each frequency point from the frequency point matrix and calculates the similarity with the corresponding feature of each template in the celestial signal frequency domain template library. If the similarity is greater than a set threshold, the corresponding frequency point is a valid frequency point, otherwise it is an invalid frequency point; 4) The primary compression module analyzes and processes the effective frequency points according to the channel dimension. If the difference between adjacent channel data is less than or equal to the difference threshold Δ m ax, the data of the adjacent channels are determined to be correlated and the difference between the data of the adjacent channels is recorded using differential coding; otherwise, the original data of the next channel in the adjacent channels is directly recorded; 5) The secondary compression module performs secondary compression on the data output by the primary compression module; wherein the method for performing secondary compression on the difference values ​​after differential encoding is as follows: first, the frequency of occurrence of each difference value is counted; if the frequency of occurrence of the difference value is greater than a set threshold, it is regarded as a high-frequency symbol; otherwise, it is a low-frequency symbol; then, the high-frequency symbols are encoded using Huffman coding, and the low-frequency symbols are encoded using arithmetic coding, and shorter codes are assigned to the high-frequency symbols and longer codes are assigned to the low-frequency symbols; 6) The result output module generates a standard Fits format file from the compressed coded data output by the secondary compression module and appends metadata parameters.

2. The method according to claim 1, characterized in that The two-stream FastICA algorithm is used to identify effective frequency points from the frequency matrix. The method is as follows: 21) For each frequency point of the frequency matrix, extract its power spectrum density vector and polarization degree vector as the eigenvector v of the frequency point, and calculate the cosine similarity between the eigenvector v of the frequency point and the eigenvector t corresponding to each template in the celestial signal frequency domain template library. If Sim(v,t) is greater than the set dynamic matching threshold, the frequency point is stored in the valid set as a valid frequency point, otherwise it is a non-matching frequency point; the feature vector v of the frequency point is saved in the candidate set; 22) constructing a linear mixed signal X=AS based on the non-matching frequency points in the frequency matrix, and separating independent components from the linear mixed signal X using an unsupervised blind source separation method based on kurtosis optimization; calculating the mean μnoise and standard deviation σnoise of the independent components; if the eigenvector v of the frequency point satisfies |v-μnoise|<3σnoise, then marking the frequency point as a noise-dominant frequency point and filtering it out, and treating the remaining signal as an unknown signal; wherein A is a mixing matrix composed of non-matching frequency points, S is an independent component, and the independent component is a celestial signal or noise; 23) According to the mean μnoise and standard deviation σnoise of each unknown signal and the eigenvalue S of the eigenvector t corresponding to each template template , calculate the signal-to-noise sensitivity factor When α≥3σnoise, the unknown signal is determined to be a valid frequency point.

3. The method according to claim 1, characterized in that The difference threshold Δ max =0.1×min(x i ,x i+1 ).

4. The method according to claim 1, wherein The data preprocessing module first converts the currently processed observation data into a unified format, and then decomposes the unified format data into low-frequency components and high-frequency components through wavelet transform; then suppresses the background noise of the low-frequency components through a threshold shrinkage algorithm, locates the weak signals in the high-frequency components through an attention mechanism algorithm and generates a gain coefficient for the weak signals, and uses the gain coefficient to enhance the weak signals in the high-frequency components; then reconstructs the denoised low-frequency components and the enhanced high-frequency components into time-domain signals through inverse wavelet transform and inputs them into a polyphase filter bank for channelization processing, then frames and windows each channel signal and performs a fast Fourier transform, calculates the frequency power and screens and calibrates the effective frequencies; finally, organizes the data according to the "channel × frequency × time" dimension to generate the frequency matrix.

5. The method according to claim 1, wherein The celestial signal frequency domain template library includes multiple templates, each template corresponding to a type of representative celestial signal frequency domain characteristics.

6. The method according to claim 6, characterized in that The polarization characteristics of the observation data are extracted and fused with the frequency domain characteristics of the celestial body signal corresponding to each template to obtain a multi-dimensional feature vector of the corresponding template.

7. The method according to claim 1, characterized in that The metadata parameters include template library version and compression parameters; dynamic gain compensation parameters are added to the header of the Fits format file to record the gain adjustment strategy for weak signals during the compression process to ensure that the original dynamic range of the signal is restored during decompression.

8. A radio astronomy data intelligent hierarchical compression system based on deep learning, characterized in that: It includes a template library management module, a data preprocessing module, a frequency domain feature screening module, a primary compression module, a secondary compression module and a result output module; wherein the template library management module is used to construct a celestial signal frequency domain template library based on observation data in the field of radio astronomy; The data preprocessing module is used to map the currently processed observation data into the frequency domain to generate a frequency matrix; The frequency domain feature screening module is used to extract frequency domain features from the frequency point matrix and match them with the corresponding features of each template in the celestial signal frequency domain template library. If they match, the current frequency point matrix is ​​valid frequency point data, otherwise it is invalid frequency point; The primary compression module is used to analyze and process the effective frequency point data according to the channel dimension. If the difference between adjacent channel data is less than or equal to the difference threshold Δ m ax, the data of the adjacent channels are determined to be correlated and the difference between the data of the adjacent channels is recorded using differential coding; otherwise, the original data of the next channel in the adjacent channels is directly recorded; The secondary compression module is configured to perform secondary compression on the data output by the primary compression module. The secondary compression method for the differential values ​​after differential encoding is as follows: first, the frequency of occurrence of each differential value is counted; if the frequency of occurrence of the differential value is greater than a set threshold, it is considered a high-frequency symbol; otherwise, it is a low-frequency symbol; then, the high-frequency symbols are encoded using a Huffman coding method, and the low-frequency symbols are encoded using an arithmetic coding method, and shorter codes are assigned to the high-frequency symbols and longer codes are assigned to the low-frequency symbols; The result output module is used to generate a standard Fits format file from the compressed coded data output by the secondary compression module and append metadata parameters.

9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images