A deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination

Abstract

The application discloses a kind of deep learning methods for low-frequency SKA broadband effect and synthetic beam effect elimination, belong to radio astronomy image processing field, including steps: S1, establish FSAS, FERM and FGFN;S2, based on the FSAS, FERM and FGFN of S1 establishment establishes IFS-Transformer network model;S3, two FERM modules are applied to obtain dirty image I ∈ R H×W×C Low-level features F0 ∈ R H×W×C ;S4, F0 is input FGFN module, and coding part is completed after twice downsampling operation;S5, the feature obtained by coding part is passed through 3 FSAS and twice upsampling, realizes feature learning and decoding operation;S6, after being processed by two FERM, final recovered image is obtained.The effect elimination method based on deep learning can more effectively eliminate the joint of broadband effect and synthetic beam effect, and can greatly reduce the time of effect elimination to a greater extent to restore and reconstruct original sky brightness;The manual operation process of the method of the patent can be greatly simplified by using deep learning technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radio astronomy image processing, in particular to a deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination. BACKGROUND

[0002] In radio astronomy, the square kilometer array (SKA) can greatly help humans explore and understand the universe due to its ultra-high sensitivity, resolution, and ultra-fast measurement speed. However, when using the SKA to conduct broadband observation imaging at low frequencies, broadband effects and synthetic beam effects are usually induced, resulting in distortion and blurring of the imaged celestial structure image.

[0003] Broadband effects mainly occur when imaging the true sky brightness in a wide band; limited bandwidth can cause the visibility function data to produce a trailing phenomenon in the radial direction, and the degree of radial trailing is positively correlated with the bandwidth size. The essence of the synthetic beam effect is that the incomplete sampling in the Fourier plane causes the point spread function of the telescope to have different degrees of sidelobes, resulting in image blurring when reconstructing the sky brightness. The coupling effect of broadband effects and synthetic beam effects causes more serious damage to the observed sky brightness, and the damage degree is affected by multiple factors such as observation frequency, observation time, bandwidth, field of view, etc.

[0004] Using traditional methods to jointly eliminate low-frequency SKA broadband effects and synthetic beam effects needs to be done in stages. First, the MFS method is applied to eliminate the broadband effect, and then the CLEAN algorithm is applied to eliminate the synthetic beam effect. Manual model design is required for the elimination of different effects, and constant parameter adjustment is required to complete the elimination of the effects. Such effect elimination method not only has low efficiency, but also cannot completely eliminate the broadband effect and the synthetic beam effect, and cannot accurately restore and reconstruct the sky brightness in the image. SUMMARY

[0005] To solve the technical problems of low efficiency of traditional methods for jointly eliminating low-frequency SKA broadband effects and synthetic beam effects, and inability to completely eliminate broadband effects and synthetic beam effects in the prior art, the present application provides a deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination.

[0006] To achieve the above technical solutions, the present application provides a deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination, comprising the steps of:

[0007] S1: establishing a frequency domain self-attention solver FSAS, a feature extraction residual module FERM, and a frequency domain gated forward network FGFN; the feature extraction residual module FERM is composed of one convolutional layer and four residual blocks;

[0008] S2: establishing an IFS-Transformer network model based on the FSAS, FERM and FGFN established in step S1;

[0009] S3, obtaining low-level features F0e R H×W×C of the dirty image Ie R H×W×C by applying two FERM modules;

[0010] S4, inputting the low-level features F0into the FGFN module to complete the encoding part through twice down-sampling operations;

[0011] S5, obtaining the features from the encoding part through three FSAS and twice up-sampling to realize feature learning and decoding operations;

[0012] S6, obtaining the final recovered image after processing by two FERM decoding modules.

[0013] Further, in step S1, the frequency domain self-attention solver FSAS is established by the following formula:

[0014] The query Q, the key K and the value V are obtained by extracting the image blocks and from the features F q , F k and F v :

[0015]

[0016] The scaled dot-product attention can be obtained by the following formula:

[0017]

[0018] Wherein, each element of QK T is obtained by inner product:

[0019]

[0020] The features F q , F k and F v are obtained by 1x1 convolution and 3x3 convolution respectively, then the fast Fourier transform is performed on F q and F k , and the correlation of F q and F k in the frequency domain is estimated by the following formula:

[0021]

[0022] The layer norm A is normalized to estimate the aggregated features:

[0023]

[0024] Finally, the output features of the FSAS are obtained by the following formula:

[0025] I att = I + Conv 1×1 (V att ).

[0026] Further, in step S1, the gated forward network FGFN is established by the following formula:

[0027] Given the input tensor FGFN can be represented by the following formula:

[0028]

[0029]

[0030]

[0031]

[0032] Further, in step S3, for the decoder part, a three-layer FGFN module is used for processing, each layer of which is embedded with a frequency domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level features F0 through twice upsampling.

[0033] Further, in step S4, residual connections are added between the two FERMs and their decoding modules respectively, and the refined features are convolved to generate residual images I res ∈R H×W×C , which are added to the dirty images to obtain the recovered images:

[0034] The present application has the following advantages:

[0035] The present application proposes a method for eliminating low-frequency SKA wideband effects and synthetic beam effects based on deep learning, which adopts an end-to-end image-to-image manner, and through large-scale model training, the model can fully learn the corresponding feature relationship between the clean image of celestial structure and the dirty image containing the wideband effect and synthetic beam effect, so as to more efficiently and thoroughly eliminate the coupling effects.

[0036] The application proposes an improved Transform model based on a frequency domain self-attention mechanism, and successfully applies it to the elimination of low-frequency SKA broadband effects and synthetic beam effects, providing a new perspective and effective solution for the restoration and reconstruction of radio astronomy images; in the improved Transform model, the application uses a feature extraction residual module (FERM) to effectively extract the features of the radio astronomy image, ensuring that spatial information is not lost; finally, the application designs a new frequency domain-based gated feedforward network (FGFN) that controls the forward transmission of information with complementary characteristics while retaining useful high-frequency and low-frequency information.

[0037] The effect elimination method based on deep learning can more effectively eliminate the broadband effect and the synthetic beam effect, and restore and reconstruct the original sky brightness to a greater extent; it is time-consuming, and through large-scale training of the model, the model can learn the corresponding features sufficiently, which can greatly reduce the time of effect elimination; it is simple to operate, and the traditional effect elimination method needs to manually design the model and is extremely dependent on parameter tuning, while the method of the patent can greatly simplify the manual operation process of effect elimination. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the application, form a part of the application and illustrate the illustrative embodiments of the application and together with the description serve to explain the application. In the drawings:

[0039] Figure 1 A flowchart of a deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination provided by the application;

[0040] Figure 2 An implementation flowchart of the frequency domain self-attention solver FSAS;

[0041] Figure 3 An implementation flowchart of the frequency domain gated feedforward network FGFN;

[0042] Figure 4 An implementation flowchart of the feature extraction residual module FERM;

[0043] Figure 5 A general design diagram of the model diagram + FSAS + FGFN + FERM. DETAILED DESCRIPTION

[0044] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] Reference Figures 1 to 5As shown, the present application provides a deep learning method for low-frequency SKA broadband effect and synthetic beam effect elimination, comprising the steps of:

[0046] S1: Establishing a frequency domain self-attention solver FSAS, a feature extraction residual module FERM, and a frequency domain gated forward network FGFN; the feature extraction residual module FERM is composed of one convolutional layer and four residual blocks;

[0047] S2: Based on the FSAS, FERM, and FGFN established in step S1, an IFS-Transformer network model is established;

[0048] S3, applying two FERM modules to obtain the low-level features F0 of the dirty image I H×W×C ; H×W×C ;

[0049] S4, inputting the low-level features F0 into the FGFN module, and completing the encoding part through twice downsampling operations;

[0050] S5, passing the features obtained from the encoding part through three FSAS and twice upsampling to realize feature learning and decoding operation;

[0051] S6, after processing by two FERM decoding modules, the final recovered image is obtained.

[0052] Regarding the IFS-Transformer network model:

[0053] From the perspective of computer vision, the elimination of low-frequency SKA observation effect can be regarded as an image deblurring problem. Based on the successful promotion of Transformer in the field of computer vision, the present patent designs an efficient IFS-Transformer model to jointly eliminate the low-frequency SKA broadband effect and synthetic beam effect. On this basis, an effective feature extraction residual module (FERM) is first used, and residual connection is added, effectively alleviating the influence of long-distance dependent features being truncated caused by the input image being divided into image blocks, thereby ensuring the accuracy and efficiency of image restoration. In order to obtain a network architecture more suitable for radio astronomy image deblurring, the feedforward network is improved, and a new frequency domain-based gated feedforward network (FGFN) is proposed, which can preserve useful frequency components while transmitting features with complementary characteristics.

[0054] Regarding the feature extraction residual module (FERM):

[0055] Standard visual Transformer models, inspired by the successful application of Transformer in natural language processing, typically split the input image into image patches and then arrange these patches linearly into a sequence as input to the Transformer. This processing of the input image means that some features with long-range dependencies are destroyed, and the model only notices features with short-range dependencies within each image patch. This can lead to the loss of spatial pixel correlations in the input image.

[0056] To address the above issues, a Feature Extraction Residual Module (FERM) is used in the standard visual Transformer model. This module consists of one convolutional layer and four residual blocks, effectively capturing features with long-range dependencies in the input image without losing spatial information. The detailed network architecture of FERM is as follows... Figure 4 As shown.

[0057] As a preferred embodiment, in step S1, the frequency domain self-attention solver FSAS is established using the following formula:

[0058] For dirty images I∈R H×W×C Standard visual Transformers typically apply a linear transformation L to I. q L k and L v Obtain feature F q F k and F v By analyzing feature F q F k and F v Extract image patches and (n represents the number of image patches) to obtain query Q, key K, and value V:

[0059]

[0060] Where R is an integer function, which makes the shape so that... H p and W p This represents the height and width of the extracted patch. The scaled dot product attention can be obtained using the following formula:

[0061]

[0062] Among them, QK T Each element is obtained through the inner product:

[0063]

[0064] Where q i and kj is from F q and F k The vectorized form of the i-th and j-th patches of F i and all patches k j Then, QK T can be obtained by a convolution operation on all elements of the i-th column of Q

[0065] The detailed module design is as follows. First, the features F q , F k and F v are obtained by 1x1 convolution and 3x3 convolution, respectively. Then, F q and F k are subjected to fast Fourier transform, and the correlation of F q and F k in the frequency domain is estimated by the following formula:

[0066]

[0067] where and denote the Fourier transform and its inverse, respectively, and - denotes the conjugate transpose operation. A is normalized by the layer norm to estimate the aggregated feature:

[0068]

[0069] Finally, the output feature of FSAS is obtained by the following formula:

[0070] I att = I + Conv 1×1 (V att )

[0071] where Conv 1×1 denotes 1x1 convolution. The detailed network architecture of FSAS is shown in Figure 2 .

[0072] As a preferred, in step S1, the gated feedforward network FGFN is established by the following formula:

[0073] The feedforward network, as the backbone part of the Transformer model, promotes the reconstruction of clear images by scaling the dot product attention to enhance the learning and transmission of features. The standard feedforward network usually uses two 1x1 convolutions, one to expand the feature channels and the other to restore the feature channels to the original input dimension. Unlike DFFN, the proposed frequency domain-based gated feedforward network (FGFN) can adaptively preserve useful frequency information while allowing each layer to focus on the fine details complementary to other layers.

[0074] Specifically, a gating mechanism is added in the feed-forward network, which is represented as the element-wise multiplication of two parallel paths of the linear transformation layer at the mathematical expression level, one of which is activated by GELU nonlinearity. In addition, a deep convolution is also included in the FGFN to encode the spatially adjacent pixel position information, which is very useful for learning the local image structure for effective recovery. Given the input tensor The FGFN can be represented by the following formula:

[0075]

[0076]

[0077]

[0078]

[0079] where, and denote the Fourier transform and its inverse, respectively, and denotes the element-wise multiplication, denotes the GELU nonlinearity, is a 1x1 point convolution, is a 3x3 deep convolution, is layer normalization. The detailed network architecture of the FGFN is shown in Figure 3 .

[0080] As a preferred, in step S3, for the decoder part, a three-layer FGFN module is used for processing, each layer of which is embedded with a frequency domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level feature F0 through two upsampling.

[0081] As a preferred, in step S4, residual connections are added between the two FERM and its decoding modules, respectively, and the refined features are convolved to generate a residual image I res ∈R H×W×C , which is added to the dirty image to obtain the recovered image:

[0082] Regarding the IFS-Transformer network structure:

[0083] As shown in Figure 5 , the main framework of the IFS-Transformer model adopts an asymmetric encoder-decoder structure. For a dirty image I∈R H×W×C containing effects, the IFS-Transformer model first applies two FERM models to obtain the low-level features of the dirty image:

[0084] F0∈R H×W×C

[0085] where H x W represents the spatial dimension, and C is the number of channels.

[0086] Then the low-level feature F0is input into FGFN and the encoding part is completed after two down-sampling operations; in the decoder part, three layers of FGFN are used, each of which is embedded with a frequency domain-based self-attention solver (FSAS), and the output is restored to the same number of channels as the low-level feature F0after two up-sampling operations.

[0087] Finally, the final recovered image is obtained after two FERM decoding modules. Among them, the low-level feature F0is finally obtained after the above asymmetric encoder-decoder structure Deep feature F1∈R H×W×C . The encoder feature is concatenated with the decoder feature through a skip connection, and a convolution is performed after the concatenation operation, which reduces the number of all concatenated channels by half.

[0088] In addition, residual connections are added between the two FERMs and their decoding modules, respectively, and the refined features are convolved to generate residual images I res ∈R H×W×C , which are added to the dirty image to obtain the recovered image:

[0089] wherein, Figures 2 to 5 The explanations of various marks used in the above are shown in Table 1:

[0090] Table 1: Various icons in the figure and their corresponding annotations

[0091]

[0092] The above is only the preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning method for eliminating broadband effects and synthetic beamforming effects in low-frequency SKA, characterized in that, Including the following steps: S1: Establish a frequency domain self-attention solver FSAS, a feature extraction residual module FERM, and a frequency domain gated feedforward network FGFN; the feature extraction residual module FERM consists of one convolutional layer and four residual blocks; The frequency domain self-attention solver FSAS is established using the following formula: By analyzing features , and Extract image patches , and The query Q, key K, and value V are obtained. ； In the formula: R represents the number of image patches; R is an integer function that shapes the image to make it more uniform. , and This indicates the height and width of the extracted patch; C is the number of channels; Scaled dot product attention V att It can be obtained from the following formula: ； Among them, QK T Each element is obtained through the inner product: ； In the formula: and It comes from and The vectorized forms of the i-th and j-th patches; Features are obtained through 1×1 convolution and 3×3 convolution, respectively. , and Then to and Perform a Fast Fourier Transform and estimate using the following formula and Correlation in the frequency domain: ； In the formula: and This represents the Fourier transform and its inverse transform. This represents the conjugate transpose operation; Through level norm (·) Normalize A to estimate aggregation features: ； Finally, the output features of FSAS are obtained using the following formula: ； In the formula: I represents a 1×1 convolution; I represents a dirty image. , where H×W represents the spatial dimension; The gated feedforward network (FGFN) is established using the following formula: Given input tensor FGFN can be represented by the following formula: ； ； ； ； In the formula: and Represents the Fourier transform and its inverse transform; This represents element-wise multiplication. This indicates that GELU is nonlinear. It is a 1×1 point convolution. It is a 3×3 depthwise convolution. (·) represents layer normalization; S2, Based on the FSAS, FERM, and FGFN established in step S1, build the IFS-Transformer network model; S3, Apply the two FERM modules to obtain the dirty image. low-level features ; S4, the low-level feature F0 is input into the FGFN module, and the encoding part is completed after two downsampling operations; S5, the features obtained from the encoding part are processed by 3 FSAS and 2 upsampling to realize feature learning and decoding operations; S6, after being processed by two FERM decoding modules, yields the final recovered image.

2. The deep learning method for eliminating broadband effects and synthetic beamforming effects in low-frequency SKA according to claim 1, characterized in that, In step S3, the decoder part is processed using a three-layer FGFN module. Each layer embeds a frequency-domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level feature F0 after two upsampling steps.

3. The deep learning method for eliminating broadband effects and synthetic beamforming effects in low-frequency SKA according to claim 1, characterized in that, In step S4, residual connections are added between the two FERMs and their decoding modules, and the refined features are convolved to generate residual images. Adding this to the dirty image yields the restored image: .

Images