Deep learning method for eliminating low-frequency SKA broadband effect and synthetic beam effect

Abstract

The invention discloses a deep learning method for eliminating a low-frequency SKA broadband effect and a synthetic beam effect, and belongs to the field of radio astronomical image processing, and the method comprises the steps: S1, building an FSAS, an FERM and an FGFN; s2, establishing an IFS-Transform network model based on the FSAS, the FERM and the FGFN established in the step S1; s3, using two FERM modules to obtain low-level features F0 of the dirty image I belonging to RH * W * C, wherein the low-level features F0 belong to RH * W * C; s4, inputting the F0 into an FGFN module, and completing a coding part through two times of down-sampling operation; s5, performing three FSAS and two times of up-sampling on the features obtained by the coding part to realize feature learning and decoding operation; and S6, after two FERM processing, a final restored image is obtained. According to the effect elimination method based on deep learning, the broadband effect and the synthetic beam effect can be more effectively and jointly eliminated, the original sky brightness can be recovered and reconstructed to a greater extent, and the effect elimination time can be greatly shortened; according to the method, the manual operation process of effect elimination can be greatly simplified by adopting the deep learning technology.

Description

Technical Field

[0001] The present invention relates to the technical field of radio astronomy image processing, and in particular to a deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA. Background Art

[0002] In radio astronomy, the Square Kilometer Array (SKA) can greatly help humans explore and understand the universe with its ultra-high sensitivity, resolution and ultra-fast measurement speed. However, when using SKA for broadband observation imaging in the low-frequency band, broadband effects and synthetic beam effects are usually triggered, resulting in distortion and blurring of the imaged celestial structure.

[0003] The broadband effect mainly occurs when broadband imaging of the real sky brightness is performed; the limited bandwidth will cause the visibility function data to tail in the radial direction, and the degree of radial tailing is positively correlated with the bandwidth. The essence of the synthetic beam effect is that the incomplete sampling in the Fourier plane causes the point spread function of the telescope to have side lobes of varying degrees, resulting in image blur when reconstructing the sky brightness. The coupling of the broadband effect and the synthetic beam effect causes more serious damage to the sky brightness after observation, and the degree of damage is affected by multiple factors such as observation frequency, observation duration, bandwidth, and field of view.

[0004] The traditional method is used to jointly eliminate the broadband effect and synthetic beam effect of low-frequency SKA, which needs to be carried out in stages. First, the MFS method is used to eliminate the broadband effect, and then the CLEAN algorithm is used to eliminate the synthetic beam effect. The elimination of different effects requires manual design of models, and the parameters need to be constantly debugged to complete the elimination of the effects. This effect elimination method is not only inefficient, but also cannot completely eliminate the broadband effect and synthetic beam effect, and cannot accurately restore and reconstruct the sky brightness in the image. Summary of the invention

[0005] In order to solve the technical problems that the traditional methods in the above-mentioned prior art are inefficient in jointly eliminating the broadband effect and synthetic beam effect of low-frequency SKA, and cannot completely eliminate the broadband effect and synthetic beam effect, the present invention provides a deep learning method for eliminating the broadband effect and synthetic beam effect of low-frequency SKA.

[0006] To implement the above technical solution, the present invention provides a deep learning method for eliminating the broadband effect and synthetic beam effect of low-frequency SKA, comprising the steps of:

[0007] S1: Establish a frequency domain self-attention solver FSAS, a feature extraction residual module FERM and a frequency domain gated forward network FGFN; the feature extraction residual module FERM consists of a convolutional layer and four residual blocks;

[0008] S2: Establish an IFS-Transformer network model based on the FSAS, FERM and FGFN established in step S1;

[0009] S3, apply two FERM modules to obtain the dirty image I∈R H×W×C The low-level features F0∈R H×W×C ;

[0010] S4, inputting the low-level feature F0 into the FGFN module, and completing the encoding part after two downsampling operations;

[0011] S5, the features obtained from the encoding part are subjected to 3 FSAS and two upsampling to achieve feature learning and decoding operations;

[0012] S6, after being processed by two FERM decoding modules, the final restored image is obtained.

[0013] Furthermore, in step S1, the frequency domain self-attention solver FSAS is established by the following formula:

[0014] By the feature F q , F k and F v Extract image blocks and Get the query Q, key K and value V:

[0015]

[0016] The scaled dot product attention can be obtained as follows:

[0017]

[0018] Among them, QK T Each element of is obtained by inner product:

[0019] (QK T ) ij = i ,k j >

[0020] The features F are obtained by 1×1 convolution and 3×3 convolution respectively. q , F k and F v , then for F q and F k Perform fast Fourier transform and estimate F by the following formula q and F k Correlation in the frequency domain:

[0021] ​

[0022] By layer norm Normalize A to estimate the aggregate features:

[0023]

[0024] Finally, the output characteristics of FSAS are obtained by the following formula:

[0025] I att =I+Conv 1×1 (V att ). Further, in step S1, the gated forward network FGFN is established by the following formula:

[0026] Given an input tensor FGFN can be expressed by the following formula:

[0027]

[0028]

[0029] Furthermore, in step S3, for the decoder part, a three-layer FGFN module is used for processing, each layer is embedded with a frequency domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level feature F0 after two upsampling.

[0030] Furthermore, in step S4, residual connections are added between the two FERMs and their decoding modules, and the refined features are convolved to generate a residual image I res ∈R H×W×C , added to the dirty image, we get the restored image:

[0031] The beneficial effects of the present invention are:

[0032] The present invention proposes a method for eliminating the low-frequency SKA broadband effect and synthetic beam effect based on deep learning. By adopting an end-to-end image-to-image approach and through large-scale model training, the model can fully learn the corresponding feature relationship between the clean image of the celestial body structure and the dirty image containing the broadband effect and the synthetic beam effect, thereby eliminating the coupling effect more efficiently and thoroughly.

[0033] The present invention proposes an improved Transformer model based on the frequency domain self-attention mechanism, and successfully applies it to the elimination of the low-frequency SKA broadband effect and synthetic beam effect, providing a new perspective and effective solution for the restoration and reconstruction of radio astronomical images; in the improved Transformer model, the present invention uses a feature extraction residual module (FERM) to effectively extract the features of the radio astronomical image to ensure that spatial information is not lost; finally, the present invention designs a new frequency domain-based gated feedforward network (FGFN), which controls the forward transmission of information with complementary characteristics while retaining useful high-frequency and low-frequency information;

[0034] The effect elimination method based on deep learning of the present invention can more effectively jointly eliminate the broadband effect and the synthetic beam effect, and restore and reconstruct the original sky brightness to a greater extent; it is time-saving, and by training the model on a large scale, the model can fully learn the corresponding features, which can greatly reduce the time for effect elimination; it is simple to operate, and the traditional effect elimination method requires manual model design and is extremely dependent on parameter adjustment, while the patented method uses deep learning technology to greatly simplify the manual operation process of effect elimination. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 A flowchart of a deep learning method for eliminating low-frequency SKA broadband effect and synthetic beam effect provided by the present invention;

[0037] Figure 2 This is the implementation flow chart of the frequency domain self-attention solver FSAS;

[0038] Figure 3 It is the implementation flow chart of the frequency domain gated forward network FGFN;

[0039] Figure 4 It is the implementation flow chart of the feature extraction residual module FERM;

[0040] Figure 5 It is the overall design diagram of model diagram + FSAS + FGFN + FERM. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] See also Figures 1 to 5As shown, the present invention provides a deep learning method for eliminating the broadband effect and synthetic beam effect of low-frequency SKA, comprising the steps of:

[0043] S1: Establish a frequency domain self-attention solver FSAS, a feature extraction residual module FERM and a frequency domain gated forward network FGFN; the feature extraction residual module FERM consists of a convolutional layer and four residual blocks;

[0044] S2: Establish an IFS-Transformer network model based on the FSAS, FERM and FGFN established in step S1;

[0046] S3, apply two FERM modules to obtain the dirty image I∈R H×W×C The low-level features F0∈R H×W×C ;

[0047] S4, inputting the low-level feature F0 into the FGFN module, and completing the encoding part after two downsampling operations;

[0049] S5, the features obtained from the encoding part are subjected to 3 FSAS and two upsampling to achieve feature learning and decoding operations;

[0051] S6, after being processed by two FERM decoding modules, the final restored image is obtained.

[0052] About the IFS-Transformer network model:

[0053] From the perspective of computer vision, the elimination of low-frequency SKA observation effects can be regarded as an image deblurring problem. Based on the successful promotion of Transformer in the field of computer vision, this patent designs an efficient IFS-Transformer model to jointly eliminate the low-frequency SKA broadband effect and synthetic beam effect. On this basis, an effective feature extraction residual module (FERM) is first used, and residual connections are added to effectively alleviate the impact of the truncation of long-distance dependent features caused by the division of the input image into image blocks, thereby ensuring the accuracy and efficiency of image restoration. In order to obtain a network architecture that is more suitable for radio astronomy image deblurring, the feedforward network is improved, and a new frequency-domain-based gated feedforward network (FGFN) is proposed, which can transfer features with complementary characteristics while retaining useful frequency components.

[0054] About Feature Extraction Residual Module (FERM):

[0055] The standard visual Transformer model is inspired by the successful application of Transformer in natural language processing. It usually splits the input image into image blocks and then linearly arranges the image blocks into a sequence as the input of the Transformer. The above processing of the input image means that some features with long-distance dependencies are destroyed, and the model only pays attention to the features of short-distance dependencies on each image block, which may cause the spatial pixel correlation of the input image to be lost.

[0056] To solve the above problems, a feature extraction residual module (FERM) is used in the standard visual Transformer model. This module consists of a convolutional layer and four residual blocks, which can effectively capture the features with long-distance dependencies in the input image without causing the loss of image spatial information. The detailed network architecture of FERM is shown in Figure 4 shown.

[0057] As a preferred embodiment, in step S1, the frequency domain self-attention solver FSAS is established by the following formula:

[0058] For a dirty image I∈R H×W×C , standard visual Transformers usually apply a linear transformation L to I q , L k and L v Get feature F q , F k and F v By considering the feature F q , F k and F v Extract image blocks and (n represents the number of image blocks) to obtain the query Q, key K and value V:

[0059]

[0060] Where R is a shaping function, which is transformed into H p and W p represents the height and width of the extracted patch. The scaled dot product attention can be obtained by:

[0061]

[0062] Among them, QK T Each element of is obtained by inner product:

[0063] (QK T ) ij = i ,k j >​

[0064] where q i and k j It is from F q and F k The vectorized form of the i-th and j-th patches of . Apply the shaping function to q respectively i and all patches k j , then QK can be obtained through convolution operation T All elements in the i-th column of .

[0065] The specific module design is as follows: First, the feature F is obtained by 1×1 convolution and 3×3 convolution respectively. q , F k and F v Then for F q and F k Perform fast Fourier transform and estimate F by the following formula q and F k Correlation in the frequency domain:

[0066]

[0067] in and represents Fourier transform and its inverse transform, represents the conjugate transpose operation. Normalize A to estimate the aggregate features:

[0068]

[0069] Finally, the output characteristics of FSAS are obtained by the following formula:

[0070] I att =I+Conv 1×1 (V att )

[0071] Among them, Conv 1×1 represents a 1×1 convolution. The detailed network architecture of FSAS is as follows Figure 2 shown.

[0072] As a preferred embodiment, in step S1, the gated forward network FGFN is established by the following formula:

[0073] As the backbone of the Transformer model, the feedforward network scales the dot product attention to enhance feature learning and transfer, facilitating the reconstruction of clear images. Standard feedforward networks usually use two 1×1 convolutions, one to expand the feature channel and the other to restore the feature channel to the original input dimension. Different from DFFN, the proposed frequency-based gated feedforward network (FGFN) can adaptively retain useful frequency information while allowing each layer to focus on fine details that complement other layers.

[0074] Specifically, a gating mechanism is added to the feedforward network, which is mathematically represented as the element-by-element product of two parallel paths of the linear transformation layer, one of which is activated by the GELU nonlinearity. In addition, FGFN also includes deep convolution to encode spatially adjacent pixel position information, which is very useful for learning local image structures for effective restoration. Given an input tensor FGFN can be expressed by the following formula:

[0075]

[0076] in, and denote Fourier transform and its inverse transform respectively, ⊙ denotes element-by-element multiplication, represents GELU nonlinearity, is a 1×1 point convolution, is a 3×3 depthwise convolution, is the layer normalization. The detailed network architecture of FGFN is as follows Figure 3 shown.

[0077] As a preferred embodiment, in step S3, for the decoder part, a three-layer FGFN module is used for processing, each layer is embedded with a frequency domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level feature F0 after two upsampling.

[0078] As a preferred embodiment, in step S4, residual connections are added between the two FERMs and their decoding modules, and the refined features are convolved to generate a residual image I res ∈R H×W×C , added to the dirty image, we get the restored image:

[0079] About IFS-Transformer network structure:

[0080] like Figure 5 As shown in the figure, the main framework of the IFS-Transformer model adopts an asymmetric encoder-decoder structure. For the dirty image I∈R containing the effect H×W×C,The IFS-Transformer model first applies two FERM models to obtain the low-level features of the dirty image:

[0081] F0∈R H×W×c

[0082] Where H×W represents the spatial dimension and C is the number of channels.

[0083] The low-level feature F0 is then input into FGFN, and the encoding part is completed after two downsampling operations; in the decoder part, three layers of FGFN are used, each layer is embedded with a frequency domain-based self-attention solver (FSAS), and the output is restored to the same number of channels as the low-level feature F0 after two upsampling operations.

[0084] Finally, the final restored image is obtained after two FERM decoding modules. Among them, the low-level feature F0 is finally obtained through the above asymmetric encoder-decoder structure to obtain the deep feature F1∈R H×W×C The encoder features are concatenated with the decoder features via skip connections, and the concatenation operation is followed by a convolution, which reduces the number of channels after all concatenation by half.

[0085] In addition, residual connections are added between the two FERMs and their decoding modules to convolve the refined features and generate the residual image I res ∈R H×W×C , added to the dirty image, we get the restored image:

[0086] in, Figures 2 to 5 The explanations of the various marks used in are shown in Table 1:

[0087] Table 1 Various icons and their corresponding annotations in the figure

[0088]

[0089] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA, characterized in that: Includes steps: S1: Establish a frequency domain self-attention solver FSAS, a feature extraction residual module FERM and a frequency domain gated forward network FGFN; the feature extraction residual module FERM consists of a convolutional layer and four residual blocks; S2: Establish an IFS-Transformer network model based on the FSAS, FERM and FGFN established in step S1; S3, apply two FERM modules to obtain the dirty image I∈R H×W×C The low-level features F0∈R H×W×C ; S4, the low-level feature F0 is input into the FGFN module, and the encoding part is completed after two downsampling operations. point; S5, the features obtained from the encoding part are subjected to 3 FSAS and two upsampling to achieve feature learning and decoding operations; S6, after being processed by two FERM decoding modules, the final restored image is obtained.

2. The deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA according to claim 1, characterized in that: In step S1, the frequency domain self-attention solver FSAS is established by the following formula: By the feature F q , F k and F v Extract image patches and Get the query Q, key K and value V: The scaled dot product attention can be obtained as follows: Among them, QK T Each element of is obtained by inner product: The features F are obtained by 1×1 convolution and 3×3 convolution respectively. q , F k and F v , then for F q and F k Perform fast Fourier transform and estimate F by the following formula q and F k Correlation in the frequency domain: By layer norm Normalize A to estimate the aggregate features: Finally, the output characteristics of FSAS are obtained by the following formula: I att =I+Conv 1×1 (V att )。 3. The deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA according to claim 1, characterized in that: In step S1, the gated forward network FGFN is established by the following formula: Given an input tensor FGFN can be expressed by the following formula:

4. The deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA according to claim 1, characterized in that: In step S3, for the decoder part, a three-layer FGFN module is used for processing, each layer is embedded with a frequency domain-based self-attention solver FSAS, and the output is restored to the same number of channels as the low-level feature F0 after two upsampling.

5. The deep learning method for eliminating broadband effect and synthetic beam effect of low-frequency SKA according to claim 1, characterized in that: In step S4, residual connections are added between the two FERMs and their decoding modules, and the refined features are convolved to generate a residual image I res ∈R H×W×C , added to the dirty image, we get the restored image:

Images