Amplitude and phase channel attention structural light illumination microscope reconstruction method

Abstract

The present application belongs to the technical field of structured light illumination super-resolution microscopic imaging, and particularly relates to a structured light illumination microscope reconstruction method with amplitude-phase channel attention, comprising the following steps: obtaining original data of a structured light microscope; if the original data is original data of multiple time points, calculating feature maps of the original data; extracting low-frequency features in the feature maps of the original data through a convolution layer; taking the low-resolution features as inputs of four cascaded residual groups to obtain high-frequency features in the feature maps of the original data; and performing image reconstruction according to the high-frequency features and the low-frequency features to obtain a super-resolution image; compared with the scUNet and DFCAN methods, the present application improves the reconstruction fidelity by 17%.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of structured illumination microscopy, and particularly relates to a structured illumination microscopy reconstruction method with amplitude-phase channel attention. BACKGROUND

[0002] In recent years, great breakthroughs have been made in the super-resolution (SR) reconstruction algorithm of structured illumination microscopy (SIM), which can solve subcellular events such as the assembly of clathrin and caveolin3, the dynamics of mitochondria, and the Xist RNA spreading without specific fluorescent probes. However, artifacts are a destructive factor that interferes with the quality of the reconstructed image in SR-SIM. For the traditional Wiener reconstruction algorithm, it is usually necessary to have sufficient signal-to-noise ratio (SNR) of the original image to distinguish the true SR signal from the artifacts. However, a higher SNR means a stronger laser intensity or a longer exposure time, resulting in greater phototoxicity and faster photobleaching. In order to solve the reconstruction artifacts of low SNR images, the traditional method is usually combined with some prior information, i.e., handcrafted continuity, to improve the quality of the reconstruction. However, these handcrafted assumptions are usually limited to suppressing artifacts and may bring loss of resolution.

[0003] Due to the adaptive feature extraction and fitting ability of neural networks, some deep learning-based SR methods have been introduced into the reconstruction of SIM. These neural networks usually focus on the SR information in the spatial domain. However, based on the imaging model, the SR information of the SIM image benefits from the expansion in the frequency domain, which prompts the network to pay more attention to the frequency information. Therefore, the previously proposed deep Fourier channel attention network (DFCAN) considers the amplitude spectrum in the attention mechanism and achieves the most realistic performance. However, it does not use the phase spectrum information, which is also important for the image restoration task. In addition, each frequency in the frequency domain is determined by the amplitude and phase. Therefore, using the amplitude or phase alone cannot fully represent the information in the frequency domain. SUMMARY

[0004] The present application is directed to the defects and deficiencies of the prior art, and proposes a structured illumination microscopy reconstruction method with amplitude-phase channel attention, comprising the following steps:

[0005] Obtaining the original data of the structured illumination microscopy, if the original data is the original data of multiple time points, calculating the feature map of the original data;

[0006] Extracting the low-frequency features in the feature map of the original data through a convolution layer;

[0007] Taking the low-resolution features as the input of four cascaded residual groups to obtain the high-frequency features in the feature map of the original data;

[0008] According to the high-frequency feature and the low-frequency feature, image reconstruction is performed to obtain a super-resolution image.

[0009] Further, the feature map of the original data includes the following steps:

[0010] The original data of each time point includes three directions and three phases, and the image is extracted into a stack of 27 frames. Each direction and phase stack is processed by a 3x3 convolution layer to obtain a feature map.

[0011] The superposition of the three directions and the three phases can output 9 frames of feature maps.

[0012] The 9 frames of feature maps are spliced together as the feature map of the original data.

[0013] Further, the process of obtaining the feature map of the original data includes:

[0014]

[0015] F=Cat(f 1,1 ,...,f 3,3 );

[0016] Wherein, F represents the feature map of the original data; represents the feature map of the direction d and the phase , Cat() represents the splicing operation, Conv 3×3 () represents a 3x3 convolution operation, represents the image of the tth time point, direction d and phase

[0017] Further, each residual group includes an amplitude-phase channel attention module, and the amplitude-phase channel attention module includes an amplitude spectrum attention module, a phase spectrum attention module and an adaptive weight fusion module. The high-frequency feature in the feature map of the original data includes the following steps:

[0018] The feature f input into the amplitude-phase channel attention module is extracted by two cascaded 3x3 convolution layers to obtain the feature f;

[0019] The amplitude spectrum attention is extracted from the feature f by the amplitude spectrum attention module;

[0020] The phase spectrum attention is extracted from the feature f by the phase spectrum attention module;

[0021] The importance of the amplitude spectrum attention module and the phase spectrum attention module is learned by the adaptive weight fusion module to obtain the importance of the two modules respectively.

[0022] The outputs of the amplitude spectrum attention module and the phase spectrum attention module are weighted according to the importance obtained, to obtain an output of the amplitude-phase channel attention module, and the output of the last-stage amplitude-phase channel attention module is the high-frequency feature.

[0023] Further, the output of the amplitude-phase channel attention module is represented as:

[0024]

[0025] wherein APCAB out is the output of the amplitude-phase channel attention module; z is a feature input into the amplitude-phase channel attention module; Conv 1×1 () is a 1x1 convolution operation; a represents the importance of the output ASAM out of the amplitude spectrum attention module; and β represents the importance of the output PSAM out of the phase spectrum attention module. represents element-wise multiplication.

[0026] Further, the output ASAM out of the amplitude spectrum attention module is represented as:

[0027] ASAM out = f x S(Conv 3×3 (Cat(Pooling avg (A_Att in ), Pooling max (A_Att in )))).

[0028] A_Att in = ReLU(Conv 3×3 (abs(FFT(f)) r )).

[0029] wherein Pooling max () represents a max-pooling operation; Pooling avg () represents an average-pooling operation; Cat() represents a concatenation operation; Conv 3×3 () represents a 3x3 convolution operation; S() represents a sigmoid activation function; ReLU() represents a ReLU activation function; abs() r represents an absolute value, and the superscript r is an index used to enhance high-frequency components; and FFT() represents a Fourier transform.

[0030] Further, the output PSAM out of the phase spectrum attention module is represented as:

[0031] PSAM out=f×S(Conv 3×3 (Cat(Pooling avg (P_Att in Pooling max (P_Att in ))))

[0032] P_Att in =ReLU(Conv 3×3 (arctan(imag(FFT(f)+ε),real(FFT(f)+ε))

[0033] Pooling max () indicates a max pooling operation; Pooling avg () indicates average pooling; Cat() indicates concatenation operation, Conv 3×3 () denotes a 3×3 convolution operation; S() denotes the sigmoid activation function; ReLU() denotes the ReLU activation function; imag() denotes the imaginary part of the parameters, and real() denotes the real part of the parameters; FFT() denotes the Fourier transform; ε is a constant.

[0034] Furthermore, the output ASAM of the amplitude spectrum attention module out The importance or output of the phase spectrum attention module PSAM out The importance is expressed as:

[0035] α,β=Softmax(Conv 1×1 (ReLU(Conv 1×1 (Pooling avg (f))));

[0036] Where Softmax() represents the Softmax function; Conv 1×1 () represents a 1×1 convolution operation; ReLU() represents the ReLU activation function; Pooling avg () indicates the average pooling operation.

[0037] Furthermore, the output ASAM of the amplitude spectrum attention module out The importance or output of the phase spectrum attention module PSAM out The importance is determined by the network's adaptive learning, and the network's loss function is expressed as:

[0038]

[0039] in, Represents the true label Y and the predicted label Loss between the loss; W, H are the width, height of the output image respectively; Lambda is a scalar weight; represents the pixel value at (i,j) in the predicted label represents the pixel value at (i,j) in the predicted label i,j represents the pixel value at (i,j) in the real label Y. represents the structural similarity loss between the real label Y and the predicted label .

[0040] Further, according to the high-frequency feature and the low-frequency feature, the image is reconstructed to obtain a super-resolution image, comprising:

[0041] SR=Conv 3×3 (Up(LR f +HR f ));

[0042] Wherein, SR represents a single gray super-resolution image; Conv 3×3 () represents a 3*3 convolution operation; Up() represents an up-sampling operation; LR f is a low-frequency feature, and HR f is a high-frequency feature.

[0043] The experimental results of the method of the application on different data sets show that, only considering using amplitude and phase information, APCAN can infer SR images of different biological structures, and compared with the most advanced scUNet and DFCAN method, the fidelity of the reconstruction is improved by 17%. In order to consider the time information in the APCAN network, the video data set with multiple time points is used for training, and the experiment shows that the time information of the SIM raw data can indeed reduce the reconstruction artifacts, compared with the traditional reconstruction methods such as Wiener and HiFi-SIM, APCAN can effectively suppress the influence of noise on the reconstruction result, the artifacts are minimized, and the most faithful result can be reconstructed. Compared with DFCAN, the error of APCAN reconstruction is reduced by 38%, and it has great resistance to noise and light bleaching. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the network architecture diagram of the amplitude-phase channel attention module of the application;

[0045] Figure 2 is the extraction and reconstruction process of the multiple time point SIM raw data of the application;

[0046] Figure 3 is the amplitude-phase channel attention module structure diagram of the application. DETAILED DESCRIPTION

[0047] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0048] The application provides a structured light illumination microscope reconstruction method with amplitude-phase channel attention, comprising the following steps:

[0049] Obtaining original data of the structured illumination microscope, if the original data is original data of multiple time points, calculating feature maps of the original data;

[0050] Extracting low-frequency features in the feature maps of the original data through a convolution layer;

[0051] Taking the low-resolution features as inputs of four cascaded residual groups, obtaining high-frequency features in the feature maps of the original data;

[0052] Performing image reconstruction according to the high-frequency features and the low-frequency features to obtain a super-resolution image.

[0053] In order to solve the defect that the amplitude and phase spectrum information is not effectively utilized in the existing deep learning SIM method, the amplitude and phase spectrum information is considered in the network according to the characteristics of SIM, and an amplitude-phase channel attention network (Amplitude-Phase channel attention block, APCAN for short) is designed. The input of the APCAN is SIM original data with three time points. We extract the three-direction and three-phase time point images into a stack of 27 frames, then each direction and phase stack is processed through a 3x3 convolution layer to obtain a feature map. Since the superposition of three directions and three phases can output 9 feature maps, we splice them in the channel dimension to obtain 9 frames of feature maps. This process can be represented as follows:

[0054]

[0055] F=Cat(f 1,1 ,...,f 3,3 ) (2)

[0056] Wherein, represents the image of the tth time point, direction d, and phase ; represents the feature map output by the image of direction d and phase , and F represents the splicing result of 9 frames of feature maps.

[0057] Figure 2 is the extraction and reconstruction process of multiple time point SIM raw data, where the input is SIM raw data of T times continuous shooting for each direction and each phase, after extraction and rearrangement, the original data with T time points is obtained, each time point is SIM raw data of three directions and three phases, then the rearranged data with multiple time points is input into APCAN to obtain super-resolution reconstruction results. In this embodiment, the feature maps of 9 phase and direction combinations under the conditions of direction d = 1, phase direction d = 1, phase direction d = 1, phase direction d = 2, phase direction d = 2, phase direction d = 2, phase direction d = 3, phase direction d = 3, phase and direction d = 3, phase T images of the feature maps under the 9 phase and direction combinations are selected, and each feature map is extracted according to time to form T combinations, each combination including 9 frames of feature maps.

[0058] The obtained 9 frames of feature maps are first subjected to a convolution layer of 3x3 (Conv 3×3 ) to extract shallow features. This operation can be represented as LR f = Conv 3×3 (F), where LR f represents the output low-resolution feature map. Then the output of the convolution layer is followed by four residual groups (RG), each of which is composed of four amplitude-phase channel attention modules (APCAB). RG is connected by a long jump, which can prevent the network from going too deep and causing gradient disappearance, so that the network focuses on learning high-frequency features, thereby reducing the learning burden of the network. The high-frequency information learned by the network can be represented as:

[0059] HR f = Conv 3×3 (RG4(...(RG4(LR f )))) (3)

[0060] RG(x) = x + Conv 3×3 (APCAB4(...(APCAB1(x)))) (4)

[0061] where HR f and x represent the high-frequency feature map and the input of the residual group, respectively.

[0062] APCAB is composed of amplitude spectrum attention module (ASAM), phase spectrum attention module (PSAM) and adaptive weight fusion module (AWFM). The input of APCAB is firstly passed through two 3x3 convolutional layers to further extract features. It can be represented as:

[0063] f = Conv 3×3 (ReLU(Conv 3×3 (z))) (5)

[0064] where z, f and ReLU represent the input of APCAB, the feature maps output by the two 3x3 convolutional layers and the activation function, respectively. The activation function ReLU can be represented as:

[0065] ReLU(x) = max(x, 0) (6)

[0066] The output of amplitude spectrum attention module ASAM out can be represented as follows:

[0067] ASAM out = f x S(Conv 3×3 (Cat(Pooling avg (A_Att in ), Pooling max (A_Att in ))) (7)

[0068] where:

[0069] A_Att in = ReLU(Conv 3×3 (abs(FFT(f)) r )) (8)

[0070] FFT represents fast Fourier transform, and r is used to enhance the contribution of high frequency components; Pooling avg and Pooling max represent global average pooling and global maximum pooling, respectively, so that each feature map of A_Att in can be compressed to a representative value. In order to utilize the results of average and maximum pooling, the two output vectors obtained are spliced and sent to a 3x3 convolutional layer to reduce the parameter amount of the network. S represents the sigmoid activation function. After a series of operations, a gating mechanism is formed, which can adaptively calculate the final scaling factor.

[0071] Similar to the operation of ASAM out , the output of phase spectrum attention module PSAM out is represented as follows:

[0072] PSAM out = f x S(Conv 3×3 ( Pooling avg ( P_Att in ), Pooling max ( P_Att in )))) (9)

[0073] where:

[0074] P_Att in = ReLU(Conv 3×3 (arctan(imag(FFT(f) + ε), real(FFT(f) + ε)) (10)

[0075] where imag and real represent the imaginary and real parts of the FFT result respectively, and ε is used to prevent division by zero.

[0076] To utilize the two results of ASAM out and PSAM out , this embodiment designs AWFM to fuse the two outputs in a weighted manner. Compared with the fusion method of fixed weighting and splicing, AWFM allows the network to automatically learn the importance of the two branches. AWFM is represented as follows:

[0077] α, β = Softmax(Conv 1×1 (ReLU(Conv 1×1 ( Pooling avg (f)))) (11)

[0078] where α, β represent the weight vectors of ASAM and PSAM respectively. The full connection is realized by a 1x1 convolution layer; finally, the output of APCAB can be represented as:

[0079]

[0080] where, represents the multiplication of corresponding elements.

[0081] The last part of the network consists of an up-sampling (Up p ) module and a 3x3 convolution layer, which is used to reconstruct the feature map with high and low frequency features into the corresponding SR image. This process can be represented as:

[0082] SR = Conv 3×3 (Up(LR f + HR f )) (13)

[0083] Here, SR represents a monochrome grayscale super-resolution image.

[0084] Figure 3 The amplitude and phase channel attention module consists of amplitude channel attention branches, phase channel attention branches, and an adaptive weighted fusion module. First, the input feature map is further processed through two 2D convolutional layers to extract features. These features are then fed into the amplitude and phase attention branches to calculate attention weights. The attention weights calculated by each branch are multiplied by the input feature map to obtain two outputs. To enable the network to adaptively learn the importance of the two branches, an adaptive weighted fusion module is designed. This module outputs two weight vectors, α and β, which are then multiplied by the outputs of the phase and amplitude attention branches respectively, allowing the network to adaptively determine the importance of the two branches. Experiments show that the network achieves the best reconstruction quality when β ≥ α, indicating that focusing the network on learning the phase branch improves its reconstruction performance.

[0085] APCAN's loss function consists of two parts: MAE loss and SSIM loss. MAE loss minimizes the pixel-level difference between the network prediction and the ground truth, while SSIM loss improves the structural similarity of the output. In this embodiment, Y is defined as the ground truth. As the output of APCAN, W and H are the width and height of the output image, respectively. The loss function can then be expressed as follows:

[0086]

[0087] Here, λ is a scalar weight used to balance the relative contributions of SSIM and MAE, and is set to 0.84 in this paper.

[0088] This embodiment provides the following: Figure 1 The network architecture diagram of APCAN shown below, in which Figure 1 'a' represents the Amplitude Phase Channel Attention Module (APCAB). Figure 1 b represents a network consisting of multiple residual sets and multiple APCABs. Figure 1 'c' represents the timing information processing module. The network input is SIM raw data with one or more time points. If it is a single time point, it does not need to go through the timing information processing module. If it is multiple time points, a timing information processing module is used to obtain feature maps of 9 frames. Then, the feature maps of 9 frames are input into the APCAN network for super-resolution reconstruction. Finally, APCAN will output a super-resolution image.

[0089] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. An amplitude-phase channel attention structured light illumination microscopy reconstruction method, characterized in that, The method comprises the following steps: Obtaining original data of a structured illumination microscope, if the original data is original data of multiple time points, calculating a feature map of the original data; Extracting low-frequency features in the feature map of the original data through a convolution layer; Taking the low-resolution features as the input of four cascaded residual groups to obtain high-frequency features in the feature map of the original data; each residual group comprises an amplitude-phase channel attention module, the amplitude-phase channel attention module comprises an amplitude spectrum attention module, a phase spectrum attention module and an adaptive weight fusion module, and obtaining the high-frequency features in the feature map of the original data comprises the following steps: Extracting features f from the features input into the amplitude-phase channel attention module through two cascaded 3×3 convolution layers; Extracting amplitude spectrum attention from the features f through the amplitude spectrum attention module; Extracting phase spectrum attention from the features f through the phase spectrum attention module; Learning the importance of the amplitude spectrum attention module and the phase spectrum attention module through the adaptive weight fusion module to obtain the importance of the two modules respectively; According to the obtained importance, weighting the outputs of the amplitude spectrum attention module and the phase spectrum attention module to obtain the output of the amplitude-phase channel attention module, and the output of the last-stage amplitude-phase channel attention module is the high-frequency features, and the output of the amplitude-phase channel attention module is expressed as: wherein, APCAB out is the output of the amplitude-phase channel attention module; z is the feature input to the amplitude-phase channel attention module; Conv 1×1 () is a 1x1 convolution operation; a represents the importance of the output ASAM out of the amplitude spectrum attention module; b represents the importance of the output PSAM out of the phase spectrum attention module; represents the multiplication of corresponding elements. According to the high-frequency features and the low-frequency features, performing image reconstruction to obtain a super-resolution image.

2. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 1, wherein, The calculation of the feature map of the original data comprises the following steps: The original data of each time point comprises three directions and three phases, the image is extracted into 27 frames of stacks, and each stack of direction and phase is processed through a 3×3 convolution layer to obtain a feature map; The superposition of the three directions and the three phases can output 9 frames of feature maps; The 9 frames of feature maps are spliced together as the feature map of the original data.

3. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 2, wherein, The process of obtaining the feature map of the original data comprises: F = Cat(f 1,1 ,...,f 3,3 ) wherein F represents a feature map of the original data; represents a feature map with direction d and phase Cat() represents a concatenation operation, Conv 3×3 () represents a 3x3 convolution operation, represents an image with direction d and phase at the tth time point.

4. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 1, wherein, output of the amplitude spectrum attention module ASAM out is represented as: ASAM out = f x S(Conv 3×3 (Cat(Pooling avg (A_Att in ), Pooling max (A_Att in ))) ; A_Att in = ReLU(Conv 3×3 (abs(FFT(f)) r )) where Pooling max () denotes max-pooling operation; Pooling avg () denotes average-pooling operation; Cat() denotes concatenation operation, Conv 3×3 () denotes 3x3 convolution operation; S() denotes sigmoid activation function; ReLU() denotes ReLU activation function; abs() r denotes taking absolute value, the superscript r is an index used to enhance high frequency components; FFT() denotes Fourier transform.

5. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 1, wherein, Output of the phase spectrum attention module, PSAM out is represented as: PSAM out = f x S(Conv 3×3 (Cat(Pooling avg (P_Att in ), Pooling max (P_Att in )))) P Att in = ReLU(Conv 3×3 (arctan(imag(FFT(f)+ε),real(FFT(f)+ε)) where Pooling max () denotes max-pooling operation; Pooling avg () denotes average-pooling operation; Cat() denotes concatenation operation, Conv 3×3 () denotes 3x3 convolution operation; S() denotes sigmoid activation function; ReLU() denotes ReLU activation function; imag() denotes imaginary part of a parameter, real() denotes real part of a parameter; FFT() denotes Fourier transform; and ε is a constant.

6. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 1, wherein, the output ASAM of the amplitude spectrum attention module out the output PSAM of the phase spectrum attention module out is represented as: a,b = Softmax(Conv 1×1 (ReLU(Conv 1×1 (Pooling avg (f)))) wherein Softmax() represents a Softmax Softmax function; Conv 1×1 () is a 1x1 convolution operation; ReLU() represents a ReLU activation function; Pooling avg () represents an average pooling operation.

7. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 6, wherein, the output ASAM of the amplitude spectrum attention module out the output PSAM of the phase spectrum attention module out The importance of the outputs ASAM and PSAM is determined by network adaptive learning, and the loss function of the network is represented as: where, denotes the loss between the true label Y and the predicted label ; W, H are the width, height of the output image, respectively; λ is a scalar weight; denotes the pixel value at (i,j) in the predicted label Y i,j denotes the pixel value at (i,j) in the true label Y; denotes the structural similarity loss between the true label Y and the predicted label Y 8. The amplitude-phase path-attentive structured light illumination microscopy reconstruction method of claim 1, wherein, According to the high-frequency features and the low-frequency features, performing image reconstruction to obtain a super-resolution image, comprising: SR = Conv 3×3 (Up(LR f + HR f )); where SR denotes a super-resolution image of a single color gray scale; Conv 3×3 denotes a 3x3 convolution operation; Up() denotes an up-sampling operation; LR f denotes a low-frequency feature; and HR f denotes a high-frequency feature.

Images