Single-frame composite structured illumination microscopy method based on ensemble learning

Abstract

The application discloses a single-frame composite structured light illumination microscopic imaging method based on ensemble learning. The application realizes single transverse isotropic spectral modulation through an efficient six-beam interference composite structured illumination strategy, without modifying the optical device of a structured illumination microscope (SIM), thereby reducing the number of original images by 9 times in a physical sense. In addition, the application solves the ill-posed inverse problem of super-resolution reconstruction by integrating the sub-networks of different frequency component features in different domains through ensemble learning, and realizes higher quality super-resolution image reconstruction at 9 times the imaging speed of the traditional illumination mode in a low signal-to-noise ratio environment. Faster imaging speed, less light damage, higher reconstruction quality and better noise resistance make the application an efficient and powerful imaging tool, which can be used for studying the dynamics of nanoscale fine structures in living cells.

Description

Technical Field

[0001] This invention belongs to the field of super-resolution fluorescence microscopy technology, specifically a single-frame composite structure illumination microscopy method based on ensemble learning (eDL-cSIM) for super-resolution observation at the subcellular scale. Background Technology

[0002] Fluorescence microscopy is of great significance to life sciences, but its spatial resolution is limited to 200 nm due to the diffraction limit. In the past few decades, a variety of fluorescence super-resolution methods have been developed, such as single-molecule localization microscopy (PALM / STORM), structured illumination microscopy (SIM), and stimulated emission depletion microscopy (STED). SIM stands out among various fluorescence super-resolution techniques due to its low phototoxicity and photobleaching properties. The Simulation Image (SIM) technique uses periodic fringes in three directions, each with three phases, to modulate the high-frequency information of the sample into the cutoff frequency of the optical system. Reconstruction is then performed in the frequency domain followed by an inverse Fourier transform to obtain a super-resolution image. However, the principle of SIM determines the complexity of illumination parameter estimation and its relatively low temporal resolution. To address this issue, a parameter estimation method based on principal component analysis (PCA-SIM) was proposed [Qian, Jiaming, et al. "Structured illumination microscopy based on principal component analysis." eLight 3.1(2023):4.]. This method can achieve fast and accurate non-iterative parameter estimation, but it does not improve the performance of SIM from the perspective of the number of original frames. Currently, there are solutions to reduce the number of original frames required for SIM [Qian J, Cao Y, Xu K, et al. Robust frame-reduced structured illumination microscopy with accelerated correlation-enabled parameter estimation[J]. Applied Physics]. Letters, 2022, 121(15). However, due to their reliance on strict optical settings or complex noise models, existing frame reduction algorithms do not achieve the same reconstruction quality as conventional 9-frame reconstruction in actual imaging experiments.

[0003] The rise of deep learning technology in recent years has greatly promoted the development of image super-resolution (SIM) technology. Some studies have proven the effectiveness of deep learning for SIM [Qiao, Chang, et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature Methods 18.2(2021):194-202.]. However, deep learning-based SIM super-resolution often uses models mainly based on convolutional neural networks for training and inference. Its reconstruction quality is limited by a single model, and due to the limited receptive field of convolutional neural networks, it is difficult to model global information. Summary of the Invention

[0004] The purpose of this invention is to provide a single-frame composite structured illumination micro-imaging method based on ensemble learning.

[0005] The technical solution to achieve the objective of this invention is: a single-frame composite structure illumination micro-imaging method based on ensemble learning, the specific steps of which are as follows:

[0006] Step 1: Acquire the original illumination image of the sample using an interferometric structured illumination microsystem;

[0007] Step 2: Acquire the raw illumination image required for single-frame composite structured light illumination super-resolution;

[0008] Step 3: Generate training data, train sub-networks and ensemble networks;

[0009] Step 4: Output super-resolution results using the trained network.

[0010] Preferably, the interferometric structured illumination microsystem includes a laser, a mirror, a spatial filter, a spatial light modulator, a filtering system consisting of four lenses and a mask, a microscope, and a camera. The laser emitted by the laser is reflected by the mirror, collimated and expanded by the spatial filter, and then enters the spatial light modulator. The spatial light modulator loads nine fringe images and one composite fringe image. The nine fringe images are fringe images with three phase shifts in each of the three directions. The composite fringe image is obtained by adding one frame from each of the nine fringe images in the three illumination directions. After passing through the spatial light modulator, the light undergoes diffraction, then passes through the filtering system and enters the microscope, interfering with the surface of a fixed bovine pulmonary artery endothelial sample. The camera then acquires nine original sample images obtained from the interference of the nine original fringe images and a composite illumination image obtained from the interference of the composite fringe images. The average value of the nine original sample images is taken to obtain a wide-field image. Finally, the nine original sample images are super-resolution reconstructed using structured illumination microscopy based on principal component analysis to obtain the ground truth image.

[0011] Preferably, an ensemble learning framework is constructed, comprising three sub-networks and an adaptive ensemble network based on a visual self-attention model. The input composite lighting image is fed into the three sub-networks respectively. Sub-network 1 extracts low-frequency information from the composite lighting image, sub-network 2 extracts high-frequency information from the composite lighting image in the spatial domain, and sub-network 3 extracts high-frequency information from the composite lighting image in the frequency domain. The adaptive ensemble network based on the visual self-attention model fuses the output images obtained from the three sub-networks to obtain a super-resolution image.

[0012] Preferably, the sub-network 1 uses a U-shaped architecture, the output is set to a wide field image, and the shape of the input composite illumination image is set to (B,C,H,W), where B represents the batch size, C is the number of channels, H and W represent the height and width of the input, respectively. The sub-network 1 is an encoder-decoder structure with five layers in the encoder stage, each layer including a convolutional block and a max pooling layer.

[0013] Corresponding to the encoder, the decoder stage also contains five layers, each including a deconvolution layer and a convolutional block, which doubles the feature map size and halves the number of channels; the decoder and encoder branches in the same layer are residually connected, connecting the deconvolution feature maps with the encoder feature maps along the channel dimension; the connected feature maps are processed by convolutional blocks to halves the number of channels, and finally, a set of convolutional layers is used to convert the number of channels to 1, refining the features to produce the final output without changing the spatial dimension.

[0014] Preferably, subnetwork 2 is a recursive residual network, the output is set as the ground truth image, local and global residual connections are used, the information flow is optimized through a set number of recursive iterations, the output tensor is processed by pixel shuffling and upsampling, and finally the image is generated.

[0015] Preferably, subnetwork 3 is constructed using a Fourier domain amplitude-phase channel attention mechanism, with the output set as the ground truth image. The input composite illumination image is first processed by a head convolutional layer to generate an output tensor x. The feature map generated by the head convolutional layer is then input into a series of residual groups. Each residual group contains four residual blocks, each consisting of two partial convolutions and an amplitude-phase channel attention block. The amplitude-phase channel attention block consists of an amplitude channel attention layer and a phase channel attention layer. x undergoes a Fourier transform after passing through the partial convolutions within the residual block.

[0016]

[0017] in Represents the Fourier transform, F real F represents the real part of the Fourier transform. image The expression represents the imaginary part of the Fourier transform, j represents the imaginary unit, PConv(x) represents performing a partial convolution on x, and the amplitude channel attention layer extracts amplitude features A from the feature map after the Fourier transform.

[0018]

[0019] Then, the amplitude feature A is processed by partial convolution, global average pooling, max pooling, and one-dimensional convolution with both input and output channels set to 1. The weights of the amplitude feature are obtained through the sigmoid function and compressed to the range of [0,1].

[0020] The weights are multiplied by the feature maps extracted by partial convolution to implement the amplitude channel attention mechanism:

[0021] X ACALayer =PConv(x)×s(Conv) 1×1 ((Pooling(Pconv(A))))) (10)

[0022] Where Pooling represents a combination of average pooling and max pooling;

[0023] The phase channel attention layer extracts the phase of the Fourier transform feature map:

[0024]

[0025] The feature map is upsampled by pixel shuffling and then transformed into an image of (B, 1, 2×H, 2×W).

[0026] Preferably, the adaptive ensemble network includes three encoder stages and one decoder stage, and each encoder stage includes several sliding window self-attention modules;

[0027] The outputs of the three sub-networks are first processed through a 3×3 convolutional layer for shallow feature extraction to obtain feature maps X. c In the encoder stage, each sliding window self-attention module in the encoder and decoder divides the input feature map into several windows using a statistical language model context-based method. Then, within each window, a multi-head self-attention mechanism is used for computation, followed by layer normalization and multilayer perceptron adjustment to adjust the dimensions. Finally, a residual connection is formed with the input of the sliding window self-attention module to form the output feature map. Within each encoder stage, the sliding window self-attention modules are sequentially connected. The output of each encoder stage is patched to reduce the size of the feature map while expanding the number of channels, thus achieving... The purpose of compression encoding is to sample every element along the rows and columns of an input of size (B,H,W,C) to obtain four patches X1, X2, X3, and X4, each with dimensions (B,H / 2,W / 2,C). Then, all patches are concatenated along the channel dimensions to obtain a tensor of size (B,H / 2,W / 2,4C), and then a dimensionality transformation is performed to reshape the shape to (B,H / 2×W / 2,4C). Encoder stages 2 and 3 use the output of the patch merging operation in the previous stage as input to gradually compress the features to higher dimensions.

[0028] The output of each encoder stage is patched and expanded to ensure that the dimensions match when performing residual connections;

[0029] The output of encoder stage i is concatenated with the patched extended output of encoder stage i+1 along the channel dimension, i>1. The concatenated output is input into the decoder; the information fused by the encoder is decoded by the decoder.

[0030] Compared with the prior art, the significant advantages of this invention are: (1) This invention reduces the number of original frames of the traditional method by 9 times through single-frame composite illumination; (2) This invention can obtain high-quality super-resolution results without complex and time-consuming parameter estimation steps; (3) The integrated learning model of this invention based on convolutional neural network and Vison Transformer has stronger global modeling ability and can maintain good reconstruction performance under low excitation power.

[0031] The present invention will now be described in further detail with reference to the accompanying drawings: Attached Figure Description

[0032] Figure 1 This is an architecture diagram of a single-frame composite structured illumination micro-imaging method based on ensemble learning.

[0033] Figure 2 This is a schematic diagram of the sub-network 3 network structure.

[0034] Figure 3 This is a schematic diagram of the integrated network structure.

[0035] Figure 4 A schematic diagram of the 19 frequency components generated by composite structured light illumination.

[0036] Figure 5 This is a comparison of the super-resolution reconstruction results of the present invention with the reconstruction results of other different deep learning methods.

[0037] Figure 6 The results of the super-resolution reconstruction of the present invention under different signal-to-noise ratios are shown. Detailed Implementation

[0038] A single-frame composite structured illumination microscopy imaging method based on ensemble learning is presented. This method composites the illumination patterns of structured illumination microscopy and constructs an ensemble learning network, achieving high-precision and robust super-resolution imaging with only one composite illumination image. The invention first constructs three convolutional-based sub-networks and an adaptive ensemble learning network based on a visual self-attention model. Composite encoding is used to composite the stripes in three directions of the SIM illumination pattern. During training, single-frame composite illumination data is used as the network input, and the reconstruction results of structured illumination microscopy based on principal component analysis are used as the ground truth for network training. After training, test set data is fed into the neural network. The data is first output by the sub-networks, and the sub-network data is concatenated along the channel dimension before being input into the ensemble network to obtain the super-resolution image. This method includes the following three steps:

[0039] Step 1: Acquire the original illumination image of the sample using an interferometric structured illumination microsystem.

[0040] In a further embodiment, the interferometric structured illumination microsystem includes a laser, a mirror, a spatial filter, a spatial light modulator, a filtering system consisting of four lenses and a mask, a microscope, and a camera. The laser emitted by the laser is reflected by the mirror, collimated and expanded by the spatial filter, and then enters the spatial light modulator. The spatial light modulator loads nine fringe images and one composite fringe image. The nine fringe images are fringe images in three directions, each with a three-step phase shift. The composite fringe image is obtained by adding one frame (one-step phase shift) of the nine fringe images in each of the three illumination directions. After passing through the spatial light modulator, the light undergoes diffraction, then passes through the filtering system and enters the microscope, interfering with the surface of a fixed bovine pulmonary artery endothelial sample. The camera then acquires nine original sample images obtained from the interference of the nine original fringe images and a composite illumination image obtained from the interference of the composite fringe images. The average value of the nine original sample images is taken to obtain a wide-field image. Finally, the nine original sample images are super-resolution reconstructed using structured illumination microscopy based on principal component analysis to obtain the ground truth.

[0041] The above process was repeated 200 times, resulting in 200 sets of images. Each set contained one frame of composite lighting image (512×512 resolution), one frame of wide-field image (512×512 resolution), and one frame of ground truth image (1024×1024 resolution). Through random cropping, a total of 12,800 sets of composite lighting and wide-field images with a resolution of 128×128, and ground truth images with a resolution of 256×256 were obtained. Random horizontal / vertical flipping and random scaling were used to further enhance the dataset. Subsequently, the dataset was divided in an 8:1:1 ratio, resulting in 10,240 sets of training data, 1,280 sets of validation data, and 1,280 sets of test data.

[0042] Step 2: As Figure 1 As shown, an ensemble learning framework is constructed, comprising three convolutional sub-networks and an adaptive ensemble network based on a visual self-attention model. The input composite lighting image is fed into each of the three sub-networks. Sub-network 1 extracts low-frequency information from the composite lighting image, sub-network 2 extracts high-frequency information from the spatial domain, and sub-network 3 extracts high-frequency information from the frequency domain. The adaptive ensemble network based on the visual self-attention model fuses the output images obtained from the three sub-networks to obtain a super-resolution image.

[0043] Subnetwork 1 uses a U-shaped architecture, with the output set to a wide-field image, and the input composite illumination image shape set to (B, C, H, W), where B represents the batch size, C is the number of channels (C = 1 for the input layer), and H and W represent the height and width of the input, respectively. Subnetwork 1 is an encoder-decoder structure with five layers in the encoder stage. Each layer mainly consists of basic convolutional blocks and max pooling, and the basic convolutional blocks are represented as follows:

[0044] X o =GELU(PConv) 3×3 (Conv 3×3 (X i ))) (3)

[0045] Among them, X i and X o Represents the input and output feature maps of a basic convolutional block, Conv 3×3 PConv represents a regular convolution with a kernel size of 3. 3×3This represents a partial convolution with a kernel size of 3. During the convolution process, the feature channels obtained from the convolution are divided according to a set ratio P. The first consecutive P channels are considered representative of the entire feature map for computation, while the remaining channels undergo identity mapping, thus reducing training and inference time. GELU is the Gaussian error linear unit activation function. Subsequently, max pooling with a kernel size of 2 and a stride of 2 is used to downsample the result, reducing the feature map size to half that of the input. The encoder's first basic convolutional block expands the number of channels from 1 to 64. In the remaining four layers, each layer consists of a basic convolutional block and max pooling, successively expanding the number of channels to 128, 256, 512, and 1024. The tensor is finally compressed to a size of (B, 1024, H / 16, W / 16). Corresponding to the encoder, the decoder stage also contains five layers, each mainly consisting of deconvolution and basic convolutional blocks. The deconvolution has a kernel size of 3 and a stride of 2, doubling the feature map size and halving the number of channels. Then, residual connections are made with the same-layer branches of the encoder, concatenating the deconvolutional feature maps with the encoder feature maps along the channel dimension, thereby expanding the number of channels. The concatenated feature maps are processed through basic convolutional blocks (e.g., for the first decoder layer, the input channels are set to 1024 and the output channels to 512) to halve the number of channels. Finally, a set of 1×1 convolutions is used to reduce the number of channels to 1, refining the features to produce the final output without changing the spatial dimension.

[0046] Subnetwork 2 is a recursive residual network with the output set as the ground truth image and the input as (B, C, H, W). Local and global residual connections are used, and the information flow is optimized through 25 recursive iterations. The output tensor is then upsampled by pixel shuffling to finally generate an image of (B, 1, 2×H, 2×W).

[0047] Subnetwork 3 is built using a Fourier domain amplitude-phase channel attention mechanism, with the output also set to the ground truth image. The input tensor size is (B, C, H, W), as shown below. Figure 2 As shown, the input tensor is first processed through a head convolutional layer to produce an output tensor x of dimension (B, 64, H, W). The feature maps generated by the head convolutional layer are then fed into a series of residual groups. The network contains four residual groups, each containing four residual blocks. Each residual block consists of two partial convolutions and an amplitude-phase channel attention block, which is composed of an amplitude channel attention layer and a phase channel attention layer. x undergoes a Fourier transform after passing through the partial convolutions within the residual blocks:

[0048]

[0049] in Represents the Fourier transform, F real F represents the real part of the Fourier transform. imagThe amplitude channel represents the imaginary part of the Fourier transform, j represents the imaginary unit, and the attention layer extracts amplitude features from the feature map after the Fourier transform:

[0050]

[0051] These amplitude features are then processed using partial convolution, global average pooling, max pooling, and one-dimensional convolution with both input and output channels set to 1. The weights of the amplitude features are obtained using a sigmoid function and compressed to the range [0,1]. The sigmoid function is expressed as:

[0052]

[0053] These weights are multiplied by the feature maps extracted by partial convolution to achieve an amplitude channel attention mechanism, as shown in Equation (10), where Pooling represents a combination of average pooling and max pooling:

[0054] X ACALayer =PConv(x)×S(Conv) 1×1 ((Pooling(Pconv(A))))) (10)

[0055] The phase channel attention layer extracts the phase of the Fourier transform feature map:

[0056]

[0057] The other steps are consistent with the amplitude channel attention layer, thus implementing the phase channel attention mechanism.

[0058] Finally, the feature map is upsampled by pixel shuffling and transformed into an image of (B, 1, 2×H, 2×W).

[0059] The outputs of the three sub-networks are superimposed along the channel dimension to form an input adaptive ensemble network (B, 3, H, W). The adaptive ensemble network is as follows: Figure 2 As shown, the network consists of three encoder stages and one decoder stage. Each encoder stage comprises several sliding window self-attention modules. Within each stage, the odd-numbered sliding window self-attention modules perform in-window multi-head self-attention mechanism calculations. The specific process is described by the equation...

[0060]

[0061] head i =Attention(Q,K,V) (13)

[0062] MutiHead(Q,K,V)=Concat(head1,…head h W O (14)

[0063] In the formula, Q, K, V are matrices obtained by linear transformation of the input matrix, Attention, head, and MultiHead represent the operations for calculating the self-attention mechanism, single-head attention mechanism, and multi-head attention mechanism, respectively, and d k Denotes the dimensions of Q, K, and V; softmax is the normalized exponential function; W O The weight matrix represents the linear transformation of the multi-head attention mechanism; Concat represents the weight matrix of the head. i (i = 1…h) in d k The splicing operation is performed on the dimension.

[0064] The even-numbered block sliding window self-attention module performs the multi-head self-attention mechanism calculation for the sliding window, specifically as follows:

[0065] Slide the input matrix one pixel to the right and down, and fill the first row and first column of the matrix with the third row and third column before sliding. Then calculate the matrix according to the process of equation (12-14).

[0066] The output from the sub-network first undergoes shallow feature extraction via a 3×3 convolutional layer, resulting in the feature map X. cIn the encoder stage, each sliding window self-attention module in the encoder and decoder divides the input feature map into several windows using a statistical language model context-based method. Then, multi-head self-attention mechanism computation is performed within each window. Subsequently, layer normalization and multi-layer perceptron adjustments are made to adjust the dimensions, and residual connections are formed with the input of the corresponding sliding window self-attention module to form the output feature map. Encoder stage 1 consists of 6 sliding window self-attention modules, encoder stage 2 consists of 4, and encoder stage 3 consists of 4. The sliding window self-attention modules within each stage are sequentially connected. The output of each encoder stage is patched to reduce the size of the feature map and expand the number of channels, achieving compressed encoding. For an input of size (B, H, W, C), sampling is performed every other element along the rows and columns to obtain four patches X1, X2, X3, and X4, each with dimensions (B, H / 2, W / 2, C). All patches are then concatenated along the channel dimensions to obtain a tensor of size (B, H / 2, W / 2, 4C), which is then reshaped to (B, H / 2 × W / 2, 4C) using a dimensionality transformation. The second and third encoder stages use the output of the patch merging operation from the previous stage as their input, progressively compressing the features to higher dimensions. Furthermore, to achieve residual connections, the output of each encoding stage is patched to ensure size matching during residual connections. This process combines pixel shuffling, convolution, layer normalization, and dimensionality concatenation. For the i-th stage of the encoder (i>1), its output is concatenated with the patched extended output of the (i+1)-th stage along the channel dimension: Let X i Let X be the output feature of the i-th stage. i+1 X is the output feature of the (i+1)th stage. i+1 First, the pixels are shuffled:

[0067] X up =PixelShuffle(X i+1 (15)

[0068] Next, a 3×3 convolutional layer is used to adjust the number of channels to match subsequent calculations and further enhance the feature information:

[0069] X d =Convolution(X) up (3×3) (16)

[0070] Then, dimensionality transformation is performed and layer normalization is applied to X. d Processing:

[0071] X p =LN(X d(17)

[0072] Finally, the layer normalized output is compared with X. i The data is concatenated along the channel dimension, and the result is input into the decoder. This step is represented as:

[0073] X c =Concat(X) i ,X p (18)

[0074] This concatenation effectively integrates multi-scale features, enhancing the network's representational capabilities. The information fused by the encoder is decoded by a separate decoder with a sliding window self-attention module of 6, whose output is combined with the shallow feature X. c Residual connections are performed, and 3×3 convolutions are used to convert the feature channels from 64 to 1 to achieve image reconstruction.

[0075] Step 3: Train the subnetworks and the ensemble network.

[0076] The sub-networks use the root mean square error (MSE) function to calculate the loss, while the ensemble network uses a weighted combination of the structural similarity loss function (SSIM) and the perceptual loss (LP) to calculate the loss, with the perceptual loss weight set to 0.2. An adaptive momentum optimizer (adam) is used to update the network parameters, and a dynamic learning rate adjustment strategy is employed to fine-tune the model weights. The initial learning rate is set to 0.05. From training epoch 0 to 60, the learning rate decays by a factor of 0.5 every 15 training epochs. From training epoch 61 to 150, the learning rate decays by a factor of 0.5 every 30 training epochs. Starting from training epoch 150, if the loss does not decrease for 20 consecutive training epochs, the learning rate decays by a factor of 0.5. The constructed neural network is computed on a workstation equipped with an Intel Core i7-13700KF CPU, 32GB RAM, and an NVIDIA GeForce RTX4090, based on the PyTorch platform (version 2.0.1, using Python 3.10.0).

[0077] Step 4: Utilize the trained network to output super-resolution results. Load the composite lighting images from the validation set, along with the network model and its parameters. First, feed the composite lighting image data into the three sub-networks for output. Then, stack the outputs of the sub-networks along the channel dimension, feed them into the ensemble network, and finally output the final result.

[0078] Example

[0079] To test the feasibility of this invention, it was first compared with two other state-of-the-art neural networks for super-resolution reconstruction of structured illumination: the Amplitude-Phase Channel Attention Network (APCAN) and the Fourier Enhancement and Shift Layered Network (FESTN). Both networks used for comparison were trained with composite structured illumination images and corresponding super-resolution images as input and output. The bovine pulmonary artery epithelial cell prediction results obtained by these three methods are attached. Figure 5 As shown. Due to the significant spectral overlap of the input images, which poses a considerable challenge to the performance of deep learning, the reconstruction quality of FESTN is relatively affected, and the prediction results for mitochondria and actin do not show significant super-resolution effects (see attached). Figure 5 (b3) and (c3) in the diagram). Although the amplitude-phase channel attention network achieved resolution-enhanced reconstruction of mitochondria, its fidelity was affected by spectral overlap, and its actin results were still affected (see appendix). Figure 5 (b4) and (c4) in the image). Conversely, eDL-cSIM can recover higher quality super-resolution images, especially in the actin results, where fine structures that other neural networks cannot distinguish are well resolved (see appendix). Figure 5 (b5 and (c5) in the text).

[0080] Next, the performance of eDL-cSIM in complex imaging environments was tested. The excitation power was gradually reduced from the rated value (55mW) to 25% of its original value, and nine raw images of mitochondria in bovine pulmonary artery epithelial cells were captured under nine-frame stripe illumination mode at different power levels, and one image was captured under compound stripe illumination mode. Super-resolution images were obtained by applying Principal Component Analysis-based Structured Illumination Microscopy (PCA-SIM) and eDL-cSIM, respectively. (See attached image.) Figure 6 As shown, the results of structured illumination microscopy based on principal component analysis are presented (attached). Figure 6 In the model (c1)-(c4), reconstruction artifacts become increasingly severe as the excitation power decreases. The structural similarity between the results under different noise environments and the results under high excitation power decreases from 0.77 to 0.47, while eDL-cSIM (with appendix) shows improved similarity. Figure 6 The reconstruction quality of (d1)-(d4) remains stable throughout, with a structural similarity above 0.85. This embodiment demonstrates the comprehensive advantages of the present invention in terms of accuracy and noise resistance, as well as its potential for long-term live-cell imaging under weak excitation conditions.

Claims

1. A single-frame composite structured illumination micro-imaging method based on ensemble learning, characterized in that, The specific steps are as follows: Step 1: Acquire the original illumination image of the sample using an interferometric structured illumination microsystem; Step 2: Construct an ensemble learning framework; the ensemble learning framework includes three sub-networks and an adaptive ensemble network based on a visual self-attention model. The input composite lighting image is fed into the three sub-networks respectively. Sub-network 1 extracts low-frequency information from the composite lighting image, sub-network 2 extracts high-frequency information from the composite lighting image in the spatial domain, and sub-network 3 extracts high-frequency information from the composite lighting image in the frequency domain. The adaptive ensemble network based on the visual self-attention model fuses the output images obtained from the three sub-networks to obtain a super-resolution image. The sub-network 1 uses a U-shaped architecture, with the output set to a wide-field image and the shape of the input composite illumination image set to (B, C, H, W), where B represents the batch size, C is the number of channels, and H and W represent the height and width of the input, respectively. Sub-network 1 is an encoder-decoder structure with five layers in the encoder stage, each layer including a convolutional block and a max pooling layer. Corresponding to the encoder, the decoder stage also contains five layers, each including a deconvolution layer and a convolutional block, which doubles the feature map size and halves the number of channels; the decoder and encoder branches in the same layer are connected by residual connections, connecting the deconvolution feature maps with the encoder feature maps along the channel dimension; the connected feature maps are processed by convolutional blocks to halves the number of channels; finally, a set of convolutional layers is used to convert the number of channels to 1, refining the features to produce the final output without changing the spatial dimension; Subnetwork 2 is a recursive residual network. The output is set as the ground truth image. It uses local and global residual connections and optimizes the information flow through a set number of recursive iterations. The output tensor is upsampled by pixel shuffling to finally generate the image. Subnetwork 3 is built using a Fourier domain amplitude-phase channel attention mechanism. The output is set as the ground truth image. The input composite illumination image is first processed by the head convolutional layer to produce an output tensor of dimension 1. The feature maps generated by the head convolutional layer are then fed into a series of residual groups, each containing four residual blocks. Each residual block consists of two partial convolutions and an amplitude-phase channel attention block, which in turn consists of an amplitude channel attention layer and a phase channel attention layer. After partial convolution within the residual block, a Fourier transform is performed: ； in Represents Fourier transform, Represents the real part of the Fourier transform. This represents the imaginary part of the Fourier transform, and j represents the imaginary unit. The representation performs partial convolution on the input x, and the amplitude channel attention layer extracts amplitude features from the feature map after Fourier transform. : ； Then, the amplitude features are processed using partial convolution, global average pooling, max pooling, and one-dimensional convolution with both input and output channels set to 1. The weights of the amplitude characteristics are obtained through a sigmoid function and compressed to the range of [0, 1]. The weights are multiplied by the feature maps extracted by partial convolution to implement the amplitude channel attention mechanism: ； Where Pooling represents a combination of average pooling and max pooling; The phase channel attention layer extracts the phase of the Fourier transform feature map: ； The feature map is upsampled by pixel shuffling and transformed into an image of (B, 1, 2×H, 2×W); The adaptive ensemble network comprises three encoder stages and one decoder stage, with each encoder stage including several sliding window self-attention modules; The outputs of the three sub-networks are first processed by a 3×3 convolutional layer for shallow feature extraction to obtain feature maps. In the encoder stage, each sliding window self-attention module in the encoder and decoder divides the input feature map into several windows using a statistical language model context-based method. Then, within each window, a multi-head self-attention mechanism is used for computation, followed by layer normalization and multilayer perceptron adjustment to adjust the dimensions. Finally, a residual connection is made with the input of the sliding window self-attention module to form the output feature map. Within each encoder stage, the sliding window self-attention modules are sequentially connected. The output of each encoder stage is patched to reduce the size of the feature map while expanding the number of channels, achieving compression encoding. For an input of size (B, H, W, C), sampling is performed along every row and column, resulting in four patches X1, X2, X3, and X4, each with dimensions (B, H / 2, W / 2, C). All patches are then concatenated along the channel dimensions to obtain a tensor of size (B, H / 2, W / 2, 4C), which is then reshaped into a tensor of size (B, H / 2 × W / 2, 4C). 4C); Encoder stages 2 and 3 use the output of the previous stage patch merging operation as input to gradually compress the features to higher dimensions; The output of each encoder stage is patched and expanded to ensure that the dimensions match when performing residual connections; Encoder stage Output and encoder stages The +1 patch extension output is concatenated along the channel dimension. >1, the spliced ​​output is input into the decoder; the information fused by the encoder is decoded by the decoder; Step 3: Generate training data, train sub-networks and ensemble networks; Step 4: Output super-resolution results using the trained network.

2. The single-frame composite structure illumination microscopic imaging method based on ensemble learning according to claim 1, characterized in that, The interferometric structured illumination microsystem includes a laser, a mirror, a spatial filter, a spatial light modulator, a filtering system consisting of four lenses and a mask, a microscope, and a camera. The laser emitted by the laser is reflected by the mirror, collimated and expanded by the spatial filter, and then enters the spatial light modulator. The spatial light modulator loads nine fringe images and one composite fringe image. The nine fringe images are fringe images with three phase shifts in each of the three directions. The composite fringe image is obtained by adding one frame from each of the nine fringe images in the three illumination directions. After passing through the spatial light modulator, the light undergoes diffraction, then passes through the filtering system before entering the microscope and interfering on a fixed sample surface. The camera then acquires nine original sample images obtained from the interference of the nine original fringe images and a composite illumination image obtained from the interference of the composite fringe images. The average value of the nine original sample images is taken to obtain a wide-field image. Finally, the nine original sample images are super-resolution reconstructed using structured illumination microscopy based on principal component analysis to obtain the ground truth image.

3. The single-frame composite structured illumination micro-imaging method based on ensemble learning according to claim 1, characterized in that, Encoder stage 1 consists of 6 sliding window self-attention modules, encoder stage 2 consists of 4 sliding window self-attention modules, and encoder stage 3 consists of 4 sliding window self-attention modules.

4. The single-frame composite structured illumination micro-imaging method based on ensemble learning according to claim 1, characterized in that, In each encoder stage, an odd-numbered sliding window self-attention module performs in-window multi-head self-attention mechanism calculations. Even-numbered block sliding window self-attention modules perform sliding window multi-head self-attention mechanism calculations.

5. The single-frame composite structured illumination micro-imaging method based on ensemble learning according to claim 4, characterized in that, The specific process of calculating the multi-head self-attention mechanism within the window using the odd-numbered sliding window self-attention module is as follows: ； ； ； In the formula, It is the matrix obtained by linear transformation of the input matrix. , , These represent the operations for computing self-attention, single-head attention, and multi-head attention mechanisms, respectively. k express dimensionality; It is a normalized exponential function; The weight matrix represents the linear transformation of the multi-head attention mechanism; Indicates will In d k The splicing operation is performed on the dimension. , where h represents the number of heads in the multi-head attention mechanism; The even-numbered block sliding window self-attention module performs the multi-head self-attention mechanism calculation for the sliding window, specifically as follows: The input matrix is ​​slid down and to the right by one pixel, while the first row and first column of the matrix are filled with the third row and third column before the sliding. The matrix is ​​then calculated using the multi-head self-attention mechanism within the window through the odd-number sliding window self-attention module.

Images