Super-resolution fluorescence microscopic imaging method and product based on deep learning

Abstract

The invention provides a super-resolution fluorescence microscopic imaging method and product based on deep learning, and the method comprises the steps: carrying out the feature extraction and image reconstruction of an actual fluorescence microscopic image based on a trained super-resolution fluorescence microscopic imaging model, and obtaining a super-resolution fluorescence microscopic image result; wherein the super-resolution fluorescence microscopic imaging model is constructed on the basis of an encoder-decoder architecture fused with a residual network, a channel attention mechanism and a self-attention module, and the trained super-resolution fluorescence microscopic imaging model is obtained through pre-training-fine tuning dual-stage training. According to the invention, based on the encoder-decoder architecture, the residual network, the channel attention mechanism and the self-attention module construction are fused, and the pre-training and fine-tuning dual-stage training strategy is combined, so that the cross-modal super-resolution migration capability under the single-sample condition is realized, the bottleneck that the traditional deep learning depends on a large amount of data repeated training is broken through, and the super-resolution migration capability is improved. When different imaging samples and systems are replaced for super-resolution reconstruction, resource consumption can be remarkably reduced.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and optical microscopy imaging, and in particular to a super-resolution fluorescence microscopy imaging method and product based on deep learning. Background Art

[0002] Fluorescence microscopy is an important tool for modern life science research, but traditional optical microscopes are limited by the Abbe diffraction limit and have difficulty resolving structures smaller than approximately 200 nanometers. To overcome this limitation, super-resolution fluorescence microscopy (SRFM) has emerged. Representative methods include stimulated emission depletion (STED), structured illumination microscopy (SIM), and single-molecule localization microscopy (STORM). These methods have significantly improved the resolution of fluorescence microscopy, enabling the visualization of cellular structures and molecular processes at the nanoscale, and have played an important role in basic scientific research and bioscience imaging.

[0003] However, these super-resolution techniques usually require high-power lasers, complex optical systems, and computationally intensive image reconstruction algorithms. For example, STED microscopy uses a loss laser with a ring-shaped focus to suppress fluorescence emission, thereby improving resolution, but high-power lasers may cause photobleaching and phototoxicity problems in biological samples. SIM improves resolution by applying a specific structured illumination pattern combined with computational reconstruction, but the imaging rate is limited and requires precise calibration of the optical system. The SMLM method achieves super-resolution reconstruction by randomly activating and precisely positioning single fluorescent molecules, but the acquisition process is often long and difficult to apply to living cell imaging. Therefore, although these techniques have significantly improved the resolution of optical microscopy, their widespread application is still limited by experimental conditions, imaging speed, and computing resources.

[0004] In recent years, the development of computational imaging methods has provided new solutions for super-resolution microscopy. By combining deep learning techniques, researchers can reconstruct super-resolution (SR) images from low-resolution (LR) images under low-power illumination and short exposure times. For example, deep learning-driven super-resolution reconstruction methods use deep neural networks to learn the mapping relationship between low-resolution and super-resolution images, thereby achieving super-resolution enhancement. These methods can reduce photobleaching effects, improve signal-to-noise ratios, and reduce dependence on high-end hardware equipment, making super-resolution imaging technology accessible to a wider range of researchers.

[0005] Although deep learning has shown great potential in super-resolution imaging, existing methods still have certain limitations. The training of most deep learning models depends on specific imaging modalities and sample types. When the imaging system or sample is replaced, it is often necessary to collect a large amount of data and retrain the model, resulting in a large consumption of resources and time. In addition, many super-resolution networks are only applicable to specific experimental conditions and lack generalization capabilities. Therefore, it is of great significance to develop a universal super-resolution reconstruction method that can cross systems and samples. This method should be able to reduce the dependence on prior physical knowledge, adapt to different types of microscopic imaging data, and simplify the super-resolution imaging process, so that non-professional users can quickly obtain high-quality super-resolution images. Summary of the Invention

[0006] The purpose of this application is to provide a super-resolution fluorescence microscopy imaging method and product based on deep learning, which can at least solve the problem of large resource consumption of imaging models in related technologies when changing different types of imaging samples for training.

[0007] In order to solve the above technical problems, the first aspect of the embodiments of the present application provides a super-resolution fluorescence microscopy imaging method based on deep learning, comprising:

[0008] Acquire actual fluorescence microscopy images;

[0009] Based on the trained super-resolution fluorescence microscopy imaging model, feature extraction and image reconstruction are performed on the actual fluorescence microscopy image to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on the encoder-decoder architecture fusing the residual network, the channel attention mechanism, and the self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage training of pre-training and fine-tuning.

[0010] A second aspect of the embodiments of the present application provides a super-resolution fluorescence microscopy imaging system based on deep learning, comprising:

[0011] An acquisition module, used for acquiring actual fluorescence microscopy images;

[0012] An imaging module is used to perform feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage pre-training-fine-tuning training.

[0013] The third aspect of the present application provides an electronic device comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory. When the processor executes the computer program, it implements each step of the super-resolution fluorescence microscopy method described in the first aspect of the embodiment of the present application.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the super-resolution fluorescence microscopy method described in the first aspect of the embodiment of the present application are implemented.

[0015] As can be seen from the above, the embodiment of the present application first obtains an actual fluorescence microscopy image, and then performs feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on an encoder-decoder architecture fusion residual network, channel attention mechanism, and self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through pre-training-fine-tuning two-stage training. The present invention is based on an encoder-decoder architecture fusion residual network, channel attention mechanism, and self-attention module construction, and combines the two-stage training strategy of pre-training and fine-tuning to train the model, so that the model can achieve cross-modal super-resolution migration capability under single sample conditions, breaking through the bottleneck of traditional deep learning methods that rely on repeated training of large amounts of data, and can significantly reduce resource consumption when changing different types of imaging samples for training.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of this application, nor is it intended to limit the scope of this application. Other features of this application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the relevant technologies or the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for the description of the relevant technologies or the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a process for super-resolution fluorescence microscopy imaging provided in the first aspect of the embodiment of the present application;

[0019] Figure 2 A simplified two-stage training flow chart for a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application;

[0020] Figure 3 A diagram showing the principle of two-stage training in a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application;

[0021] Figure 4 This is an example diagram of the wide-field data recognition effect of a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application;

[0022] Figure 5 This is an example diagram of the confocal data recognition effect of a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application;

[0023] Figure 6 A schematic diagram of a detailed process of a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application;

[0024] Figure 7 A schematic diagram of a program module of a super-resolution fluorescence microscopy imaging device provided in the second aspect of an embodiment of the present application;

[0025] Figure 8 A module block diagram of an electronic device provided in the third aspect of an embodiment of the present application;

[0026] Figure 9 A module block diagram of a computer-readable storage medium provided in the fourth aspect of an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application more obvious and easy to understand, the present application will be clearly and completely described below in conjunction with the embodiments of the present application and the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. It should be understood that the various embodiments of the present application described below are merely used to explain the present application and are not used to limit the present application. That is, based on the various embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0028] See also Figure 1 , Figure 1 This is a flow chart of a super-resolution fluorescence microscopy method provided in the first aspect of an embodiment of the present application. The super-resolution fluorescence microscopy method includes the following steps.

[0029] Step 101: Acquire an actual fluorescence microscopy image.

[0030] Step 102: Perform feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein the super-resolution fluorescence microscopy imaging model is constructed based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module, and is trained through a pre-training-fine-tuning two-stage training to obtain the trained super-resolution fluorescence microscopy imaging model.

[0031] In an embodiment of the present application, an actual fluorescence microscopy image is first obtained, and then feature extraction and image reconstruction are performed on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on an encoder-decoder architecture fusion residual network, channel attention mechanism, and self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through pre-training-fine-tuning two-stage training. The present invention is based on an encoder-decoder architecture fusion residual network, channel attention mechanism, and self-attention module construction, and combines the two-stage training strategy of pre-training and fine-tuning to train the model, so that the model can achieve cross-modal super-resolution migration capability under single sample conditions, breaking through the bottleneck of traditional deep learning methods that rely on repeated training of large amounts of data, and can significantly reduce resource consumption when changing different types of imaging samples for training. The specific construction of the model will be described in detail in subsequent steps.

[0032] In an optional embodiment of the present embodiment, before the step of performing feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model, the step also includes: constructing a simulated data set containing simulated image samples, and dividing the data set into a training sample set and a test sample set according to a preset ratio; randomly selecting simulated image samples from the training sample set and inputting them into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain a training result; adjusting the network parameters in the original super-resolution fluorescence microscopy imaging model based on the training result and a preset loss function to obtain an adjusted super-resolution fluorescence microscopy imaging model; randomly selecting simulated image samples from the test sample set and inputting them into the adjusted super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain a test result; if the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model; if the test fails, returning to the step of randomly selecting simulated image samples from the training sample set and inputting them into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction.

[0033] In some embodiments, the dataset is expanded by rotating, flipping, and color normalizing the simulated image samples in the dataset. Specifically, seven geometric transformations (e.g., 90° counterclockwise rotation followed by vertical flip, vertical flip, 90° clockwise rotation, etc.) can be randomly applied, with each transformation having a 50% probability of being applied or not applied. This allows a single simulated image sample to be transformed into multiple simulated image sample variants, significantly expanding the dataset.

[0034] It is understandable that in some embodiments, min-max normalization is further performed on the input image and the target image to scale the pixel values ​​to the range of [0, 1] to ensure the numerical stability of the training process.

[0035] Furthermore, the PyTorch framework was used for training in the embodiments of this application to ensure efficient model training and deployment. The Adam optimizer was used, with an initial learning rate set to 1e-4. A cosine annealing strategy was used to adjust the learning rate, dynamically adjusting the learning rate to accelerate model convergence. Specifically, the training dataset consisted of 4,000 training samples and 1,000 validation samples, with the input image size set to 256×256. Training was performed for 500 epochs, with an initial learning rate set to 1e-4 and a batch size of 8, for approximately 14 hours.

[0036] In an optional embodiment of the present embodiment, if the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model, including the following steps: if the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the pre-trained super-resolution fluorescence microscopy imaging model; obtaining a fine-tuning data set containing fluorescence microscopy image samples of a target sample; wherein the target sample is a sample contained in an actual fluorescence microscopy image, the fine-tuning data set contains a first fluorescence microscopy image sample and a second fluorescence microscopy image sample, and the second fluorescence microscopy image sample is a super-resolution fluorescence microscopy image sample corresponding to the first fluorescence microscopy image sample; The weights of the last deconvolution layer in the pre-trained super-resolution microscopy imaging model are fine-tuned based on the fine-tuning dataset to obtain a fine-tuned super-resolution fluorescence microscopy imaging model; the first fluorescence microscopy image sample is input into the fine-tuned super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction, and the output result is compared with the second fluorescence microscopy image sample for verification; if the verification passes, the fine-tuned super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model; if the verification fails, the process returns to the step of fine-tuning the weights of the last deconvolution layer in the pre-trained super-resolution fluorescence microscopy imaging model based on the fine-tuning dataset.

[0037] The embodiment of the present application adopts a two-stage training strategy, including pre-training and fine-tuning. In order to make the method logic more intuitive, the first aspect of the embodiment of the present application provides a simplified two-stage training flow chart and principle diagram of a super-resolution fluorescence microscopy imaging model, such as Figure 2 and Figure 3 As shown in the figure, the pre-training phase is performed on simulated data, and the fine-tuning phase is performed on real data to ensure the model's adaptability across different modalities. During the pre-training phase, the model is trained on a large-scale dataset to learn universal feature representations. The training dataset contains 4000 training samples and 1000 validation samples, and the input image size is set to 256×256. Training is performed for 500 epochs, with an initial learning rate of 1e-4 and a batch size of 8. The training time is approximately 14 hours. After training is completed and tested, the pre-trained super-resolution fluorescence microscopy model is obtained. The fine-tuning phase then begins, where the resulting model is optimized using the fine-tuning dataset. During training, all layers except the last deconvolution layer are frozen, and only the weights of the last deconvolution layer are updated to adapt to the new modality requirements.

[0038] It should be noted that the fluorescence microscopy image data obtained by confocal microscopy and wide-field microscopy in the embodiments of the present application are considered to be low-resolution (LR) images, while the fluorescence microscopy image data obtained by methods such as structured illumination microscopy (SIM), single-molecule localization microscopy (SMLM), and stimulated emission depletion microscopy (STED) are considered to be super-resolution (SR) images. The model of the present application will reconstruct the low-resolution image obtained by the wide-field microscope into a structured illumination microscopy (SIM) image or a single-molecule localization microscopy (SMLM) image, and reconstruct the low-resolution image obtained by the confocal microscope into a stimulated emission depletion microscopy (STED) image. Therefore, in the step of obtaining the actual fluorescence microscopy image in the above method, the actual fluorescence microscopy image is generally a low-resolution image. Since there is a difference of one times in the number of pixels between the wide field and the SIM, in the embodiments of the present application, a bilinear interpolation method will be further used before training to interpolate and amplify the wide field image by two times so that it is consistent with the pixel size of the SIM image.

[0039] Furthermore, the fine-tuning in the embodiment of the present application is to adjust the pre-trained model through a pair of data of a specific modality. Simply put, if it is necessary to reconstruct the target sample in the actual fluorescence microscopy image, a pair of fluorescence microscopy images corresponding to the target sample is collected as a fine-tuning data set, and the model is fine-tuned based on the fine-tuning data set to obtain a fine-tuned model adapted to the modality of the target sample. This model can then be used to reconstruct the actual fluorescence microscopy image into a super-resolution image. Among them, the fine-tuning data set requires at least a pair of LR-SR data, and the pair of LR-SR data includes a first fluorescence microscopy image sample and a second fluorescence microscopy image sample. The first fluorescence microscopy image sample is a low-resolution image (LR data), and the second fluorescence microscopy image sample is a super-resolution fluorescence microscopy image (SR data) corresponding to the first fluorescence microscopy image sample.

[0040] Therefore, the embodiment of the present application only needs to re-collect high-resolution and low-resolution image samples of a pair of samples for the sample, and input them into the trained model for fine-tuning. The model can quickly adapt to the new modality and generate high-quality super-resolution images. The super-resolution fluorescence microscopy model in this application breaks through the bottleneck of traditional deep learning methods that rely on repeated training of large amounts of data, and greatly enhances the generalization ability of the model. In addition, it is worth noting that low-resolution fluorescence microscopy images and super-resolution fluorescence microscopy images are actually collected through corresponding different optical systems. Therefore, the embodiment of the present application not only realizes cross-sample reconstruction, but also realizes a universal super-resolution reconstruction method across systems, which can reduce dependence on prior physical knowledge, adapt to different types of microscopic imaging data, and simplify the super-resolution imaging process, so that non-professional users can also quickly obtain high-quality super-resolution images.

[0041] Furthermore, in an optional embodiment of the present embodiment, the step of performing feature extraction on an actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model includes: randomly capturing multiple local windows of a single fluorescence microscopy image sample based on the trained super-resolution fluorescence microscopy imaging model, and synchronously performing feature extraction on the captured multiple local windows.

[0042] Specifically, when the super-resolution fluorescence microscopy imaging model extracts features from the input fluorescence microscopy image samples, it also adopts a multi-field sampling strategy. By randomly extracting multiple local windows on a single image, feature extraction is performed synchronously to maximize information utilization. Specifically, 8 256×256 image areas are randomly captured from each image. This operation expands the data channel by 8 times and ensures that information from different positions of the image can be obtained simultaneously. The training parameters in the fine-tuning stage are adjusted to an initial learning rate of 8e-5, and a total of 5000 epochs are performed, with a training time of about 2 hours. The specific training time is adjusted according to different modalities. Usually, 500 epochs (about 10 minutes) can achieve a significant super-resolution effect. This application designs a random multi-window synchronous sampling strategy to achieve parallel feature capture of N 256×256 windows on a single input image, breaking through the information bottleneck of traditional single-window sampling.

[0043] Specifically, in terms of model optimization, this application uses a preset loss function that is composed of the mean square error (MSE) loss function, the structural similarity (SSIM) loss function, and the perceptual loss function to balance pixel-level accuracy, structural fidelity, and perceptual quality. The total loss function is defined as follows:

[0044] L=0.8*L MSE +0.2*L SSIM +0.05*L perceptual

[0045] Among them, the mean square error (MSE) loss is used to calculate the mean square error of the pixel values ​​between the predicted image and the real image, ensuring the consistency of the predicted image and the real image at the pixel level. The structural similarity (SSIM) loss is used to calculate the loss by comparing the structural similarity of the predicted image and the real image, maintaining the structural integrity and contrast of the reconstructed image. The perceptual loss is used to calculate the mean square error of the generated image and the real image on the VGG network feature layer by extracting high-level semantic features using a pre-trained VGG network, thereby enhancing the perceptual quality of the generated image. The embodiment of the present application jointly optimizes the mean square error loss, structural similarity loss and perceptual loss to ensure that the generated image is highly consistent with the real image at the pixel level, structural level and perceptual level, thereby achieving high-quality super-resolution reconstruction.

[0046] In an optional embodiment of this embodiment, the super-resolution fluorescence microscopy model includes: an encoder module, including four downsampling blocks, each downsampling block including two residual channel attention modules and a convolution layer; a decoder module, symmetrical in structure to the decoder module, including four upsampling blocks, each upsampling block including two residual channel attention modules and a deconvolution layer; wherein the upsampling blocks are jump-connected to the downsampling blocks to retain key spatial information; a self-attention module, connected between the upsampling blocks, for capturing long-distance dependencies and fine-grained features; an output module, connected to the end of the decoder module, including a 2×2 deconvolution layer, for reducing the number of channels from 64 to 1 to generate a grayscale image.

[0047] The embodiment of the present application adopts an encoder-decoder framework as the core network structure, integrating the residual network (ResNet) and the attention mechanism, taking into account both the global semantic feature extraction of the image and the preservation of local details. This architecture achieves accurate reconstruction from low-resolution input to high-resolution output through multi-level feature transformation and information fusion. The encoder module is used for coarse-to-fine feature extraction, gradually downsampling the low-resolution image, and extracting multi-scale features (such as organelle contours, textures, etc.). The encoder extracts spatial features through four downsampling blocks, each of which contains two RCAB (Residual Channel Attention Block) and a convolutional layer. The channel attention mechanism of RCAB can enhance significant structures, retain key details, and improve the network's ability to extract high-frequency information. The global spatial information is aggregated by global average pooling, then processed by two 1×1 convolutional layers and Leaky ReLU activation functions, and finally the attention weights are normalized by the Sigmoid activation function. Batch normalization is introduced after the 3×3 convolutional layer to enhance model stability and accelerate convergence.

[0048] Furthermore, the decoder is used to reconstruct from features to high resolution, gradually upsampling the abstract features extracted by the encoder to restore the high-resolution image. It consists of four upsampling blocks, which are symmetrical with the encoder structure. Each block contains two RCABs and a deconvolution layer for reconstructing high-quality images. Among them, the present application also splices the feature map of the corresponding layer of the encoder with the current layer of the decoder through a jump connection to transmit the underlying spatial information (such as the position of the organelle boundary), avoid the loss of details, and retain key spatial information. And after passing the last upsampling block, a 2×2 deconvolution layer is added to restore the original resolution, reducing the number of channels from 64 to 1 to generate a high-quality grayscale image.

[0049] In addition, the model also introduces a self-attention module in the transition layer to capture long-distance dependencies and fine-grained features. Within the local window of the feature map, the correlation between pixels is calculated through the Q (Query), K (Key), and V (Value) matrices to capture long-distance dependencies (such as the spatial continuity of the mitochondrial network), and the attention mechanism is performed within the local window. The matrix is ​​group normalized and projected by convolution operations to ensure training stability. Furthermore, the self-attention module also uses multiple attention heads for parallel calculations. Specifically, 8 parallel attention heads are used for calculations, focusing on features of different dimensions (such as shape, texture, brightness), improving the model's expression capabilities, and finally splicing the results. In addition, the module also includes the introduction of learnable relative position encoding, adding a position offset parameter for each position, so that the network understands the spatial arrangement of organelle structures, and further improves the model's understanding of spatial information.

[0050] In an optional embodiment of the present embodiment, the step of performing feature extraction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model further includes: if the trained super-resolution fluorescence microscopy imaging model detects that the actual fluorescence microscopy image is three-dimensional data, then replacing the two-dimensional convolution in the trained super-resolution fluorescence microscopy imaging model with a three-dimensional convolution, and then performing feature extraction on the actual fluorescence microscopy image.

[0051] Specifically, in the aforementioned embodiments of the present application, the fluorescence microscopy image data obtained by confocal microscopy and wide-field microscopy are considered to be low-resolution (LR) images, while the fluorescence microscopy image data obtained by structured light illumination microscopy (SIM), single-molecule localization microscopy (SMLM), stimulated emission depletion microscopy (STED) and other methods are considered to be super-resolution (SR) images. The model of the present application will reconstruct the low-resolution image obtained by the wide-field microscope into a structured light illumination microscopy (SIM) image, and reconstruct the low-resolution image obtained by the confocal microscope into a stimulated emission depletion microscopy (STED) image. Among them, the fluorescence microscopy image data obtained by the wide-field microscope is two-dimensional, and the image data obtained by the confocal microscope is three-dimensional. In order to realize the generalization ability of the model from two-dimensional to three-dimensional, specifically, the two-dimensional operation is expanded to three-dimensional (such as a 3×3×3 convolution kernel), and the convolution and deconvolution operations are changed to three-dimensional form, thereby constructing a network model suitable for processing three-dimensional data and capturing spatial information in the depth direction (such as the three-dimensional structure of the inner and outer membranes of mitochondria). Finally, the super-resolution fluorescence microscopy model in the embodiments of the present application achieves generalization capabilities in three-dimensional depth.

[0052] Figure 4 This is an example diagram of the wide-field data recognition effect of a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application. Figure 5 The example diagram of the confocal data recognition effect of a super-resolution fluorescence microscopy imaging method provided in the first aspect of the embodiment of the present application clearly shows that the texture of the microtubules after processing is clear, the structure of the mitochondrial outer membrane is obvious, the boundaries are clear, and the resolution of the two-dimensional nuclear pore complex and the three-dimensional nuclear pore complex in all directions is significantly improved.

[0053] In some embodiments, if large-scale volume data that exceeds the GPU memory limit is input into the model, a sliding window strategy can be used to divide the input data into multiple sub-data, which are processed separately and then spliced ​​together to generate a complete output result.

[0054] As can be seen from the above, in the embodiments of the present application, an actual fluorescence microscopy image is first obtained, and then feature extraction and image reconstruction are performed on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on the encoder-decoder architecture fusing the residual network, the channel attention mechanism, and the self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage training of pre-training and fine-tuning. The present invention is based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module. The model is trained in combination with a two-stage training strategy of pre-training and fine-tuning, so that the model can achieve cross-modal super-resolution migration capabilities under single-sample conditions. When the sample is not in the model database, only a pair of LR-SR corresponding modality data is required for training to reconstruct low-resolution fluorescence microscopy images into super-resolution images. Compared with traditional physical optics methods, this method can achieve super-resolution reconstruction under conditions of lower light power and shorter exposure time, improve the signal-to-noise ratio and reduce the photobleaching effect. At the same time, it has the ability to generalize across systems and organelles, breaking through the bottleneck of traditional deep learning methods that rely on repeated training of large amounts of data, and can significantly reduce resource consumption when replacing different types of imaging samples and systems for super-resolution reconstruction.

[0055] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order in which the steps are executed. The order in which the steps are executed should be determined by their functions and internal logic, and should not constitute a sole limitation on the implementation process of the embodiments of this application.

[0056] In summary, Figure 6 A detailed flow chart of a super-resolution fluorescence microscopy imaging method provided in an embodiment of the present application is as follows:

[0057] Step 601: construct a simulated data set containing simulated image samples, and divide the data set into a training sample set and a test sample set according to a preset ratio;

[0058] Step 602: Construct a super-resolution fluorescence microscopy model based on an encoder-decoder architecture by integrating a residual network, a channel attention mechanism, and a self-attention module.

[0059] Step 603: pre-training the original super-resolution fluorescence microscopy imaging model based on the training sample set, adjusting the network parameters in the original super-resolution fluorescence microscopy imaging model according to the pre-training results and a preset loss function, and obtaining a pre-trained super-resolution fluorescence microscopy imaging model through testing;

[0060] Step 604: Acquire a fine-tuning dataset containing fluorescence microscopy image samples of the target sample;

[0061] Step 605: fine-tune the weight of the last deconvolution layer in the pre-trained super-resolution microscopy imaging model based on the fine-tuning dataset, and obtain a fine-tuned super-resolution fluorescence microscopy imaging model after verification;

[0062] Step 606: Acquire an actual fluorescence microscopy image;

[0063] Step 607: Input the actual fluorescence microscopy image into the fine-tuned super-resolution fluorescence microscopy model to perform feature extraction and image reconstruction to obtain a super-resolution fluorescence microscopy image result.

[0064] For a more detailed process of each step in steps 601 to 607, please refer to the description of the relevant parts shown above, and the embodiment of the present application will not be repeated here.

[0065] See also Figure 7 , Figure 7 Schematic diagram of the program modules of the super-resolution fluorescence microscopy imaging device provided in the second aspect of the embodiment of the present application. This device can be used to implement the super-resolution fluorescence microscopy imaging method involved in the embodiment of the present application. The super-resolution fluorescence microscopy imaging device mainly includes:

[0066] An acquisition module 701 is used to acquire an actual fluorescence microscopy image;

[0067] Imaging module 702 is used to perform feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on the encoder-decoder architecture fusing the residual network, the channel attention mechanism, and the self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage training of pre-training and fine-tuning.

[0068] In some implementations of this embodiment, before the imaging module 702 executes the step of performing feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model, it also includes a training module, which is used to: construct a simulated data set containing simulated image samples, and divide the data set into a training sample set and a test sample set according to a preset ratio; randomly select simulated image samples from the training sample set and input them into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain a training result; adjust the network parameters in the original super-resolution fluorescence microscopy imaging model based on the training result and a preset loss function to obtain an adjusted super-resolution fluorescence microscopy imaging model; randomly select simulated image samples from the test sample set and input them into the adjusted super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain a test result; if the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model; if the test fails, return to execute the step of randomly selecting simulated image samples from the training sample set and inputting them into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction.

[0069] Furthermore, in some implementations of this embodiment, when the training module executes the step of using the adjusted super-resolution fluorescence microscopy imaging model as the trained super-resolution fluorescence microscopy imaging model if the test passes, the training module is specifically used to: if the test passes, use the adjusted super-resolution fluorescence microscopy imaging model as the pre-trained super-resolution fluorescence microscopy imaging model; obtain a fine-tuning data set containing a fluorescence microscopy image sample of a target sample; wherein the target sample is a sample contained in an actual fluorescence microscopy image, and the fine-tuning data set contains a first fluorescence microscopy image sample and a second fluorescence microscopy image sample, and the second fluorescence microscopy image sample is a super-resolution fluorescence microscopy image sample corresponding to the first fluorescence microscopy image sample. microscopic image samples; fine-tuning the weights of the last deconvolution layer in the pre-trained super-resolution microscopic imaging model based on the fine-tuning dataset to obtain a fine-tuned super-resolution fluorescence microscopic imaging model; inputting the first fluorescence microscopic image sample into the fine-tuned super-resolution fluorescence microscopic imaging model for feature extraction and image reconstruction, and comparing the output result with the second fluorescence microscopic image sample for verification; if the verification passes, the fine-tuned super-resolution fluorescence microscopic imaging model is used as the trained super-resolution fluorescence microscopic imaging model; if the verification fails, returning to the step of fine-tuning the weights of the last deconvolution layer in the pre-trained super-resolution fluorescence microscopic imaging model based on the fine-tuning dataset.

[0070] In some embodiments of this embodiment, when the training module executes the step of extracting features from an actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model, it is specifically used to: randomly capture multiple local windows from a single fluorescence microscopy image sample based on the trained super-resolution fluorescence microscopy imaging model, and synchronously perform feature extraction on the captured multiple local windows.

[0071] In some embodiments of this embodiment, the super-resolution fluorescence microscopy model includes: an encoder module, including four downsampling blocks, each downsampling block including two residual channel attention modules and a convolution layer; a decoder module, symmetrical in structure to the decoder module, including four upsampling blocks, each upsampling block including two residual channel attention modules and a deconvolution layer; wherein the upsampling blocks are jump-connected to the downsampling blocks to retain key spatial information; a self-attention module, connected between the upsampling blocks, for capturing long-distance dependencies and fine-grained features; an output module, connected to the end of the decoder module, including a 2×2 deconvolution layer, for reducing the number of channels from 64 to 1 to generate a grayscale image.

[0072] In some embodiments of this embodiment, when the training module executes the step of extracting features from an actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model, it is also specifically used to: if the trained super-resolution fluorescence microscopy imaging model detects that the actual fluorescence microscopy image is three-dimensional data, then the two-dimensional convolution in the trained super-resolution fluorescence microscopy imaging model is replaced with a three-dimensional convolution, and then feature extraction is performed on the actual fluorescence microscopy image.

[0073] In some implementations of this embodiment, the preset loss function is a composite loss function including mean square error loss, structural similarity loss, and perceptual loss. The expression of the preset loss function L is:

[0074] L=0.8*L MSE +0.2*L SSIM +0.05*L perceptual

[0075] Among them, L MSE represents the mean square error loss function, LSSIM represents the structural similarity loss function, L perceptual represents the perceptual loss function.

[0076] According to the super-resolution fluorescence microscopy device provided in this embodiment, an actual fluorescence microscopy image is first obtained, and then feature extraction and image reconstruction are performed on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy model is constructed based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module, and the trained super-resolution fluorescence microscopy model is obtained through a two-stage pre-training-fine-tuning training. The present invention is based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module. The model is trained in combination with a two-stage training strategy of pre-training and fine-tuning, so that the model can achieve cross-modal super-resolution migration capabilities under single-sample conditions. When the sample is not in the model database, only a pair of LR-SR corresponding modality data is required for training to reconstruct low-resolution fluorescence microscopy images into super-resolution images. Compared with traditional physical optical devices, this device can achieve super-resolution reconstruction under conditions of lower light power and shorter exposure time, improve the signal-to-noise ratio and reduce the photobleaching effect. At the same time, it has the ability to generalize across systems and organelles, breaking through the bottleneck of traditional deep learning methods that rely on repeated training of large amounts of data, and can significantly reduce resource consumption when replacing different types of imaging samples and systems for super-resolution reconstruction.

[0077] See also Figure 8 , Figure 8 This is a module block diagram of the electronic device provided in an embodiment of the present application.

[0078] like Figure 8 As shown, an embodiment of the present application further provides an electronic device, which can be used to implement the super-resolution fluorescence microscopy imaging method in the aforementioned embodiment, and includes a memory 801 and at least one processor 802; wherein, the memory 801 is used to store at least one program, and when the at least one program is executed by the at least one processor 802, the at least one processor 802 executes the super-resolution fluorescence microscopy imaging method provided in the embodiment of the present application.

[0079] See also Figure 9 , Figure 9 A module block diagram of a computer-readable storage medium provided in an embodiment of the present application.

[0080] like Figure 9 As shown, the embodiment of the present application further provides a computer-readable storage medium 900, on which executable instructions 910 are stored. When the executable instructions 910 are executed, the super-resolution fluorescence microscopy method provided in the embodiment of the present application is executed.

[0081] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0082] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0083] It should be noted that the various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For product-related embodiments, since they are similar to method-related embodiments, their description is relatively simple. For relevant details, refer to the description of the method-related embodiments.

[0084] It should also be noted that, in the present application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0085] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein but is intended to be applied in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A super-resolution fluorescence microscopy method based on deep learning, characterized in that: include: Acquire actual fluorescence microscopy images; Based on the trained super-resolution fluorescence microscopy imaging model, feature extraction and image reconstruction are performed on the actual fluorescence microscopy image to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on the encoder-decoder architecture fusing the residual network, the channel attention mechanism, and the self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage training of pre-training and fine-tuning.

2. The super-resolution fluorescence microscopy method according to claim 1, wherein Before the step of extracting features and reconstructing the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model, the method further includes: Constructing a simulated data set containing simulated image samples, and dividing the data set into a training sample set and a test sample set according to a preset ratio; Randomly selecting the simulated image samples from the training sample set and inputting them into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain training results; Adjusting the network parameters in the original super-resolution fluorescence microscopy imaging model based on the training results and a preset loss function to obtain an adjusted super-resolution fluorescence microscopy imaging model; Randomly selecting the simulated image sample from the test sample set and inputting it into the adjusted super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction to obtain a test result; If the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model; if the test fails, the process returns to the step of randomly selecting the simulated image sample from the training sample set and inputting it into the original super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction.

3. The super-resolution fluorescence microscopy method according to claim 2, wherein: If the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the trained super-resolution fluorescence microscopy imaging model, comprising: If the test passes, the adjusted super-resolution fluorescence microscopy imaging model is used as the pre-trained super-resolution fluorescence microscopy imaging model; Acquire a fine-tuning dataset containing a fluorescence microscopy image sample of a target sample; wherein the target sample is a sample contained in the actual fluorescence microscopy image, and the fine-tuning dataset contains a first fluorescence microscopy image sample and a second fluorescence microscopy image sample, wherein the second fluorescence microscopy image sample is a super-resolution fluorescence microscopy image sample corresponding to the first fluorescence microscopy image sample; Fine-tuning the weight of the last deconvolution layer in the pre-trained super-resolution microscopy imaging model based on the fine-tuning dataset to obtain a fine-tuned super-resolution fluorescence microscopy imaging model; Inputting the first fluorescence microscopy image sample into the fine-tuned super-resolution fluorescence microscopy imaging model for feature extraction and image reconstruction, and comparing and verifying the output result with the second fluorescence microscopy image sample; If the verification passes, the fine-tuned super-resolution fluorescence microscopy model is used as the trained super-resolution fluorescence microscopy model; if the verification fails, the process returns to the step of fine-tuning the weights of the last deconvolution layer in the pre-trained super-resolution fluorescence microscopy model based on the fine-tuning dataset.

4. The super-resolution fluorescence microscopy method according to claim 3, wherein: The step of extracting features from the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model includes: randomly capturing multiple local windows from a single fluorescence microscopy image sample based on the trained super-resolution fluorescence microscopy imaging model, and synchronously extracting features from the captured multiple local windows.

5. The super-resolution fluorescence microscopy method according to claim 4, characterized in that: The super-resolution fluorescence microscopy model includes: An encoder module comprising four downsampling blocks, each of which comprises two residual channel attention modules and a convolutional layer; A decoder module, symmetrical in structure to the decoder module, comprising four upsampling blocks, each of which comprises two residual channel attention modules and a deconvolution layer; wherein the upsampling blocks are skip-connected to the downsampling blocks to preserve key spatial information; A self-attention module, connected between the upsampling blocks, is used to capture long-range dependencies and fine-grained features; The output module, connected to the end of the decoder module, includes a 2×2 deconvolution layer for reducing the number of channels from 64 to 1 to generate a grayscale image.

6. The super-resolution fluorescence microscopy method according to claim 5, characterized in that: The step of extracting features from the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model further includes: If the trained super-resolution fluorescence microscopy imaging model detects that the actual fluorescence microscopy image is three-dimensional data, the two-dimensional convolution in the trained super-resolution fluorescence microscopy imaging model is replaced with a three-dimensional convolution, and then feature extraction is performed on the actual fluorescence microscopy image.

7. The super-resolution fluorescence microscopy method according to claim 2, wherein: The preset loss function is a composite loss function including mean square error loss, structural similarity loss and perceptual loss. The expression of the preset loss function L is: L=0.8*L MSE +0.2*L SSIM +0.05*L perceptual Among them, L MSE represents the mean square error loss function, LSSIM represents the structural similarity loss function, L perceptual Denotes the perceptual loss function.

8. A super-resolution fluorescence microscopy system based on deep learning, characterized in that: include: An acquisition module, used for acquiring actual fluorescence microscopy images; An imaging module is used to perform feature extraction and image reconstruction on the actual fluorescence microscopy image based on the trained super-resolution fluorescence microscopy imaging model to obtain a super-resolution fluorescence microscopy image result; wherein, the super-resolution fluorescence microscopy imaging model is constructed based on an encoder-decoder architecture that integrates a residual network, a channel attention mechanism, and a self-attention module, and the trained super-resolution fluorescence microscopy imaging model is obtained through a two-stage pre-training-fine-tuning training.

9. An electronic device, characterized in that: Comprising a memory and a processor, wherein: The processor is configured to execute a computer program stored in the memory; When the processor executes the computer program, the steps of the super-resolution fluorescence microscopy method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the super-resolution fluorescence microscopy method according to any one of claims 1 to 7 are implemented.

Images