Deconvolutional super-resolution imaging method and device based on deep learning denoising and medium

Abstract

The present disclosure relates to the technical field of biological microscopic imaging, and discloses a deconvolution super-resolution imaging method and device based on deep learning denoising and a medium; the method comprises the following steps: obtaining a to-be-denoised image of a to-be-imaged structure labeled by fluorescent molecules; using a preset image denoising neural network to denoise the to-be-denoised image to obtain a denoised image; the training data set of the network is generated by using the bright and dark states of the fluorescent molecules, and the fluorescent molecules are also used to label the to-be-imaged structure of the samples in the training data set; and performing deconvolution processing on the denoised image to obtain a super-resolution imaging image. The present disclosure uses the bright and dark states of the fluorescent molecules to obtain large-scale and high-quality true value images, thereby improving the denoising effect of the image denoising neural network; the image is denoised by using the network, and then the denoised image is deconvoluted, thereby realizing single-frame super-resolution imaging of a large field of view of living cells.

Description

Technical Field

[0001] This disclosure relates to the field of biological microscopy imaging technology, and more specifically, to a deconvolutional super-resolution imaging method, apparatus, electronic device, and computer-readable storage medium based on deep learning denoising. Background Technology

[0002] Super-resolution imaging technology refers to the process of transforming low-resolution images into high-resolution images using computer algorithms or models. This allows us to see more details and obtain a clearer, more realistic visual experience. Recent super-resolution imaging techniques, such as stimulated emission depletion microscopy (STED), stochastic optical reconstruction microscopy (STORM), and photoactivated localization microscopy (PALM), have broken the limitations of the optical diffraction limit, enabling researchers to study the fine features of subcellular structures at the nanoscale resolution level.

[0003] However, these imaging methods are generally multi-frame super-resolution imaging techniques, which utilize the different spatial information exhibited by the sample over time, sacrificing temporal resolution for spatial resolution, making it difficult to achieve single-frame super-resolution imaging. For example, Structured Illumination Microscopy (SIM), the most suitable super-resolution imaging technique for live cells, requires at least nine frames of raw data to reconstruct a single ultra-high-resolution image. Besides the drawback of sacrificing temporal resolution for spatial resolution, the aforementioned super-resolution imaging techniques are also prone to reconstruction artifacts caused by the rapid movement of the observed molecules; furthermore, multi-frame sampling can easily lead to fluorescent bleaching, posing a significant challenge to live-cell 3D super-resolution imaging. Therefore, developing a large-field-of-view single-frame super-resolution imaging technique that does not sacrifice temporal resolution is of great significance. Summary of the Invention

[0004] In view of the above, this disclosure provides a deconvolutional super-resolution imaging method, apparatus, electronic device, and computer-readable storage medium based on deep learning denoising, which aims to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, embodiments of this disclosure provide a deconvolutional super-resolution imaging method based on deep learning denoising, the method comprising: Acquire an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with fluorescent molecules; The image to be denoised is denoised using a preset image denoising neural network to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules, the light-switching fluorescent molecules are used to label the imaging structures of each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the light-switching fluorescent molecules. The denoised image is deconvolved to obtain the super-resolution imaging image corresponding to the denoised image.

[0006] Secondly, embodiments of this disclosure also provide a deconvolutional super-resolution imaging device based on deep learning denoising, the device comprising: An acquisition module is used to acquire an image to be denoised, wherein the imaging structure of the image to be denoised is labeled with fluorescent molecules. A denoising module is used to denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules, and the light-switching fluorescent molecules are used to label the imaging structures of each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the light-switching fluorescent molecules. The super-resolution imaging module is used to perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0007] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the aforementioned deconvolutional super-resolution imaging method based on deep learning denoising.

[0008] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the steps of the aforementioned deconvolutional super-resolution imaging method based on deep learning denoising.

[0009] By employing the above technical solutions, the deconvolutional super-resolution imaging method, apparatus, and medium based on deep learning denoising provided in this disclosure obtain a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training dataset used by this preset image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules. Utilizing this feature, large-scale, more realistic, and extremely low-noise high-quality ground truth images can be obtained, which is particularly advantageous for deep tissue imaging. Training the network using the training dataset containing these ground truth images can greatly improve the robustness and stability of the preset image denoising neural network. Simultaneously, because light-switching fluorescent molecules can interact with various subcellular... The structure is combined without relying on a specific structure. Therefore, it is not necessary for the imaging structure of the image to be denoised to be the same as or similar to the imaging structure in the training dataset. As long as the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photo-switched fluorescent molecule, the preset image denoising neural network can be applied to denoise the image. Therefore, the preset image denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network. The noise of the denoised image is extremely low. Then, the deconvolution processing of the denoised image can reverse and restore the convolution phenomenon unique to optical microscopes, thereby improving the spatiotemporal resolution of the image and obtaining the fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image. This realizes large field-of-view single-frame super-resolution imaging and solves the defects of existing multi-frame super-resolution imaging methods. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings: Figure 1 A flowchart illustrating the deconvolutional super-resolution imaging method based on deep learning denoising provided in this embodiment of the present disclosure is shown. Figure 2 A flowchart illustrating a deconvolutional super-resolution imaging method based on deep learning denoising, according to another embodiment of this disclosure, is shown. Figure 3 A laser modulation timing diagram provided in an embodiment of this disclosure is shown; Figure 4 This illustration shows a comparison of image processing results provided by embodiments of the present disclosure, specifically deconvolution without denoising and deconvolution after denoising. Figure 5 A schematic diagram of the structure of the deconvolutional super-resolution imaging device based on deep learning denoising provided in an embodiment of this disclosure is shown. Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0012] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0014] As mentioned earlier, existing imaging methods are generally multi-frame super-resolution imaging techniques, which utilize the different spatial information displayed by the sample over time, sacrificing temporal resolution for spatial resolution, making it difficult to achieve single-frame super-resolution imaging. For example, Structured Illumination Microscopy (SIM), the most suitable super-resolution imaging technique for live cells, requires at least nine frames of raw data to reconstruct a single ultra-high-resolution image. Besides the drawback of sacrificing temporal resolution for spatial resolution, the aforementioned super-resolution imaging techniques are also prone to reconstruction artifacts caused by the rapid movement of the observed molecules; furthermore, multi-frame sampling can easily lead to fluorescent bleaching, posing a significant challenge to live-cell 3D super-resolution imaging. Therefore, developing a large-field-of-view single-frame super-resolution imaging technique that does not sacrifice temporal resolution is of great significance.

[0015] Based on this, the embodiments of this disclosure obtain a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training dataset used by this preset image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules. Utilizing this feature, large-scale, more realistic, and extremely low-noise high-quality ground truth images can be obtained, which is particularly advantageous for deep tissue imaging. Training the network using the training dataset containing these ground truth images can greatly improve the robustness and stability of the preset image denoising neural network. Furthermore, since light-switching fluorescent molecules can bind to various subcellular structures without depending on a specific structure, Therefore, it is not necessary for the imaging structure of the image to be denoised to be the same as or similar to the imaging structure in the training dataset. As long as the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photo-switched fluorescent molecule, the preset image denoising neural network can be applied to denoise the image. Therefore, the preset image denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network, making the noise of the denoised image extremely low. Then, deconvolution processing is performed on the denoised image to reverse and restore the convolution phenomenon unique to optical microscopes, thereby improving the spatiotemporal resolution of the image and obtaining the fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image. This realizes large field-of-view single-frame super-resolution imaging and solves the defects of existing multi-frame super-resolution imaging methods.

[0016] To facilitate understanding of this embodiment, a detailed description of the deconvolutional super-resolution imaging method based on deep learning denoising disclosed in this disclosure is provided first. The execution entity of the deconvolutional super-resolution imaging method based on deep learning denoising provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), computing device, wearable device, etc. In some possible implementations, this deconvolutional super-resolution imaging method based on deep learning denoising can be implemented by a processor calling computer-readable instructions stored in memory.

[0017] Figure 1 This illustration shows a flowchart of a deconvolutional super-resolution imaging method based on deep learning denoising provided in an embodiment of this disclosure. Figure 1 It can be seen that the embodiments of this disclosure include at least steps S101-S103: Step S101: Obtain the image to be denoised, and label the structure to be imaged in the image with fluorescent molecules.

[0018] In this step, the structures to be imaged in the denoised image are cellular structures observed through fluorescence microscopy, including but not limited to: endoplasmic reticulum, mitochondria, centrosomes, Golgi apparatus, morphology of the nucleolus, ribosomes, lysosomes, etc. Fluorescent molecules include ordinary fluorescent molecules and light-switching fluorescent molecules with light-switching properties.

[0019] Step S102: Denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules, and the light-switching fluorescent molecules are used to label the structures to be imaged in each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the light-switching fluorescent molecules.

[0020] Research has shown that deep learning-based methods can learn complex image features and noise distribution patterns by building deep learning models and utilizing large-scale training data and parallel computing capabilities, thereby achieving efficient noise removal. Deep learning, through end-to-end training, learns directly from the input image to the output image, reducing reliance on manual feature design and improving the algorithm's robustness and generalization ability. Deep learning denoising methods also have other advantages. For example, deep learning models can automatically adjust network parameters and learning strategies based on different characteristics of the input image and noise types, achieving adaptive denoising for different types of noise; through multi-layer nonlinear transformations, they can model the complex relationship between images and noise, thus better capturing details and texture information in the image; and by utilizing contextual information in the image, i.e., the spatial and intensity correlations around pixels, they can more accurately restore details and structure in the image. Finally, deep learning models can also improve their expressive power and denoising performance by increasing network depth and width, introducing more hidden layers and parameters. In summary, deep learning denoising methods have strong learning capabilities and adaptability, enabling them to better restore details and structure in images and improve image quality and clarity.

[0021] However, in practical applications, such as in the field of biological microscopy imaging, deep learning-based denoising methods require training with a large number of ground-value images. Obtaining large-scale, high-quality ground-value images is difficult, and deep learning network models are typically trained on specific datasets. When applying the trained network model, the content of the image to be processed needs to be the same as or similar to the content of the images in the training dataset. This results in poor robustness, limited generality, and limited applicability of the trained neural network model. Furthermore, many deep learning methods introduce translation invariance during denoising, meaning they are insensitive to the translation of the input image. This can lead to blurring or distortion in the denoising results. Deep learning methods are highly sensitive to perturbations in the input data; even small perturbations can cause significant changes in the output. In denoising tasks, the input image may contain subtle noise, which may prevent deep learning methods from accurately removing the noise. In conclusion, deep learning denoising methods need to develop more effective data augmentation techniques to improve the quality and quantity of ground-value images, and introduce more prior knowledge to enhance the robustness and generality of the model, in order to achieve better denoising results. Therefore, in this step, after obtaining the image to be denoised, a preset image denoising neural network can be used to denoise the image, resulting in a denoised image. Here, the preset image denoising neural network can be a deep learning network model, and the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules. Specifically, light-switching fluorescent molecules are molecules that can undergo reversible changes in structure and properties under light stimulation. Specifically, light-switching fluorescent molecules can switch between bright and dark states under different illumination conditions; that is, under illumination of a specific wavelength, the light-switching fluorescent molecules are in an "on" state (bright or luminescent), and under illumination of another specific wavelength, they are in an "off" state (dark or non-luminescent).

[0022] Step S103: Perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0023] After obtaining the denoised image, deconvolution processing can be performed on the denoised image to obtain the corresponding super-resolution imaging image. In practice, for example, the denoised image can be deconvolved directly to obtain the corresponding super-resolution imaging image, or iterative deconvolution can be used to deconvolve the denoised image. The embodiments of this disclosure do not limit the method of deconvolution processing.

[0024] from Figure 1As can be seen from the method shown, this disclosure innovatively proposes a deconvolutional super-resolution imaging method based on deep learning denoising. By performing deconvolution processing on the denoised image, a super-resolution imaging image corresponding to the denoised image is obtained. The denoised image is the image processed by a preset image denoising neural network. The training dataset used by this preset image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules. Utilizing this feature, large-scale, more realistic, and extremely low-noise high-quality ground truth images can be obtained, which is particularly advantageous for deep tissue imaging. Training the network using the training dataset containing these ground truth images can greatly improve the robustness and stability of the preset image denoising neural network. Simultaneously, because light-switching fluorescent molecules can interact with various subcellular structures... This method combines different structures without relying on any specific structure. Therefore, it does not require the imaging structure of the image to be denoised to be the same as or similar to the imaging structure in the training dataset. It only requires that the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photo-switched fluorescent molecule. The preset image denoising neural network can then be used to denoise the image. Therefore, the preset image denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network. The denoised image has extremely low noise. Then, deconvolution processing is performed on the denoised image to reverse and restore the convolution phenomenon unique to optical microscopes, thereby improving the spatiotemporal resolution of the image and obtaining the fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image. This achieves large field-of-view single-frame super-resolution imaging and solves the defects of existing multi-frame super-resolution imaging methods.

[0025] Furthermore, to better illustrate the process of the deconvolutional super-resolution imaging method based on deep learning denoising, as a refinement and extension of the above embodiments, the present invention provides several optional embodiments, but is not limited thereto, as detailed below: In one possible implementation, the image to be denoised is determined by the following methods A1 / A2 / A3: Method A1: Obtain the original image and use it as the image to be denoised. The original image is obtained by sampling using a camera.

[0026] Method A2: Preprocess the original image and use the processed image as the image to be denoised. The preprocessing methods include: optical lock-in amplification algorithm, Gaussian filtering, mean filtering, Wiener filtering, or deconvolution.

[0027] Method A3: Use the acquired target super-resolution imaging image as the image to be denoised. The target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image, or MSSR super-resolution imaging image, etc.

[0028] In this embodiment, for method A2, the original image sampled by the camera can be preprocessed, and the resulting processed image can be used as the image to be denoised. For example, the original image 1 can be processed using an optical lock-in amplification algorithm to obtain processed image 1, which can be used as the image to be denoised. The original image 2 can also be processed using Gaussian filtering to obtain processed image 2, which can be used as the image to be denoised. The original image 3 can also be processed using mean filtering to obtain processed image 3, which can be used as the image to be denoised. The original image 4 can also be processed using Wiener filtering to obtain processed image 4, which can be used as the image to be denoised. The original image 5 can also be deconvolved to obtain processed image 5, which can be used as the image to be denoised. It should be noted that the examples given here are merely illustrative and do not limit the scope of this application.

[0029] Regarding method A3, image 1 obtained using Airyscan super-resolution imaging technology can be used as the image to be denoised. Image 2 obtained using ISM super-resolution imaging technology can also be used as the image to be denoised. Image 3 obtained using A-POD super-resolution imaging technology can also be used as the image to be denoised. Image 4 obtained using MSSR super-resolution imaging technology can also be used as the image to be denoised. It should be noted that these examples are merely illustrative and do not limit the embodiments of this application.

[0030] In one possible implementation, fluorescent molecules label the structures to be imaged in the image to be denoised using the following method B1 / method B2: Method B1: Fluorescent molecules are used to label the structures to be imaged in the image to be denoised using molecular cloning.

[0031] Method B2: Fluorescent molecules bind to the target antibody to label the structure to be imaged in the image to be denoised, wherein the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0032] In this embodiment, the target antibody can be a conventional antibody or a nanobody. For example, fluorescent molecule A can be fused with the structure to be imaged using molecular cloning (a method for purifying and amplifying specific DNA fragments at the molecular level) and expressed (referring to the construction of a fusion gene by combining a foreign protein gene with the 3' end of another gene for expression) to label the structure to be imaged. Alternatively, fluorescent molecule B can be bound to nanobody A to label the structure to be imaged, with target antibody A specifically binding to the structure. It should be noted that the examples described here are merely illustrative and do not limit the scope of this disclosure.

[0033] In one possible implementation, in step S102 above, the photo-switching fluorescent molecule includes: a fluorescent protein with photo-switching properties, a fluorescent dye with photo-switching properties, or a quantum dot.

[0034] In this embodiment, for fluorescent proteins with light-switching properties, specifically, in one possible implementation, the fluorescent proteins with light-switching properties respond to light modulation and have two states: a bright state and a dark state, including: light-switching fluorescent proteins, reversible light-switching fluorescent proteins, and photo-scintillation fluorescent proteins. In one possible implementation, the light-switching fluorescent proteins include: Skylan-S, rsEGFP2, or Dronpa, etc. Here, Skylan-S (a novel light-switching protein) is a monomer that can be used for live-cell imaging and has extremely high optical stability and switching properties. rsEGFP2 (reversible conversion-enhanced green fluorescent protein 2) has the characteristics of easy crystallization and good optical properties. Dronpa (an erasable optical protein) is a monomeric fluorescent protein extracted from Pectiniidae (a fossilized coral), possessing unique photochromic properties, and its fluorescence can be controlled to turn on and off using two different excitation lights.

[0035] Specifically, in one possible implementation, fluorescent dyes with photo-switching properties include photo-switching fluorescent dyes and photo-scintillation fluorescent dyes. In practice, the fluorescent dyes with photo-switching properties can be, for example, fluorescein, geranium dyes, rhodamine fluorescent dyes, etc. It should be noted that the types of fluorescent proteins and fluorescent dyes with photo-switching properties are not limited in this disclosure.

[0036] Quantum dots, also known as semiconductor nanocrystals, can lock electrons in a very small three-dimensional space, restricting their movement. When a certain electric field or light is applied to them, they will emit light at a specific frequency. The emission frequency changes with the particle size. By adjusting the size of the quantum dots, the color of their emission can be controlled.

[0037] In one possible implementation, the type of light emitted by the photo-switching fluorescent molecules includes blue light, green light, yellow light, red light, or far-infrared wavelengths. This disclosure does not limit the type of light emitted by the photo-switching fluorescent molecules.

[0038] In one possible implementation, in step S102 above, the preset image denoising neural network is generated according to the following method: Step C1: Acquire multiple fluorescence image sequences, each containing at least one bright image and at least one dark image.

[0039] Step C2: Perform first image processing on each bright state image and each dark state image to obtain input images of multiple fluorescence image sequences.

[0040] Step C3: Perform a second image processing on each bright state image and each dark state image to obtain ground truth images of multiple fluorescence image sequences, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the ground truth image.

[0041] Step C4: Generate a training dataset based on each input image and each ground truth image; train the initial image denoising neural network using the training dataset to obtain the preset image denoising neural network.

[0042] In this embodiment, to obtain the preset image denoising neural network, a training dataset can be acquired first. Specifically, multiple fluorescence image sequences can be acquired first. In one possible implementation, the fluorescence image sequences are acquired using optical imaging techniques, including: wide-field fluorescence microscopy, confocal fluorescence microscopy, structured light illumination imaging, single-molecule localization microscopy, stimulated emission loss super-resolution microscopy, optical wave super-resolution imaging, single-photon fluorescence imaging, two-photon microscopy, multi-photon microscopy, or photoacoustic imaging, etc.

[0043] For example, after labeling the structure to be imaged with photo-switched fluorescent molecules, the structure can be illuminated with modulated light. When the photo-switched fluorescent molecules are in an luminescent state, an optical imaging device, such as a wide-field fluorescence microscope, can be used to acquire fluorescence images, resulting in one or more bright-state images. Since the structure to be imaged is labeled with photo-switched fluorescent molecules, each bright-state image contains signal data generated by the structure and noise data, including Gaussian noise, Poisson noise, and sample autofluorescence. When most of the photo-switched fluorescent molecules are in a non-luminescent state, fluorescence images can be acquired, resulting in one or more dark-state images. Each dark-state image contains weak signal data and noise data. Ultimately, multiple fluorescence image sequences can be obtained, such as fluorescence image sequence a: bright-state image a1, dark-state image a2; fluorescence image sequence b: bright-state image b1, bright-state image b2, dark-state image b3; and fluorescence image sequence c: bright-state image c1, bright-state image c2, dark-state image c3, dark-state image c4. The number of fluorescence image sequences is not limited in this embodiment and can be set according to actual conditions; for example, the number can be set to 500. The optical imaging technique used to obtain the fluorescence image sequences is not limited in this embodiment and can be selected according to actual needs.

[0044] After obtaining multiple fluorescence image sequences, a first image processing step can be performed on each bright-state image and each dark-state image to obtain the input image of the fluorescence image sequence. Specifically, in one possible implementation, after obtaining multiple fluorescence image sequences, the first image processing step is performed on each bright-state image and each dark-state image using the following method D1 / method D2 to obtain the input image of the multiple fluorescence image sequences, including: Method D1: Take any one of the bright state images as the input image.

[0045] Method D2: Select any one first fluorescence image pair from multiple fluorescence image sequences, subtract the bright state image from the dark state image in the first fluorescence image pair to obtain the first subtracted image; and use the first subtracted image as the input image.

[0046] In this embodiment, for method D1, any one of the bright images can be used as the input image. For example, if the multiple fluorescence image sequences include bright image 1, bright image 2, and bright image 3, then any one of bright image 1, bright image 2, and bright image 3 can be selected, such as bright image 1, as the input image. It should be noted that this example is merely illustrative and does not limit the scope of this application's embodiments.

[0047] Regarding method D2, a first fluorescent image pair can be selected from multiple fluorescent image sequences. The bright image in the first fluorescent image pair is subtracted from the dark image to obtain the first subtracted image; this first subtracted image is then used as the input image. For example, the multiple fluorescent image sequences include fluorescent image sequence d and fluorescent image sequence e. Fluorescent image sequence d includes bright images d1, d2, d3, and d4, and fluorescent image sequence e includes bright images e1, e2, and e3. A first fluorescent image pair can be selected from each sequence, for example, if the first fluorescent image pair includes bright image d1 and dark image e3. Then, the pixel values ​​of corresponding pixels in bright image d1 and dark image e3 can be subtracted to obtain the first subtracted image, which can then be used as the input image. It should be noted that this example is merely illustrative and does not limit the embodiments of this application.

[0048] Next, a second image processing step can be performed on each bright-state image and each dark-state image to obtain a ground truth image of multiple fluorescence image sequences, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the ground truth image. Specifically, in one possible implementation, after obtaining multiple fluorescence image sequences, a second image processing step is performed on each bright-state image and each dark-state image using the following method E1 / method E2 to obtain a ground truth image of multiple fluorescence image sequences: Method E1: For any number of second fluorescence image pairs in multiple fluorescence image sequences, subtract the bright state image from the dark state image in each second fluorescence image pair to obtain each second subtracted image; generate the true image based on each second subtracted image.

[0049] Method E2: Using an optical lock-in amplification algorithm, multiple fluorescence image sequences are processed to obtain a ground truth image.

[0050] In this embodiment, for method E1, for any number of second fluorescence image pairs in multiple fluorescence image sequences, the pixel values ​​of the corresponding pixels of the bright state image and the dark state image in each second fluorescence image pair can be subtracted to obtain each second subtracted image. Then, a true image is generated based on each second subtracted image. For example, the multiple fluorescence image sequences include a fluorescence image sequence f and a fluorescence image sequence g. For fluorescence image sequence f, the pixel values ​​of the corresponding pixels of the bright state image f1 and the dark state image f2 in fluorescence image sequence f can be subtracted to obtain second subtracted image 1; for fluorescence image sequence g, the pixel values ​​of the corresponding pixels of the bright state image g1 and the dark state image g3 in fluorescence image sequence g can be subtracted to obtain second subtracted image 2. Then, a weighted average can be performed on each second subtracted image to obtain the true image. For example, the weights of second subtracted images 1 and 2 can both be set to 1 / 2, then the true image = 1 / 2 second subtracted image 1 + 1 / 2 second subtracted image 2. The weights of each of the second subtracted images are not limited in this embodiment and can be set according to actual needs. It should be noted that the examples here are merely illustrative and do not limit the embodiments of this disclosure.

[0051] For method E2, for example, an optical lock-in amplification algorithm (OLID algorithm) can be used to process the fluorescence image sequence h, fluorescence image sequence i, fluorescence image sequence j, and fluorescence image sequence k to obtain the ground truth image. It should be noted that the examples here are merely illustrative and do not limit the embodiments of this disclosure.

[0052] Then, a training dataset can be generated based on each input image and each ground truth image. In practice, for example, 1000 frames of images can be acquired using an image acquisition device. Assuming that one input image and one ground truth image can be generated from 20 fluorescence image sequences, the 1000 frames can be divided into: fluorescence image sequence group 1, fluorescence image sequence group 2, ..., fluorescence image sequence group 50, according to the order of acquisition. Input image 1 and ground truth image 1 can be generated from fluorescence image sequence group 1; input image 2 and ground truth image 2 can be generated from fluorescence image sequence group 2; ...; input image 50 and ground truth image 50 can be generated from fluorescence image sequence group 50. Each input image and ground truth image is a training pair, and 50 training pairs constitute the training dataset.

[0053] After obtaining the training dataset, the initial image denoising neural network can be trained using the training dataset to obtain the preset image denoising neural network. In a specific application scenario, one possible implementation is that the deep learning model on which the initial image denoising neural network is based includes: supervised models, unsupervised models, or self-supervised models, etc.; wherein, the supervised models include DnCNN models or Noise2Clean models, etc.; the unsupervised models include BM3D models or BM4D models, etc.; and the self-supervised models include Deep CAD models, Deep CAD-RT models, Noise2Noise models, Noise2Void models, Self2Self models, Self2Self+ models, Neighbor2Neighbor models, Noise2Fast models, Dilated Blind-Spot Network models, blind2unblind models, or Deep Image Prior models, etc.

[0054] Here, the Denoising Convolutional Neural Network (DnCNN) model is a deep residual network specifically designed for image denoising. It learns noise features in an image and removes noise by stacking multiple convolutional layers and residual connections.

[0055] The 3D block matching model (BM3D) can perform simple denoising by matching the original image to form a basic estimate. Then, more detailed denoising can be performed using the original image and the basic estimate to further improve the peak signal-to-noise ratio. The 4D block matching filtering model (BM4D model) is similar in principle to the BM3D model, and can extend denoising in 2D space to 3D space.

[0056] The self-supervised deep learning model (DeepCAD) requires no high SNR observation images; it can be trained using only a single low SNR calcium imaging sequence, suppressing detection noise and improving the SNR by more than 10 times. A self-supervised deep learning model for real-time noise suppression (DeepCAD-RT) has significant advantages in functional imaging. Noise2Noise uses paired noisy images of the same scene as training image pairs, achieving the same training results as under supervised conditions. The Noise2Void model can use the noisy image itself as supervision, preventing the learning of identity mappings through a blind spot network. A self-supervised single-image denoising network model (Self2Self model) only requires a single noisy image (without ground truth) for training. The Self2Self+ model is an optimization of the Self2Self model. The Neighbor2Neighbor model, based on the Noise2Noise model, replaces noise-clean image pairs with noise-noise image pairs, solving the problem of collecting large amounts of noise-clean image data pairs. Self-supervised image denoising models with visible blind spots (blind2unblind) can overcome the information loss in blind spot-driven denoising methods. Dilated Blind-Spot Network (D-BSN) models can learn denoising models solely from real, noisy images. DeepImage Prior models are image denoising methods that utilize the implicit prior information inherent in the structure of neural networks.

[0057] In practice, other deep learning models can also be used to generate a neural network for denoising images to be trained. This disclosure does not limit this approach. For example, a neural network for denoising images to be trained can also be generated based on a Recorded2Recorrupted model.

[0058] In this embodiment, the initial image denoising neural network generated based on the above-mentioned deep learning model can be used to train the training dataset. The deep learning model can efficiently remove various noises in the imaging process, such as Gaussian noise and Poisson noise, thereby greatly improving the denoising effect of the preset image denoising neural network.

[0059] In practice, each input image in the training sample set can be input into the initial image denoising neural network to obtain multiple predicted images. Then, based on the preset loss function, the parameters in the initial image denoising neural network are updated according to the multiple predicted images and the corresponding ground truth images to obtain the preset image denoising neural network.

[0060] In one possible implementation, the denoised image is deconvolved to obtain the corresponding super-resolution imaging image, specifically including: Step F: Based on the Richardson-Lucy algorithm, deconvolve the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0061] Research has shown that for blurry images with no noise, the original image can be completely recovered using the Richardson-Lucy (RL) deconvolution method. However, in actual imaging processes, due to uncertainties such as optical system noise and point spread function (PSF), the RL deconvolution algorithm often results in numerous artifacts in the image. Moreover, as the number of deconvolution iterations increases, the noise signal is amplified, further introducing more artifacts into the image.

[0062] Based on this, in this embodiment, the denoised image can be deconvolved using the Richardson-Lucy algorithm to obtain the corresponding super-resolution imaging image. Since the denoised image is processed by a preset image denoising neural network, and the training dataset used by the preset image denoising neural network is obtained by utilizing the light-switching characteristics of light-switching fluorescent molecules, the preset image denoising neural network has strong robustness, stability, and versatility, and the denoising effect is greatly improved. Therefore, the noise of the denoised image is extremely low. Thus, by using the RL algorithm to deconvolve the denoised image, the original image can be completely restored, improving the spatiotemporal resolution of the image, thereby achieving the purpose of single-frame super-resolution imaging.

[0063] In practice, the system point spread function can be determined according to the following formula (1): (1); in, The normalization coefficient is... Let λ be the Bessel function of the first kind, NA be the wavelength, NA be the numerical aperture, and ρ be the polar radius in polar coordinates. When the fluorescence emitted by the sample passes through the optical system, the image formed on the camera due to diffraction is the convolution of the fluorescence emitted by the sample and the system's point spread function. Therefore, to better reflect the diffraction process of the optical system, the system's point spread function can be represented by a Bessel function of the first kind.

[0064] Considering the impact of noise in imaging, too many deconvolution iterations will produce artifacts, while too few iterations will not improve the image resolution well. Therefore, the number of deconvolution iterations can be determined based on the quality of the denoised image. Specifically, the number of iterations can be determined by calculating the Fourier ring correlation resolution (FRC resolution) of two consecutive frames in the actual sample to ensure that the image resolution reaches the highest level without obvious reconstruction artifacts.

[0065] Specifically, in one possible implementation, the imaging objects of the deep learning-based deconvolutional super-resolution imaging method include: fixed cells, live cells, tissues, live animals, fungi, or fluorescent standards, etc. This disclosure does not limit the imaging objects of the method. This disclosure can achieve rapid 3D super-resolution imaging for fixed cells, and for live cell tissues, it can obtain a series of dynamic images that change over time to reveal rapid dynamic changes in subcellular structures.

[0066] Figure 2 This illustration shows a flowchart of a deconvolutional super-resolution imaging method based on deep learning denoising, according to another embodiment of this disclosure. Figure 2 As can be seen, this embodiment includes the following steps S201-S209: Step S201: Using the optically switched molecular labeling structure, acquire multiple fluorescence image sequences, each including at least one bright-state image and at least one dark-state image. In practice, the fluorescence image sequences are acquired using optical imaging techniques, including: wide-field fluorescence microscopy, confocal fluorescence microscopy, structured light illumination imaging, single-molecule localization microscopy, stimulated emission loss super-resolution microscopy, optical wave super-resolution microscopy, single-photon fluorescence imaging, two-photon microscopy, multi-photon microscopy, or photoacoustic imaging, etc.

[0067] The following describes how to obtain multiple fluorescence image sequences of a target molecule, using a photo-switching fluorescent protein as an example. First, F-actin was fused with photo-switching fluorescent protein (RSFP) and expressed in eukaryotic cells: human osteosarcoma cells U-2 OS, via cell transfection technology (a technique for introducing exogenous molecules such as DNA and RNA into eukaryotic cells). After 36 hours, the cytoskeleton was fixed with fixative, and fluorescence images were acquired using a self-made or commercial fluorescence microscopy platform. The microscope used was model A, employing 405 nm and 488 nm lasers, with a 150×, 1.45NA objective lens; the camera was model B, with a pixel size of 6.5 μm. Figure 3 A laser modulation timing diagram provided in an embodiment of this disclosure is shown. See also... Figure 3 The laser modulation mode shown allows for the simultaneous activation of both 405 nm and 488 nm lasers. After image acquisition begins, the 405 nm laser is deactivated to obtain an image sequence. Conversely, the 488 nm laser is activated, and after image acquisition begins, the 405 nm laser is activated again to obtain another image sequence. Ultimately, a series of fluorescence images with repeatedly changing on / off states can be obtained, resulting in multiple fluorescence image sequences.

[0068] Step S202: Take any one of the bright images as the input image. In practice, a first pair of fluorescence images can be selected from multiple fluorescence image sequences, and the bright image in the first pair of fluorescence images can be subtracted from the dark image to obtain the first subtracted image; and the first subtracted image can be used as the input image.

[0069] Step S203: For any number of second fluorescence image pairs in multiple fluorescence image sequences, subtract the bright state image from the dark state image in each second fluorescence image pair to obtain each second subtracted image.

[0070] Step S204: Generate the ground truth image based on each of the second subtraction images. In practice, an optical lock-in amplification algorithm can also be used to process multiple fluorescence image sequences to obtain the ground truth image. Note that the signal-to-noise ratio of the input image is less than that of the ground truth image.

[0071] Step S205: Generate a training dataset based on the input image and the ground truth image.

[0072] Step S206: Train the initial image denoising neural network using the training dataset to obtain the preset image denoising neural network. Here, the training dataset is generated using the light-switching properties of light-switching fluorescent molecules, which are used to label the structures to be imaged in each sample of the training dataset. Light-switching fluorescent molecules include: fluorescent proteins with light-switching properties, fluorescent dyes with light-switching properties, or quantum dots. Among them, fluorescent proteins with light-switching properties respond to light modulation and have two states: bright and dark. These include: light-switching fluorescent proteins, reversible light-switching fluorescent proteins, or photo-scintillation fluorescent proteins, etc. Light-switching fluorescent proteins include: Skylan-S, rsEGFP2, or Dronpa, etc. Fluorescent dyes with light-switching properties include: light-switching dyes or photo-scintillation fluorescent dyes, etc. The types of light emitted by the light-switching fluorescent molecules include: blue light, green light, yellow light, red light, or far-infrared wavelengths, etc. The deep learning models upon which the initial image denoising neural network is based include: supervised models, unsupervised models, or self-supervised models; the supervised models include DnCNN models or Noise2Clean models; the unsupervised models include BM3D models or BM4D models; and the self-supervised models include Deep CAD models, Deep CAD-RT models, Noise2Noise models, Noise2Void models, Self2Self models, Self2Self+ models, Neighbor2Neighbor models, Noise2Fast models, Dilated Blind-SpotNetwork models, blind2unblind models, or Deep Image Prior models.

[0073] Step S207: Preprocess the original image obtained by camera sampling, and use the processed image as the image to be denoised. Preprocessing methods include: optical lock-in amplification algorithm, Gaussian filtering, mean filtering, Wiener filtering, or deconvolution. Alternatively, the original image can be used directly as the image to be denoised. The acquired target super-resolution imaging image can also be used as the image to be denoised, including Airyscan super-resolution imaging images, ISM super-resolution imaging images, A-POD super-resolution imaging images, or MSSR super-resolution imaging images. The structure to be imaged in the image to be denoised is labeled with fluorescent molecules. For example, fluorescent molecules can be used to label the structure to be imaged in the image to be denoised using molecular cloning. Another example is that fluorescent molecules can bind to target antibodies to label the structure to be imaged in the image to be denoised, where the target antibody specifically binds to the structure to be imaged in the image to be denoised. In practice, the imaging object, i.e., the structure to be imaged, includes: fixed cells, live cells, tissues, live animals, fungi, or fluorescent standards.

[0074] Step S208: Denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image. Here, the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photo-switched fluorescent molecule.

[0075] Step S209: Based on the Richardson-Lucy algorithm, perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0076] Figure 4 This diagram illustrates a comparison of image processing results provided in this disclosure: direct deconvolution without denoising and deconvolution after denoising. In the diagram, 'a' corresponds to the wide-field image (top left), the image obtained by direct deconvolution of the wide-field image (middle right), and the image obtained by deconvolution after denoising (top right). 'b' is a magnified view of 'a' (scale bar: 2μm (a); 1μm (b)). 'c' is a comparison of image resolution between direct deconvolution without denoising and deconvolution after denoising. See also... Figure 4 As shown in Figures a and b, the image after deconvolution processing is clearer, indicating higher resolution and fewer artifacts. Figure c shows that the image after deconvolution processing has a lower nm number, indicating higher resolution. Therefore, the image denoising neural network based on the optical switching properties of optically switching molecules can effectively remove noise from images. Further deconvolution processing significantly improves the spatiotemporal resolution of the denoised image, resulting in a super-resolution imaging image and enabling large-field-of-view single-frame super-resolution imaging.

[0077] Those skilled in the art will understand that in the methods described in the specific embodiments, the order in which the steps are written does not imply a strict execution order, nor does it constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. It should be noted that in practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this disclosure, which will not be elaborated here.

[0078] Based on the same concept, this disclosure also provides a deconvolutional super-resolution imaging device based on deep learning denoising. Figure 5 A schematic diagram of the structure of the deconvolutional super-resolution imaging device based on deep learning denoising provided in this disclosure embodiment is shown. See also: Figure 5 As shown, the deconvolutional super-resolution imaging device 500 based on deep learning denoising provided in this embodiment includes: The acquisition module 501 is used to acquire the image to be denoised, wherein the imaging structure of the image to be denoised is labeled with fluorescent molecules. The denoising module 502 is used to denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules, and the light-switching fluorescent molecules are used to label the imaging structures of each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the light-switching fluorescent molecules. The super-resolution imaging module 503 is used to perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0079] In one possible implementation, in the above-described device, the light-switching fluorescent molecule includes: a fluorescent protein with light-switching properties, a fluorescent dye with light-switching properties, or a quantum dot.

[0080] In one possible implementation, in the above-described device, the fluorescent protein with light-switching properties responds to light modulation and has two states: a bright state and a dark state, including: a light-switching fluorescent protein, a reversible light-switching fluorescent protein, or a light-scintillation fluorescent protein, etc.

[0081] In one possible implementation, the photo-switching fluorescent protein in the above-described device includes Skylan-S, rsEGFP2, or Dronpa, etc.

[0082] In one possible implementation, the fluorescent dye with light-switching properties in the above-described device includes light-switching dyes or light-scintillation fluorescent dyes, etc.

[0083] In one possible implementation, in the above-described device, the types of light emitted by the optically switched fluorescent molecules include: blue light, green light, yellow light, red light, or far-infrared wavelengths.

[0084] In one possible implementation, the apparatus further includes a network training module, the network training module being used for: Acquire multiple fluorescence image sequences, each fluorescence image sequence including at least one bright state image and at least one dark state image; The bright and dark images are subjected to a first image processing step to obtain the input image of the plurality of fluorescence image sequences; A second image processing is performed on each of the bright-state images and each of the dark-state images to obtain a ground truth image of the plurality of fluorescence image sequences, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the ground truth image; The training dataset is generated based on each input image and each ground truth image; the initial image denoising neural network is trained using the training dataset to obtain the preset image denoising neural network.

[0085] In one possible implementation, the network training module, when performing a first image processing on each of the bright-state images and each of the dark-state images to obtain the input image of the plurality of fluorescence image sequences, is used to: Take any one of the bright images as the input image; or Select a first pair of fluorescence images from the plurality of fluorescence image sequences, subtract the bright image from the dark image in the first pair of fluorescence images to obtain a first subtracted image; and use the first subtracted image as the input image.

[0086] In one possible implementation, the network training module, when performing second image processing on each of the bright-state images and each of the dark-state images to obtain the ground truth images of the plurality of fluorescence image sequences, is used to: For any multiple pairs of second fluorescence image sequences, subtract the bright-state image from the dark-state image in the second fluorescence image pair to obtain a second subtracted image; generate the true image based on each of the second subtracted images; or The multiple fluorescence image sequences are processed using an optical lock-in amplification algorithm to obtain the ground truth image.

[0087] In one possible implementation, the deep learning model upon which the initial image denoising neural network is based in the aforementioned network training module includes: supervised models, unsupervised models, or self-supervised models; wherein, the supervised models include DnCNN models or Noise2Clean models; the unsupervised models include BM3D models or BM4D models; and the self-supervised models include Deep CAD models, Deep CAD-RT models, Noise2Noise models, Noise2Void models, Self2Self models, Self2Self+ models, Neighbor2Neighbor models, Noise2Fast models, DilatedBlind-Spot Network models, blind2unblind models, or Deep Image Prior models.

[0088] In one possible implementation, in the above-described apparatus, the acquisition module 501 is used for: Acquire the original image and use it as the image to be denoised, wherein the original image is obtained using camera sampling; or The original image is preprocessed, and the resulting processed image is used as the image to be denoised. The preprocessing methods include: optical lock-in amplification, Gaussian filtering, mean filtering, Wiener filtering, or deconvolution; or... The acquired target super-resolution imaging image is used as the image to be denoised, wherein the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image or MSSR super-resolution imaging image, etc.

[0089] In one possible implementation, in the above-described apparatus, the fluorescent molecules are used to label the structures to be imaged in the image to be denoised by the following method: The fluorescent molecules are used to label the structures to be imaged in the image to be denoised using molecular cloning methods; or The fluorescent molecule binds to the target antibody to label the structure to be imaged in the image to be denoised, wherein the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0090] In one possible implementation, in the above-described apparatus, the super-resolution imaging module 503 is used for: Based on the Richardson-Lucy algorithm, the denoised image is deconvolved to obtain the super-resolution imaging image corresponding to the denoised image.

[0091] In one possible implementation, in the above-mentioned network training module, the fluorescence image sequence is acquired through optical imaging technology, including: wide-field fluorescence microscopy, confocal fluorescence microscopy, structured light illumination imaging, single-molecule localization microscopy, stimulated emission loss super-resolution microscopy, optical wave super-resolution imaging, single-photon fluorescence imaging, two-photon microscopy, multi-photon microscopy, or photoacoustic imaging, etc.

[0092] In one possible implementation, the imaging object of the method device includes: fixed cells, live cells, tissues, live animals, fungi, or fluorescent standards, etc.

[0093] It should be noted that any of the aforementioned deep learning-based denoising deconvolutional super-resolution imaging devices can implement the aforementioned deep learning-based denoising deconvolutional super-resolution imaging method one by one, which will not be elaborated here.

[0094] Based on the same technical concept, this disclosure also provides an electronic device. (See also...) Figure 6 The diagram shows the structure of an electronic device provided in this embodiment, including a processor 601, a memory 602, and a bus 603. The memory 602 stores execution instructions and includes a main memory 6021 and an external memory 6022. The main memory 6021, also called internal memory, is used to temporarily store computational data in the processor 601 and data exchanged with external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the main memory 6021. When the electronic device 600 is running, the processor 601 and the memory 602 communicate through the bus 603, causing the processor 601 to execute the following instructions: Acquire an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with fluorescent molecules; The image to be denoised is denoised using a preset image denoising neural network to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the light-switching properties of light-switching fluorescent molecules, the light-switching fluorescent molecules are used to label the imaging structures of each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the light-switching fluorescent molecules. The denoised image is deconvolved to obtain the super-resolution imaging image corresponding to the denoised image.

[0095] The specific processing flow of the processor 601 can be referred to the description in the above method embodiments, and will not be repeated here.

[0096] Furthermore, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the deep learning-based denoising deconvolutional super-resolution imaging method described in the above-described method embodiments. The storage medium can be either volatile or non-volatile computer-readable storage.

[0097] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the deconvolutional super-resolution imaging method based on deep learning denoising in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0098] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0102] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A deconvolutional super-resolution imaging method based on deep learning denoising, characterized in that, The method includes: Acquire an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with fluorescent molecules; The image to be denoised is denoised using a preset image denoising neural network to obtain a denoised image. The training dataset for training the image denoising neural network is generated using the bright and dark states of photo-switching fluorescent molecules. These photo-switching fluorescent molecules are used to label the structures to be imaged in each sample of the training dataset. The emission spectra of the fluorescent molecules overlap with those of the photo-switching fluorescent molecules. The photo-switching fluorescent molecules are fluorescent molecules that can switch between bright and dark states under different lighting conditions. The training dataset includes multiple samples, each containing an input image and a ground truth image. The input image and the ground truth image are generated based on the bright and dark state images of the structure to be imaged, and the signal-to-noise ratio of the ground truth image is higher than that of the input image. The denoised image is deconvolved to obtain the super-resolution imaging image corresponding to the denoised image.

2. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The light-switching fluorescent molecules include: fluorescent proteins with light-switching properties, fluorescent dyes with light-switching properties, or quantum dots.

3. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 2, characterized in that, The fluorescent proteins with light-switching properties respond to light modulation and have two states: a bright state and a dark state. They include: light-switching fluorescent proteins, reversible light-switching fluorescent proteins, or light-scintillation fluorescent proteins.

4. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 3, characterized in that, The light-switching fluorescent proteins include Skylan-S, rsEGFP2, or Dronpa.

5. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 2, characterized in that, The fluorescent dyes with light-switching properties include: light-switching dyes or light-scintillation fluorescent dyes.

6. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The types of light emitted by the optically switched fluorescent molecules include: blue light, green light, yellow light, red light, or far-infrared wavelengths.

7. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The preset image denoising neural network is generated according to the following method: Acquire multiple fluorescence image sequences, each fluorescence image sequence including at least one bright state image and at least one dark state image; The bright and dark images are subjected to a first image processing step to obtain the input image of the plurality of fluorescence image sequences; A second image processing is performed on each of the bright-state images and each of the dark-state images to obtain a ground truth image of the plurality of fluorescence image sequences, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the ground truth image; The training dataset is generated based on each input image and each ground truth image; the initial image denoising neural network is trained using the training dataset to obtain the preset image denoising neural network.

8. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 7, characterized in that, The first image processing step, which involves processing each of the bright and dark images to obtain the input image of the plurality of fluorescence image sequences, includes: Take any one of the bright images as the input image; or Select a first pair of fluorescence images from the plurality of fluorescence image sequences, subtract the bright image from the dark image in the first pair of fluorescence images to obtain a first subtracted image; and use the first subtracted image as the input image.

9. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 7, characterized in that, The step of performing a second image processing on each of the bright-state images and each of the dark-state images to obtain the ground truth image of the plurality of fluorescence image sequences includes: For any pair of second fluorescence images in the plurality of fluorescence image sequences, the bright state image and the dark state image in each pair of second fluorescence image pairs are subtracted to obtain each second subtracted image; the weighted average of each second subtracted image is then performed to generate the true image; or The multiple fluorescence image sequences are processed using an optical lock-in amplification algorithm to obtain the ground truth image.

10. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 7, characterized in that, The deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model, or a self-supervised model; The supervised model includes the DnCNN model or the Noise2Clean model; the unsupervised model includes the BM3D model or the BM4D model; and the self-supervised model includes the Deep CAD model, the Deep CAD-RT model, the Noise2Noise model, the Noise2Void model, the Self2Self model, the Self2Self+ model, the Neighbor2Neighbor model, the Noise2Fast model, the Dilated Blind-Spot Network model, the blind2unblind model, or the Deep ImagePrior model.

11. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The image to be denoised is determined according to the following method: Acquire the original image and use it as the image to be denoised, wherein the original image is obtained using camera sampling; or The original image is preprocessed, and the resulting processed image is used as the image to be denoised. The preprocessing methods include: optical lock-in amplification, Gaussian filtering, mean filtering, Wiener filtering, or deconvolution; or... The acquired target super-resolution imaging image is used as the image to be denoised, wherein the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image or MSSR super-resolution imaging image.

12. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The fluorescent molecules are used to label the structures to be imaged in the image to be denoised using the following method: The fluorescent molecules are used to label the structures to be imaged in the image to be denoised using molecular cloning methods; or The fluorescent molecule binds to the target antibody to label the structure to be imaged in the image to be denoised, wherein the target antibody specifically binds to the structure to be imaged in the image to be denoised.

13. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that, The step of performing deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image includes: Based on the Richardson-Lucy algorithm, the denoised image is deconvolved to obtain the super-resolution imaging image corresponding to the denoised image.

14. The deconvolutional super-resolution imaging method based on deep learning denoising according to claim 7, characterized in that, The fluorescence image sequence is acquired through optical imaging techniques, including: wide-field fluorescence microscopy, confocal fluorescence microscopy, structured light illumination imaging, single-molecule localization microscopy, stimulated emission loss super-resolution microscopy, optical wave super-resolution microscopy, single-photon fluorescence imaging, two-photon microscopy, multi-photon microscopy, or photoacoustic imaging.

15. The deconvolutional super-resolution imaging method based on deep learning denoising according to any one of claims 1-14, characterized in that, The imaging targets of the method include: fixed cells, live cells, tissues, live animals, fungi, or fluorescent standards.

16. A deconvolutional super-resolution imaging device based on deep learning denoising, characterized in that, The device includes: An acquisition module is used to acquire an image to be denoised, wherein the imaging structure of the image to be denoised is labeled with fluorescent molecules. A denoising module is used to denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image. The training dataset for training the image denoising neural network is generated using the bright and dark states of photo-switching fluorescent molecules. These photo-switching fluorescent molecules are used to label the structures to be imaged in each sample of the training dataset. The emission spectra of the fluorescent molecules overlap with those of the photo-switching fluorescent molecules. The photo-switching fluorescent molecules are fluorescent molecules that can switch between bright and dark states under different lighting conditions. The training dataset includes multiple samples, each containing an input image and a ground truth image. The input image and the ground truth image are generated based on the bright and dark state images of the structure to be imaged, and the signal-to-noise ratio of the ground truth image is higher than that of the input image. The super-resolution imaging module is used to perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

17. An electronic device comprising: processor; And a memory arranged to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the steps of the method as described in any one of claims 1-15.

18. A computer-readable storage medium storing one or more programs, characterized in that, When the one or more programs are executed by an electronic device including multiple applications, the electronic device causes the electronic device to perform the steps of the method as described in any one of claims 1-15.

Images