Deconvolution-based super-resolution imaging method and apparatus by means of deep learning-based denoising, and medium

Abstract

The present disclosure relates to the technical field of biological microscopic imaging. Disclosed are a deconvolution-based super-resolution imaging method and apparatus by means of deep learning-based denoising, and a medium. The method comprises: acquiring an image to be denoised having a structure to be imaged thereof marked by a fluorescent molecule; using a preset image denoising neural network to denoise the image to be denoised to obtain a denoised image, wherein a training data set for training the network is generated by using the photoswitching characteristic of a photoswitchable fluorescent molecule, the photoswitchable fluorescent molecule is used for marking the structure to be imaged of each sample in the training data set, and an emission spectrum of the fluorescent molecule overlaps an emission spectrum of the photoswitchable fluorescent molecule; and performing deconvolution processing on the denoised image to obtain a super-resolution image. According to the present disclosure, a large-scale and high-quality ground truth image is obtained by using the photoswitching characteristic of the photoswitchable fluorescent molecule, thereby enhancing the denoising effect of the image denoising neural network; the image is denoised by using the network, and then deconvolution is performed on the denoised image, thereby achieving single-frame super-resolution imaging for live cells over a large field of view.

Description

Deconvolution super-resolution imaging method, device and medium based on deep learning denoising

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 28, 2023, with application number 202311839953.2 and application name “Deconvolution super-resolution imaging method, device and medium based on deep learning denoising”, the entire contents of which are incorporated by reference into the application. Technical Field

[0002] The present disclosure relates to the field of biological microscopy imaging technology, and in particular to a deconvolution super-resolution imaging method, device, electronic device, and computer-readable storage medium based on deep learning denoising. Background Art

[0003] Super-resolution imaging technology, which uses computer algorithms or models to convert low-resolution images into high-resolution images, allows us to see more details and obtain a clearer, more realistic visual experience. In recent years, various super-resolution imaging technologies, such as stimulated emission depletion microscopy (STED), stochastic optical reconstruction microscopy (STORM), and photoactivated localization microscopy (PALM), have broken the limitations of the optical diffraction limit, enabling researchers to study the fine features of subcellular structures in cells at nanometer-scale resolution.

[0004] However, these imaging methods are generally multi-frame super-resolution imaging technologies, which use the different spatial information displayed by the sample in a time series, sacrificing temporal resolution in exchange for spatial resolution, making it difficult to achieve single-frame super-resolution imaging. For example, the structured illumination microscopy (SIM), which is most suitable for super-resolution imaging of living cells, also requires at least 9 frames of raw data to reconstruct an ultra-high-resolution image. In addition to the defect of sacrificing temporal resolution in exchange for spatial resolution, the aforementioned super-resolution imaging technologies are also prone to reconstruction artifacts caused by the rapid movement of the molecules to be observed; in addition, multi-frame sampling can easily lead to bleaching of fluorescent molecules, making 3D super-resolution imaging of living cells face huge challenges. Therefore, it is of great significance to develop a large-field single-frame super-resolution imaging technology that does not sacrifice temporal resolution.

[0005] Contents of this application

[0006] In response to the above situation, the embodiments of the present disclosure provide a deconvolution super-resolution imaging method, device, electronic device and computer-readable storage medium based on deep learning denoising, which aims to solve the above problems or at least partially solve the above problems.

[0007] In a first aspect, an embodiment of the present disclosure provides a deconvolution super-resolution imaging method based on deep learning denoising, the method comprising:

[0008] Acquiring an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule;

[0009] Denoising the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein a training data set for training the image denoising neural network is generated using the photoswitchable properties of a photoswitchable fluorescent molecule, the photoswitchable fluorescent molecule being used to mark the structure to be imaged of each sample in the training data set; and the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photoswitchable fluorescent molecule;

[0010] Deconvolution is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0011] In a second aspect, the present disclosure further provides a deconvolution super-resolution imaging device based on deep learning denoising, the device comprising:

[0012] An acquisition module is used to acquire an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule;

[0013] a denoising module, configured to denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein a training data set for training the image denoising neural network is generated using the photoswitchable properties of a photoswitchable fluorescent molecule, the photoswitchable fluorescent molecule being used to mark the structure to be imaged of each sample in the training data set; and the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photoswitchable fluorescent molecule;

[0014] The super-resolution imaging module is used to perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device performs the steps of the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising.

[0017] By means of the above technical solution, the deconvolution super-resolution imaging method, device and medium based on deep learning denoising provided by the embodiments of the present disclosure obtain a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training data set used by the preset image denoising neural network is generated by utilizing the optical switching characteristics of the optical switch fluorescent molecules. By utilizing this feature, large-scale, high-quality true value images that are more consistent with the real image and have extremely low noise can be obtained, which has great advantages especially for deep tissue imaging. By training the network with the training data set containing the true value image, the robustness and stability of the preset image denoising neural network can be greatly improved. At the same time, since the optical switch fluorescent molecules can be combined with various subcellular The denoising neural network can be used to denoise the denoised image without relying on a specific structure. The denoising neural network can be used to denoise the denoised image only if the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switch fluorescent molecule. Therefore, the denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network and makes the noise of the denoised image extremely low. The denoised image is then deconvolved to reversely infer and restore the convolution phenomenon unique to the optical microscope, thereby improving the spatiotemporal resolution of the image and obtaining a fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image, realizing large-field single-frame super-resolution imaging, and solving the defects of the multi-frame super-resolution imaging method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0019] FIG1 is a schematic diagram showing a flow chart of a deconvolution super-resolution imaging method based on deep learning denoising provided by an embodiment of the present disclosure;

[0020] FIG2 is a schematic diagram showing a flow chart of a deconvolution super-resolution imaging method based on deep learning denoising provided by another embodiment of the present disclosure;

[0021] FIG3 shows a laser modulation timing diagram provided by an embodiment of the present disclosure;

[0022] FIG4 shows a comparison diagram of image processing results of direct deconvolution without denoising and deconvolution after denoising provided by an embodiment of the present disclosure;

[0023] FIG5 shows a schematic structural diagram of a deconvolution super-resolution imaging device based on deep learning denoising provided by an embodiment of the present disclosure;

[0024] FIG6 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0026] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0027] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in sequences other than those illustrated or described herein. In addition, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to."

[0028] As mentioned above, existing imaging methods are generally multi-frame super-resolution imaging technologies, which use the different spatial information displayed by the sample in a time series, sacrificing temporal resolution in exchange for spatial resolution, making it difficult to achieve single-frame super-resolution imaging. For example, the structured illumination microscopy (SIM), which is most suitable for super-resolution imaging of living cells, also requires at least 9 frames of raw data to reconstruct one ultra-high-resolution image. In addition to the defect of sacrificing temporal resolution in exchange for spatial resolution, the aforementioned various super-resolution imaging technologies are also prone to reconstruction artifacts caused by the rapid movement of the molecules to be observed; in addition, multi-frame sampling can easily lead to bleaching of fluorescent molecules, making 3D super-resolution imaging of living cells face huge challenges. Therefore, it is of great significance to develop a large-field single-frame super-resolution imaging technology that does not sacrifice temporal resolution.

[0029] Based on this, the embodiment of the present disclosure obtains a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training data set used by the preset image denoising neural network is generated by utilizing the photoswitching characteristics of photoswitchable fluorescent molecules. By utilizing this feature, large-scale, high-quality true-value images that are more consistent with real images and have extremely low noise can be obtained. This has great advantages, especially for deep tissue imaging. Training the network using the training data set containing the true-value images can greatly improve the robustness and stability of the preset image denoising neural network. At the same time, since photoswitchable fluorescent molecules can bind to various subcellular structures without relying on a specific structure, Therefore, it is not necessary for the structure to be imaged of the denoised image to be the same or similar to the structure to be imaged in the training data set. Instead, it is only necessary for the emission spectrum of the fluorescent molecule to overlap with the emission spectrum of the photoswitch fluorescent molecule. The preset image denoising neural network can be applied to denoise the denoised image. Therefore, the preset image denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network, making the noise of the denoised image extremely low. Deconvolution processing is then performed on the denoised image to reverse and restore the convolution phenomenon unique to the optical microscope, thereby improving the spatiotemporal resolution of the image and obtaining a fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image, realizing large-field single-frame super-resolution imaging, and solving the defects of the multi-frame super-resolution imaging method in the prior art.

[0030] To facilitate understanding of this embodiment, a deconvolution super-resolution imaging method based on deep learning denoising disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the deconvolution super-resolution imaging method based on deep learning denoising provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a computing device, a wearable device, etc. In some possible implementations, the deconvolution super-resolution imaging method based on deep learning denoising can be implemented by a processor calling computer-readable instructions stored in a memory.

[0031] FIG1 shows a flow chart of a deconvolution super-resolution imaging method based on deep learning denoising provided by an embodiment of the present disclosure. As can be seen from FIG1 , the embodiment of the present disclosure includes at least steps S101 to S103:

[0032] Step S101: Acquire an image to be denoised, where the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule.

[0033] In this step, the structures to be imaged in the image to be denoised are cellular structures observed using a fluorescence microscope, including but not limited to the endoplasmic reticulum, mitochondria, centrosomes, Golgi apparatus, the morphology of the nucleolus, ribosomes, and lysosomes. Fluorescent molecules include ordinary fluorescent molecules and photoswitchable fluorescent molecules with photoswitchable properties.

[0034] Step S102: Denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein, a training data set for training the image denoising neural network is generated using the photoswitching properties of photoswitchable fluorescent molecules, and the photoswitchable fluorescent molecules are used to mark the structures to be imaged of each sample in the training data set; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the photoswitchable fluorescent molecules.

[0035] Research has found that deep learning-based methods can achieve efficient noise removal by building deep learning models and leveraging large-scale training data and parallel computing capabilities to learn complex image features and noise distribution patterns. Deep learning uses an end-to-end training approach, learning directly from input images to output images, reducing reliance on handcrafted feature design and improving algorithm robustness and generalization. Deep learning denoising methods also offer other advantages. For example, deep learning models can automatically adjust network parameters and learning strategies based on the characteristics of the input image and the type of noise, achieving adaptive denoising for different noise types. They use multi-layer nonlinear transformations to model the complex relationship between image and noise, thereby better capturing image detail and texture information. They also leverage image contextual information—the spatial and intensity correlations around pixels—to more accurately restore image detail and structure. Finally, deep learning models can enhance their expressiveness and denoising performance by increasing network depth and width, introducing more hidden layers and parameters. In summary, deep learning denoising methods possess strong learning capabilities and adaptability, enabling them to better restore image detail and structure, improving image quality and clarity.

[0036] However, in practical applications, such as biological microscopy, deep learning-based denoising methods require training on a large number of ground-truth images. However, obtaining large-scale, high-quality ground-truth images is difficult. Furthermore, the network models used in deep learning methods are typically trained on specific datasets. When applying a trained network model, the image content to be processed must be identical or similar to that in the training dataset. This results in poor robustness, generalizability, and applicability of the trained neural network model. Furthermore, many deep learning methods introduce translation invariance during the denoising process, making them insensitive to translations of the input image. This can lead to blurry or distorted denoising results. Deep learning methods are highly sensitive to perturbations in the input data; even minor perturbations can significantly alter the output. In denoising tasks, input images may contain subtle noise, which can prevent deep learning methods from accurately removing the noise. In summary, deep learning denoising methods require the development of more effective data augmentation techniques to improve the quality and quantity of ground-truth images, as well as the incorporation of more prior knowledge to enhance the robustness and generalizability of the models, in order to achieve better denoising results. Based on this, in this step, after obtaining the image to be denoised, the preset image denoising neural network can be used to denoise the image to be denoised to obtain a denoised image. Here, the preset image denoising neural network can be a deep learning network model, and the training data set for training the image denoising neural network is generated using the photoswitching properties of photoswitchable fluorescent molecules. Among them, photoswitchable fluorescent molecules refer to molecules that can undergo reversible changes in structure and properties under light stimulation. Specifically, photoswitchable fluorescent molecules can achieve conversion between bright and dark states under different lighting conditions, that is, under irradiation with a specific wavelength, the photoswitchable fluorescent molecules are in an "on" state, that is, a bright state or a luminescent state, and under irradiation with another specific wavelength, the photoswitchable fluorescent molecules are in an "off" state, that is, a dark state or a non-luminescent state.

[0037] Step S103: performing deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0038] After obtaining the denoised image, deconvolution processing can be performed on the denoised image to obtain a super-resolution image corresponding to the denoised image. In implementation, for example, the denoised image can be deconvolved using direct deconvolution to obtain the corresponding super-resolution image, or the denoised image can be deconvolved using iterative deconvolution to obtain the denoised image. The disclosed embodiments do not limit the deconvolution processing method.

[0039] As can be seen from the method shown in FIG1 , the embodiment of the present disclosure innovatively proposes a deconvolution super-resolution imaging method based on deep learning denoising, which obtains a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training data set used by the preset image denoising neural network is generated by utilizing the optical switching characteristics of the optical switch fluorescent molecules. By utilizing this feature, large-scale, more realistic, and extremely low-noise high-quality true value images can be obtained, which has great advantages in particular for deep tissue imaging. By training the network with the training data set containing the true value image, the robustness and stability of the preset image denoising neural network can be greatly improved. At the same time, since the optical switch fluorescent molecules can be combined with various subcellular The denoising neural network can be used to denoise the denoised image, and the denoising effect can be greatly improved, so the denoising effect of the denoised image is greatly improved, and the noise of the denoised image is extremely low. The denoised image is then deconvolved to reverse and restore the convolution phenomenon unique to the optical microscope, thereby improving the spatiotemporal resolution of the image and obtaining a completely restored original image, that is, the super-resolution imaging image corresponding to the denoised image, thereby realizing single-frame super-resolution imaging with a large field of view, and solving the defects of the multi-frame super-resolution imaging method in the prior art.

[0040] Furthermore, in order to better illustrate the process of the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising, as a refinement and extension of the above-mentioned embodiment, the embodiment of this application provides several optional embodiments, but is not limited thereto, as shown below:

[0041] In one possible implementation, the image to be denoised is determined by the following method A1 / method A2 / method A3:

[0042] Method A1: Acquire an original image and use the original image as the image to be denoised, wherein the original image is obtained by camera sampling.

[0043] Method A2: preprocess the original image and use the processed image as the image to be denoised, wherein the preprocessing methods include: optical phase-locked amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing or deconvolution processing, etc.

[0044] Method A3: The acquired target super-resolution imaging image is used as the image to be denoised, wherein the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image or MSSR super-resolution imaging image, etc.

[0045] In this embodiment, for method A2, the original image obtained by camera sampling can be preprocessed, and the obtained processed image can be used as the image to be denoised. For example, the original image 1 can be processed by an optical phase-locked amplification algorithm to obtain a processed image 1, and the processed image 1 can be used as the image to be denoised. The original image 2 can also be Gaussian filtered to obtain a processed image 2, and the processed image 2 can be used as the image to be denoised. The original image 3 can also be mean filtered to obtain a processed image 3, and the processed image 3 can be used as the image to be denoised. The original image 4 can also be Wiener filtered to obtain a processed image 4, and the processed image 4 can be used as the image to be denoised. The original image 5 can also be deconvolved to obtain a processed image 5, and the processed image 5 can be used as the image to be denoised. It should be noted that the examples given here are only illustrative and do not limit the embodiments of the present application.

[0046] For method A3, image 1 obtained using Airyscan super-resolution imaging technology can be used as the image to be denoised. Image 2 obtained using ISM super-resolution imaging technology can also be used as the image to be denoised. Image 3 obtained using A-POD super-resolution imaging technology can also be used as the image to be denoised. Image 4 obtained using MSSR super-resolution imaging technology can also be used as the image to be denoised. It should be noted that the examples listed here are merely illustrative and do not limit the embodiments of this application.

[0047] In one possible embodiment, the fluorescent molecule marks the structure to be imaged in the image to be denoised by the following method B1 / method B2:

[0048] Method B1: Fluorescent molecules are used to mark the structures to be imaged in the image to be denoised by molecular cloning.

[0049] Mode B2: The fluorescent molecule binds to the target antibody to label the structure to be imaged in the image to be denoised, wherein the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0050] In this embodiment, the types of target antibodies can be ordinary antibodies and nano antibodies. For example, fluorescent molecule A can be used to express in fusion with the structure to be imaged (referring to constructing a fusion gene with the 3' end of an exogenous protein gene and another gene for expression) through a molecular cloning method (providing a method for purifying and amplifying specific DNA fragments at the molecular level) to mark the structure to be imaged. The structure to be imaged can also be marked by combining fluorescent molecule B with nano antibody A, and the target antibody A specifically binds to the structure to be imaged. It should be noted that the examples given here are only illustrative and do not limit the embodiments of the present disclosure.

[0051] In a possible embodiment, in the above step S102, the photoswitchable fluorescent molecules include: fluorescent proteins with photoswitchable properties, fluorescent dyes with photoswitchable properties, or quantum dots.

[0052] In this embodiment, for fluorescent proteins with photoswitch properties, specifically, in one possible embodiment, the fluorescent proteins with photoswitch properties respond to light modulation and have two states, bright and dark, including: photoswitch fluorescent proteins, reversible photoswitch fluorescent proteins, and light-blinking fluorescent proteins. In one possible embodiment, the photoswitch fluorescent proteins include: Skylan-S, rsEGFP2, or Dronpa, etc. Here, Skylan-S (a new photoswitch protein) is a monomer that can be used for living cell imaging and has extremely high optical stability and switching properties. rsEGFP2 (reversible conversion enhanced green fluorescent protein 2) has the characteristics of easy crystallization and good optical properties. Dronpa (erasable optical protein) is a monomer fluorescent protein extracted from Pectiniidae (a petrified coral polyp) with unique photochromic properties, which can control the on and off of fluorescence with two different excitation lights.

[0053] Regarding fluorescent dyes with photoswitchable properties, specifically, one possible embodiment includes photoswitchable fluorescent dyes and scintillating fluorescent dyes. In practice, fluorescent dyes with photoswitchable properties may include fluorescein, cyanine dyes, rhodamine fluorescent dyes, and the like. It should be noted that the disclosed embodiments do not limit the types of fluorescent proteins and fluorescent dyes with photoswitchable properties.

[0054] Quantum dots, also known as semiconductor nanocrystals, can lock electrons in a very small three-dimensional space, restricting the movement of electrons. When a certain electric field or light is applied to them, they will emit light of a specific frequency. The frequency of the light emission changes with the change of particle size. By adjusting the size of the quantum dots, the color of their light emission can be controlled.

[0055] In one possible embodiment, the types of light emitted by the photoswitchable fluorescent molecules include: blue light, green light, yellow light, red light, or light of far-infrared wavelength, etc. The embodiments of the present disclosure do not limit the types of light emitted by the photoswitchable fluorescent molecules.

[0056] In one possible implementation, in step S102, the preset image denoising neural network is generated according to the following method:

[0057] Step C1: Acquire multiple fluorescence image sequences, where the fluorescence image sequences include at least one bright state image and at least one dark state image.

[0058] Step C2: performing first image processing on each bright-state image and each dark-state image to obtain input images of a plurality of fluorescence image sequences.

[0059] Step C3: performing a second image processing on each bright-state image and each dark-state image to obtain a true-value image of a plurality of fluorescence image sequences, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the true-value image.

[0060] Step C4: Generate a training data set based on each input image and each true value image; use the training data set to train the initial image denoising neural network to obtain a preset image denoising neural network.

[0061] In this embodiment, in order to obtain a preset image denoising neural network, a training data set can be first obtained. Specifically, multiple fluorescence image sequences can be obtained first. Specifically, in one possible embodiment, the fluorescence image sequences are obtained using optical imaging technology, including: wide-field fluorescence microscopy technology, confocal fluorescence microscopy technology, structured light illumination imaging technology, single-molecule localization microscopy technology, stimulated emission depletion super-resolution microscopy technology, optical fluctuation super-resolution imaging technology, single-photon fluorescence imaging technology, two-photon microscopy technology, multi-photon microscopy technology, or photoacoustic imaging technology.

[0062] For example, after labeling the structure to be imaged with photoswitchable fluorescent molecules, modulated light can be used to illuminate the structure to be imaged. When the photoswitchable fluorescent molecules are in a luminescent state, fluorescence images can be captured using an optical imaging device, such as a wide-field fluorescence microscope, to obtain one or more bright-state images. Since the structure to be imaged is labeled with the photoswitchable fluorescent molecules, each bright-state image contains signal data and noise data generated by the structure to be imaged, where the noise data includes Gaussian noise, Poisson noise, and sample autofluorescence. When the majority of the photoswitchable fluorescent molecules are in a non-luminescent state, fluorescence images can be captured to obtain one or more dark-state images, each of which contains weak signal data and noise data. Ultimately, multiple fluorescence image sequences can be obtained, such as fluorescence image sequence a: bright-state image a1, dark-state image a2; fluorescence image sequence b: bright-state image b1, bright-state image b2, dark-state image b3; and fluorescence image sequence c: bright-state image c1, bright-state image c2, dark-state image c3, and dark-state image c4. The number of fluorescence image sequences is not limited in this embodiment and can be set according to actual conditions, for example, the number can be set to 500. The optical imaging technology used to obtain the fluorescence image sequence is not limited in this embodiment and can be selected according to actual needs.

[0063] After obtaining the plurality of fluorescence image sequences, first image processing may be performed on each bright-state image and each dark-state image to obtain input images for the fluorescence image sequences. Specifically, in one possible implementation, after obtaining the plurality of fluorescence image sequences, first image processing may be performed on each bright-state image and each dark-state image to obtain input images for the plurality of fluorescence image sequences using the following method D1 / method D2, including:

[0064] Method D1: Use any one of the bright state images as the input image.

[0065] Method D2: Select a first fluorescent image pair from a plurality of fluorescent image sequences, subtract the bright state image from the dark state image in the first fluorescent image pair to obtain a first subtracted image; and use the first subtracted image as the input image.

[0066] In this embodiment, for mode D1, any one of the bright-state images can be used as the input image. For example, if the multiple fluorescence image sequences include bright-state image 1, bright-state image 2, and bright-state image 3, then any one of these images, for example, bright-state image 1, can be selected as the input image. It should be noted that this example is for illustrative purposes only and does not limit the present embodiment.

[0067] For method D2, a group of first fluorescent image pairs can be selected from multiple fluorescent image sequences, and the bright state image and the dark state image in the first fluorescent image pair can be subtracted to obtain a first subtracted image; the first subtracted image can be used as the input image. Exemplarily, the multiple fluorescent image sequences include a fluorescent image sequence d and a fluorescent image sequence e, and the fluorescent image sequence d includes a bright state image d1, a bright state image d2, a bright state image d3, and a dark state image d4. The fluorescent image sequence e includes a bright state image e1, a bright state image e2, and a dark state image e3. A group of first fluorescent image pairs can be selected from each sequence. For example, the first fluorescent image pair includes a bright state image d1 and a dark state image e1. Then, the pixel values ​​of the corresponding pixel points of the bright state image d1 and the dark state image e1 can be subtracted to obtain a first subtracted image, and the first subtracted image can be used as the input image. It should be noted that the examples given here are only illustrative and do not limit the embodiments of the present application.

[0068] Next, a second image processing can be performed on each bright-state image and each dark-state image to obtain a plurality of true-value images of the fluorescence image sequence, wherein the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the true-value image. Specifically, in one possible embodiment, after obtaining the plurality of fluorescence image sequences, the second image processing can be performed on each bright-state image and each dark-state image using the following method E1 / method E2 to obtain the true-value images of the plurality of fluorescence image sequences:

[0069] Method E1: for any plurality of second fluorescence image pairs in a plurality of fluorescence image sequences, subtract the bright state image from the dark state image in each second fluorescence image pair to obtain each second subtracted image; and generate a true value image based on each second subtracted image.

[0070] Method E2: Using an optical lock-in amplification algorithm, multiple fluorescence image sequences are processed to obtain a true value image.

[0071] In this embodiment, for method E1, for any number of second fluorescence image pairs in multiple fluorescence image sequences, the pixel values ​​of corresponding pixels in the bright-state image and the dark-state image in each second fluorescence image pair can be subtracted to obtain respective second subtracted images. Then, a true image is generated based on each second subtracted image. For example, the multiple fluorescence image sequences include fluorescence image sequence f and fluorescence image sequence g. For fluorescence image sequence f, the pixel values ​​of corresponding pixels in the bright-state image f1 and the dark-state image f2 in fluorescence image sequence f can be subtracted to obtain second subtracted image 1. For fluorescence image sequence g, the pixel values ​​of corresponding pixels in the bright-state image g1 and the dark-state image g3 in fluorescence image sequence g can be subtracted to obtain second subtracted image 2. A weighted average of the second subtracted images can then be performed to obtain the true image. For example, the weights of second subtracted images 1 and 2 can be set to 1 / 2, so that the true image = 1 / 2 second subtracted image 1 + 1 / 2 second subtracted image 2. The weights of the second subtracted images are not limited in the present embodiment and can be set according to actual needs. It should be noted that the examples given here are only illustrative and do not limit the present embodiment.

[0072] For method E2, for example, an optical lock-in amplification algorithm (OLID algorithm) can be used to process the fluorescence image sequence h, the fluorescence image sequence i, the fluorescence image sequence j, and the fluorescence image sequence k to obtain a true value image. It should be noted that the examples given here are merely illustrative and do not limit the embodiments of the present disclosure.

[0073] Then, a training data set can be generated based on each input image and each ground-truth image. In a specific implementation, for example, 1000 frames of images can be collected using an image acquisition device. Assuming that one input image and one ground-truth image can be generated based on 20 fluorescence image sequences, the 1000 frames of images can be first divided into the following groups according to the order of collection: fluorescence image sequence group 1, fluorescence image sequence group 2, ..., fluorescence image sequence group 50. Input image 1 and ground-truth image 1 can be generated based on fluorescence image sequence group 1, input image 2 and ground-truth image 2 can be generated based on fluorescence image sequence group 2, ...; input image 50 and ground-truth image 50 can be generated based on fluorescence image sequence group 50. The input image and ground-truth image form a training pair, and the 50 training pairs constitute the training data set.

[0074] After obtaining the training data set, the initial image denoising neural network can be trained using the training data set to obtain a preset image denoising neural network. In a specific application scenario, in one possible implementation, the deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model, or a self-supervised model, etc.; wherein the supervised model includes a DnCNN model or a Noise2Clean model, etc.; the unsupervised model includes a BM3D model or a BM4D model, etc.; the self-supervised model includes a Deep CAD model, a Deep CAD-RT model, a Noise2Noise model, a Noise2Void model, a Self2Self model, a Self2Self+ model, a Neighbor2Neighbor model, a Noise2Fast model, a Dilated Blind-Spot Network model, a blind2unblind model, or a Deep Image Prior model, etc.

[0075] Here, the denoising convolutional neural network model (DnCNN) is a deep residual network specially designed for image denoising. It learns the noise features in the image and removes the noise by stacking multiple convolutional layers and residual connections.

[0076] The 3D Block Matching model (BM3D) performs simple denoising by matching the original image to form a basic estimate. It then uses the original image and the basic estimate for more detailed denoising, further improving the peak signal-to-noise ratio. The 4D Block Matching Filtering model (BM4D) uses similar principles to the BM3D model and can extend 2D denoising to 3D space.

[0077] The self-supervised deep learning model (DeepCAD) does not require any high-SNR observation images; network training can be achieved using only a single low-SNR calcium imaging sequence. This can suppress detection noise and improve the SNR by more than 10-fold. A self-supervised deep learning model for real-time noise suppression (DeepCAD-RT) has significant advantages in functional imaging. Noise2Noise uses paired noisy images of the same scene as training images, achieving the same training results as supervised conditions. The Noise2Void model uses the noisy images themselves as supervision, preventing the learning of identity mappings through a blind network. A self-supervised learning single-image denoising network model (Self2Self model) requires only a single noisy image (no ground truth) for training. The Self2Self+ model is an optimization of the Self2Self model. The Neighbor2Neighbor model, based on the Noise2Noise model, replaces noise-clean image pairs with noise-noise image pairs, solving the problem of collecting large amounts of noise-clean image data. A self-supervised image denoising model with visible blind spots (blind2unblind) can overcome the information loss in blind-spot-driven denoising methods. The Dilated Blind-Spot Network (D-BSN) can learn denoising models solely from real noisy images. The Deep Image Prior model is an image denoising method that leverages the implicit prior information inherent in the structure of neural networks.

[0078] During implementation, the image denoising neural network to be trained can also be generated based on other deep learning models. The embodiments of the present disclosure are not limited to this. For example, the image denoising neural network to be trained can also be generated based on the Recorded2Recorrupted model.

[0079] In this embodiment, the initial image denoising neural network generated based on the above-mentioned deep learning model can be used to train the training data set. The above-mentioned deep learning model can efficiently remove various noises in the imaging process, such as Gaussian noise, Poisson noise, etc., thereby greatly improving the denoising effect of the preset image denoising neural network.

[0080] During implementation, each input image in the training sample set can be input into the initial image denoising neural network to obtain multiple predicted images. Then, based on the preset loss function, the parameters in the initial image denoising neural network are updated according to the multiple predicted images and the corresponding true value images to obtain the preset image denoising neural network.

[0081] In one possible implementation, performing deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image specifically includes:

[0082] Step F: Based on the Richardson-Lucy algorithm, deconvolution is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0083] Research has found that for completely noise-free blurred images, the original image can be fully restored using the Richardson-Lucy (RL) deconvolution method. However, in the actual imaging process, due to uncertainties such as optical system noise and the point spread function (PSF), the RL deconvolution algorithm often results in a large number of artifacts in the image. Moreover, as the number of deconvolution iterations increases, the noise signal is amplified, and deconvolution further introduces more artifacts into the image.

[0084] Based on this, in this embodiment, the denoised image can be deconvolved based on the Richardson-Lucy algorithm to obtain a super-resolution imaging image corresponding to the denoised image. Since the denoised image is an image processed by a preset image denoising neural network, the training data set used by the preset image denoising neural network is obtained by utilizing the photoswitching characteristics of photoswitching fluorescent molecules, which makes the preset image denoising neural network more robust, stable, and versatile, and the denoising effect is greatly improved. Therefore, the noise of the denoised image is extremely low. Therefore, by using the RL algorithm to deconvolve the denoised image, the original image can be fully restored, and the spatiotemporal resolution of the image can be improved, thereby achieving the purpose of single-frame super-resolution imaging.

[0085] In specific implementation, the system point spread function can be determined according to the following formula (1):

[0086] Where I0 is the normalization coefficient, J1 is the Bessel function of the first kind, λ is the wavelength, NA is the numerical aperture, and ρ is the polar diameter in polar coordinates. When fluorescence emitted by a sample passes through an optical system, due to diffraction, the image formed on the camera is the convolution of the sample fluorescence and the system's point spread function. Therefore, to better reflect the diffraction process of an optical system, the system's point spread function can be represented using the Bessel function of the first kind.

[0087] Taking into account the influence of noise in imaging, too many deconvolution iterations will produce artifacts, while insufficient iterations cannot improve the image resolution well. Therefore, the number of deconvolution iterations can be determined according to the quality of the denoised image. Specifically, the number of iterations can be determined by calculating the Fourier ring correlation resolution (FRC resolution) of two consecutive frames of images in the actual sample to ensure that the image resolution reaches the highest and there are no obvious reconstruction artifacts.

[0088] Specifically, in one possible embodiment, the imaging objects of the deconvolution super-resolution imaging method based on deep learning denoising include: fixed cells, living cells, tissues, living animals, fungi, or fluorescent standards. The present disclosure does not limit the imaging objects of the method. The present disclosure can achieve fast 3D super-resolution imaging for fixed cells, and for living cell tissues, a series of dynamic images that change over time can be obtained to reveal the rapid dynamic changes of subcellular structures.

[0089] FIG2 shows a flow chart of a deconvolution super-resolution imaging method based on deep learning denoising provided by another embodiment of the present disclosure. As shown in FIG2 , this embodiment includes the following steps S201 to S209:

[0090] Step S201: Using a photoswitchable molecular labeling structure, a plurality of fluorescence image sequences are acquired, wherein the fluorescence image sequences include at least one bright-state image and at least one dark-state image. During implementation, the fluorescence image sequences are acquired using optical imaging techniques, including wide-field fluorescence microscopy, confocal fluorescence microscopy, structured light illumination imaging, single-molecule localization microscopy, stimulated emission depletion super-resolution microscopy, optical wave super-resolution imaging, single-photon fluorescence imaging, two-photon microscopy, multiphoton microscopy, or photoacoustic imaging.

[0091] The following uses photoswitchable fluorescent protein as an example to specifically describe how to obtain multiple fluorescence image sequences of the target molecule. First, through cell transfection technology (a technology that introduces exogenous molecules such as DNA, RNA, etc. into eukaryotic cells), F-actin is fused with photoswitchable fluorescent protein (RSFP) and expressed in eukaryotic cells: human osteosarcoma cells U-2 OS. After 36 hours, the cytoskeleton is fixed with a fixative, and fluorescence images are collected using a homemade or commercial fluorescence microscopy imaging platform. Among them, the microscope body model is model A, using 405nm laser and 488nm laser, 150×, 1.45NA objective lens, camera model is model B, and pixel size is 6.5μm. Figure 3 shows the laser modulation timing diagram provided by the embodiment of the present disclosure. Referring to the laser modulation mode shown in Figure 3, the 405nm and 488nm lasers can be turned on at the same time, and the 405nm laser can be turned off after starting to take pictures and collect images to obtain an image sequence; the 488nm laser is turned on, and the 405nm laser is turned on after starting to take pictures and collect images to obtain an image sequence. Ultimately, a series of fluorescence images with repeated switching states, i.e., multiple fluorescence image sequences, can be obtained.

[0092] Step S202: Select any one of the bright-state images as the input image. Alternatively, a first fluorescence image pair may be selected from a plurality of fluorescence image sequences, and the bright-state image and the dark-state image in the first fluorescence image pair may be subtracted to obtain a first subtracted image; and the first subtracted image may be used as the input image.

[0093] Step S203 : for any plurality of second fluorescence image pairs in the plurality of fluorescence image sequences, subtract the bright state image from the dark state image in each second fluorescence image pair to obtain second subtracted images.

[0094] Step S204: Generate a true image based on each second subtracted image. During implementation, an optical lock-in amplification algorithm can also be used to process multiple fluorescence image sequences to obtain a true image. The signal-to-noise ratio of the input image is lower than the signal-to-noise ratio of the true image.

[0095] Step S205: Generate a training data set based on the input image and the true value image.

[0096] Step S206: The initial image denoising neural network is trained using the training data set to obtain a preset image denoising neural network. Here, the training data set is generated using the photoswitching properties of photoswitchable fluorescent molecules, which are used to mark the structures to be imaged for each sample in the training data set. Photoswitchable fluorescent molecules include: fluorescent proteins with photoswitchable properties, fluorescent dyes with photoswitchable properties, or quantum dots. Among them, fluorescent proteins with photoswitchable properties respond to light modulation and have two states: bright state and dark state, including: photoswitchable fluorescent proteins, reversible photoswitchable fluorescent proteins, or light-blinking fluorescent proteins, among others. Among them, photoswitchable fluorescent proteins include: Skylan-S, rsEGFP2, or Dronpa, among others. Fluorescent dyes with photoswitchable properties include: photoswitchable dyes or light-blinking fluorescent dyes, among others. The types of light emitted by photoswitchable fluorescent molecules include: blue light, green light, yellow light, red light, or light with far-infrared wavelengths, among others. Among them, the deep learning model based on the initial image denoising neural network includes: a supervised model, an unsupervised model or a self-supervised model, etc.; among them, the supervised model includes a DnCNN model or a Noise2Clean model, etc.; the unsupervised model includes a BM3D model or a BM4D model, etc.; the self-supervised model includes a Deep CAD model, a Deep CAD-RT model, a Noise2Noise model, a Noise2Void model, a Self2Self model, a Self2Self+ model, a Neighbor2Neighbor model, a Noise2Fast model, a Dilated Blind-Spot Network model, a blind2unblind model or a Deep Image Prior model, etc.

[0097] Step S207: Preprocess the original image acquired by camera sampling, and use the processed image as the image to be denoised. Preprocessing methods include optical lock-in amplification, Gaussian filtering, mean filtering, Wiener filtering, or deconvolution. During implementation, the original image can also be used directly as the image to be denoised. Alternatively, the acquired target super-resolution image can be used as the image to be denoised. The target super-resolution image includes an Airyscan super-resolution image, an ISM super-resolution image, an A-POD super-resolution image, or an MSSR super-resolution image. The structures to be imaged in the image to be denoised are labeled with fluorescent molecules. For example, fluorescent molecules can be used to label the structures to be imaged in the image to be denoised through molecular cloning. In another example, fluorescent molecules can be used to label the structures to be imaged in the image to be denoised by binding to a target antibody, where the target antibody specifically binds to the structures to be imaged in the image to be denoised. During implementation, the imaging object, i.e., the structures to be imaged, includes fixed cells, living cells, tissues, living animals, fungi, or fluorescent standards.

[0098] Step S208: De-noise the image to be de-noised using a preset image de-noising neural network to obtain a de-noised image. Here, the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optically switchable fluorescent molecule.

[0099] Step S209: performing deconvolution processing on the denoised image based on the Richardson-Lucy algorithm to obtain a super-resolution imaging image corresponding to the denoised image.

[0100] FIG4 shows a comparison diagram of the image processing results of direct deconvolution without denoising and deconvolution after denoising provided by the embodiment of the present disclosure. Among them, a corresponds to the wide-field image (upper left), the image obtained by direct deconvolution of the wide-field image (middle right), the image obtained by deconvolution after denoising (upper right), b is the corresponding magnified image of a (scale: 2μm (a); 1μm (b)), and c is a comparison diagram of the image resolution without direct deconvolution and after deconvolution after denoising. Referring to FIG4 , according to a and b, it can be seen that the image processed by deconvolution after denoising is clearer, which means that the image resolution is higher and the artifacts are less. According to c, the image processed by deconvolution after denoising has a lower nm number, indicating a higher resolution. It can be seen that the image denoising neural network based on the optical switching properties of the photoswitch molecule can effectively remove noise in the image, and then through deconvolution processing, the spatiotemporal resolution of the denoised image is greatly improved, and a super-resolution imaging image is obtained, thereby realizing large-field single-frame super-resolution imaging.

[0101] Those skilled in the art will understand that in the above-mentioned methods of specific embodiments, the order in which the steps are written does not imply a strict order of execution, but rather does not limit the implementation process. The specific order of execution of each step should be determined by its function and possible internal logic. It should be noted that in actual applications, all possible implementation methods described above can be combined in any manner to form possible embodiments of the present disclosure, and will not be described in detail here.

[0102] Based on the same concept, the present disclosure also provides a deconvolution super-resolution imaging device based on deep learning denoising. FIG5 shows a schematic structural diagram of the deconvolution super-resolution imaging device based on deep learning denoising provided by the present disclosure. Referring to FIG5 , the deconvolution super-resolution imaging device 500 based on deep learning denoising provided by the present disclosure includes:

[0103] An acquisition module 501 is configured to acquire an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule;

[0104] Denoising module 502, configured to denoise the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein a training data set for training the image denoising neural network is generated using the photoswitchable properties of a photoswitchable fluorescent molecule, the photoswitchable fluorescent molecule being used to mark the structure to be imaged of each sample in the training data set; and the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photoswitchable fluorescent molecule;

[0105] The super-resolution imaging module 503 is configured to perform a deconvolution process on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0106] In a possible embodiment, in the above device, the photoswitchable fluorescent molecules include: fluorescent proteins with photoswitchable properties, fluorescent dyes with photoswitchable properties, or quantum dots.

[0107] In one possible embodiment, in the above device, the fluorescent protein with photoswitch properties responds to light modulation and has two states: bright state and dark state, including: photoswitchable fluorescent protein, reversible photoswitchable fluorescent protein or light-blinking fluorescent protein, etc.

[0108] In a possible embodiment, in the above device, the photoswitchable fluorescent protein includes: Skylan-S, rsEGFP2 or Dronpa, etc.

[0109] In a possible implementation, in the above device, the fluorescent dye with photoswitch properties includes: photoswitch dye or light-scintillation fluorescent dye, etc.

[0110] In a possible implementation, in the above device, the types of light emitted by the optical switch fluorescent molecules include: blue light, green light, yellow light, red light, or light with far-infrared wavelengths.

[0111] In a possible implementation, the apparatus further includes a network training module, wherein the network training module is configured to:

[0112] Acquire a plurality of fluorescence image sequences, wherein the fluorescence image sequences include at least one bright state image and at least one dark state image;

[0113] performing a first image processing on each of the bright-state images and each of the dark-state images to obtain input images of the plurality of fluorescence image sequences;

[0114] performing a second image processing on each of the bright-state images and each of the dark-state images to obtain a true-value image of the plurality of fluorescence image sequences, wherein a signal-to-noise ratio of the input image is less than a signal-to-noise ratio of the true-value image;

[0115] The training data set is generated according to each of the input images and each of the true value images; the initial image denoising neural network is trained using the training data set to obtain the preset image denoising neural network.

[0116] In one possible implementation, the network training module, when performing the first image processing on each of the bright-state images and each of the dark-state images to obtain the input images of the plurality of fluorescence image sequences, is configured to:

[0117] Using any one of the bright state images as the input image; or

[0118] Selecting a first fluorescent image pair from the plurality of fluorescent image sequences, subtracting the bright state image from the dark state image in the first fluorescent image pair to obtain a first subtracted image; and using the first subtracted image as the input image.

[0119] In one possible implementation, the network training module, when performing the second image processing on each of the bright-state images and each of the dark-state images to obtain true-value images of the plurality of fluorescence image sequences, is configured to:

[0120] For any plurality of second fluorescence image pairs in the plurality of fluorescence image sequences, subtract the bright-state image from the dark-state image in the second fluorescence image pair to obtain a second subtracted image; and generate the true value image based on each of the second subtracted images; or

[0121] The plurality of fluorescence image sequences are processed using an optical lock-in amplification algorithm to obtain the true value image.

[0122] In one possible implementation, in the above-mentioned network training module, the deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model or a self-supervised model, etc.; wherein, the supervised model includes a DnCNN model or a Noise2Clean model, etc.; the unsupervised model includes a BM3D model or a BM4D model, etc.; the self-supervised model includes a Deep CAD model, a Deep CAD-RT model, a Noise2Noise model, a Noise2Void model, a Self2Self model, a Self2Self+ model, a Neighbor2Neighbor model, a Noise2Fast model, a Dilated Blind-Spot Network model, a blind2unblind model or a Deep Image Prior model, etc.

[0123] In a possible implementation, in the above device, the acquisition module 501 is configured to:

[0124] Acquire an original image and use the original image as the image to be denoised, wherein the original image is obtained by camera sampling; or

[0125] Preprocessing the original image and using the processed image as the image to be denoised, wherein the preprocessing method includes: optical lock-in amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing or deconvolution processing; or

[0126] The acquired target super-resolution imaging image is used as the image to be denoised, wherein the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image or MSSR super-resolution imaging image, etc.

[0127] In one possible embodiment, in the above device, the fluorescent molecule marks the structure to be imaged in the image to be denoised by the following method:

[0128] The fluorescent molecules are labeled with the structure to be imaged in the image to be denoised by a molecular cloning method; or

[0129] The fluorescent molecule marks the structure to be imaged in the image to be denoised by binding to the target antibody, wherein the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0130] In one possible implementation, in the above device, the super-resolution imaging module 503 is configured to:

[0131] Based on the Richardson-Lucy algorithm, deconvolution processing is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0132] In one possible embodiment, in the above-mentioned network training module, the fluorescence image sequence is obtained through optical imaging technology, and the optical imaging technology includes: wide-field fluorescence microscopy technology, confocal fluorescence microscopy technology, structured light illumination imaging technology, single-molecule localization microscopy technology, stimulated emission depletion super-resolution microscopy technology, optical fluctuation super-resolution imaging technology, single-photon fluorescence imaging technology, two-photon microscopy technology, multi-photon microscopy technology or photoacoustic imaging technology, etc.

[0133] In a possible implementation manner, the imaging objects of the method and apparatus include: fixed cells, living cells, tissues, living animals, fungi, or fluorescent standards, etc.

[0134] It should be noted that any of the above-mentioned deconvolution super-resolution imaging devices based on deep learning denoising can implement the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising one by one, which will not be repeated here.

[0135] Based on the same technical concept, an embodiment of the present disclosure also provides an electronic device. Referring to Figure 6, a schematic diagram of the structure of the electronic device provided in the embodiment of the present disclosure includes a processor 601, a memory 602, and a bus 603. Among them, the memory 602 is used to store execution instructions, including a memory 6021 and an external memory 6022; the memory 6021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 601, as well as the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the electronic device 600 is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the following instructions:

[0136] Acquiring an image to be denoised, wherein the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule;

[0137] Denoising the image to be denoised using a preset image denoising neural network to obtain a denoised image; wherein a training data set for training the image denoising neural network is generated using the photoswitchable properties of a photoswitchable fluorescent molecule, the photoswitchable fluorescent molecule being used to mark the structure to be imaged of each sample in the training data set; and the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photoswitchable fluorescent molecule;

[0138] Deconvolution is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0139] The specific processing flow of the processor 601 can refer to the description of the above method embodiment and will not be repeated here.

[0140] In addition, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the deconvolution super-resolution imaging method based on deep learning denoising described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0141] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the deconvolution super-resolution imaging method based on deep learning denoising in the above-mentioned method embodiment. For details, please refer to the above-mentioned method embodiment, which will not be repeated here.

[0142] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0143] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0144] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0146] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.

[0147] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A deconvolution super-resolution imaging method based on deep learning denoising, wherein, The method includes: Obtaining a to-be-denoised image, where the to-be-imaged structure of the to-be-denoised image is labeled with a fluorescent molecule; Denosing the to-be-denoised image by using a preset image denoising neural network to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated by using the optical switching characteristics of an optically switchable fluorescent molecule, and the optically switchable fluorescent molecule is used to label the to-be-imaged structures of the samples in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optically switchable fluorescent molecule; Performing deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

2. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The optically switchable fluorescent molecule includes: a fluorescent protein with optical switching properties, a fluorescent dye with optical switching properties, or a quantum dot.

3. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 2, wherein, The fluorescent protein with optical switching properties responds to light modulation and has two states, a bright state and a dark state, including: an optically switchable fluorescent protein, a reversibly optically switchable fluorescent protein, or an optically blinking fluorescent protein, etc.

4. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 3, wherein, The optically switchable fluorescent protein includes: Skylan-S, rsEGFP2, or Dronpa, etc.

5. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 2, wherein, The fluorescent dye with optical switching properties includes: an optically switchable dye or an optically blinking fluorescent dye, etc.

6. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The types of light emitted by the optically switchable fluorescent molecule include: blue light, green light, yellow light, red light, or light with a far-infrared wavelength, etc.

7. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The preset image denoising neural network is generated according to the following method: Obtaining a plurality of fluorescent image sequences, where each fluorescent image sequence includes at least one bright-state image and at least one dark-state image; Performing first image processing on each of the bright-state images and each of the dark-state images to obtain input images of the plurality of fluorescent image sequences; Performing second image processing on each of the bright-state images and each of the dark-state images to obtain ground-truth images of the plurality of fluorescent image sequences, where the signal-to-noise ratio of the input images is less than the signal-to-noise ratio of the ground-truth images; Generating the training data set according to each of the input images and each of the ground-truth images; using the training data set to train an initial image denoising neural network to obtain the preset image denoising neural network.

8. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 7, wherein, The performing first image processing on each of the bright-state images and each of the dark-state images to obtain input images of the plurality of fluorescent image sequences includes: Taking any one of the bright-state images as the input image; or Selecting any group of first fluorescent image pairs from the plurality of fluorescent image sequences, subtracting the dark-state image from the bright-state image in the first fluorescent image pair to obtain a first subtracted image; and taking the first subtracted image as the input image.

9. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 7, wherein, The performing second image processing on each of the bright-state images and each of the dark-state images to obtain ground-truth images of the plurality of fluorescent image sequences includes: For any multiple groups of second fluorescent image pairs in the plurality of fluorescent image sequences, subtracting the dark-state image from the bright-state image in each of the second fluorescent image pairs to obtain second subtracted images; generating the ground-truth images according to each of the second subtracted images; or Using an optical lock-in amplification algorithm to process the plurality of fluorescent image sequences to obtain the ground-truth images.

10. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 7, wherein, The deep learning model on which the initial image denoising neural network is based includes: supervised models, unsupervised models, self-supervised models, etc.; Among them, the supervised models include DnCNN model, Noise2Clean model, etc.; the unsupervised models include BM3D model, BM4D model, etc.; the self-supervised models include Deep CAD model, Deep CAD-RT model, Noise2Noise model, Noise2Void model, Self2Self model, Self2Self+ model, Neighbor2Neighbor model, Noise2Fast model, Dilated Blind-Spot Network model, blind2unblind model, Deep Image Prior model, etc.

11. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The image to be denoised is determined by the following method: Obtain the original image and use the original image as the image to be denoised, where the original image is obtained by camera sampling; or Preprocess the original image and use the processed image obtained as the image to be denoised, where the preprocessing methods include: optical lock-in amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing, deconvolution processing, etc.; or Use the obtained target super-resolution imaging image as the image to be denoised, where the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image, MSSR super-resolution imaging image, etc.

12. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The fluorescent molecule labels the structure to be imaged in the image to be denoised by the following method: The fluorescent molecule labels the structure to be imaged in the image to be denoised by molecular cloning; or The fluorescent molecule binds to the target antibody to label the structure to be imaged in the image to be denoised, where the target antibody specifically binds to the structure to be imaged in the image to be denoised.

13. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, Performing deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image includes: Based on the Richardson-Lucy algorithm, perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

14. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 7, wherein, The fluorescent image sequence is obtained by optical imaging technology, and the optical imaging technology includes: wide-field fluorescence microscopy imaging technology, confocal fluorescence microscopy imaging technology, structured illumination imaging technology, single molecule localization microscopy imaging technology, stimulated emission depletion super-resolution microscopy imaging technology, optical fluctuation super-resolution imaging technology, single photon fluorescence imaging technology, two-photon microscopy imaging technology, multi-photon microscopy imaging technology, photoacoustic imaging technology, etc.

15. The deconvolution super-resolution imaging method based on deep learning denoising according to any one of claims 1-14, wherein, The imaging objects of the method include: fixed cells, live cells, tissues, living animals, fungi, fluorescent standards, etc.

16. A deconvolution super-resolution imaging device based on deep learning denoising, wherein, The device includes: An acquisition module for acquiring an image to be denoised, where the structure to be imaged in the image to be denoised is labeled with a fluorescent molecule; A denoising module, configured to denoise the image to be denoised by using a preset image denoising neural network, so as to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated by using the optical switching characteristics of an optical switching fluorescent molecule, and the optical switching fluorescent molecule is used to label the structure to be imaged in each sample in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switching fluorescent molecule; A super-resolution imaging module, configured to perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

17. An electronic device, comprising: A processor; And a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to execute the steps of the method according to any one of claims 1-15.

18. A computer-readable storage medium storing one or more programs, wherein, When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the steps of the method according to any one of claims 1-15.

Images